-----============= acceptance-small: sanity-flr ============----- Sat Sep 12 22:09:03 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg244-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 22:09:27 (1789265367) === oleg244-client.virtnet: executing check_config_client /mnt/lustre oleg244-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg244-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9bbb50f76000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9bbb50f76000.idle_timeout=debug disable quota as required oleg244-server: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 22:10:06 (1789265406) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7642] == sanity-flr test 0a: lfs mirror create with -N option == 22:10:09 (1789265409) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 22:10:27 (1789265427) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 22:10:31 (1789265431) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 22:10:35 (1789265435) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 22:11:03 (1789265463) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 22:11:07 (1789265467) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 22:11:12 (1789265472) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9bbb50f76000.stats=clear osc.lustre-OST0001-osc-ffff9bbb50f76000.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 22:11:21 (1789265481) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 22:11:32 (1789265492) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 22:11:41 (1789265501) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 22:11:45 (1789265505) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 22:11:54 (1789265514) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 22:12:04 (1789265524) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 22:12:13 (1789265533) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 22:12:17 (1789265537) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 22:12:25 (1789265545) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 22:12:36 (1789265556) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 22:12:44 (1789265564) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 1.68108 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133843 s, 7.8 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 22:12:55 (1789265575) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.425671 s, 37.0 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 22:13:04 (1789265584) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 22:13:16 (1789265596) Stopping /mnt/lustre-ost1 (opts:) on oleg244-server oleg244-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9bbb50f76000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9bbb50f76000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg244-server: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg244-server: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg244-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg244-server oleg244-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9bbb50f76000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9bbb50f76000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg244-server: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg244-server: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg244-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (81s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 22:14:38 (1789265678) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.746482 s, 12.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg244-server oleg244-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9bbb50f76000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9bbb50f76000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg244-server: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg244-server: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg244-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (45s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 22:15:23 (1789265723) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg244-server oleg244-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9bbb50f76000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9bbb50f76000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg244-server: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg244-server: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg244-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg244-server oleg244-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9bbb50f76000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9bbb50f76000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg244-server: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg244-server: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg244-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (89s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 22:16:53 (1789265813) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 6.15126 s, 8.5 MB/s -rw-r--r-- 1 root root 52428800 Sep 12 22:17 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.633s user 0m0.000s sys 0m0.198s Stopping /mnt/lustre-ost1 (opts:) on oleg244-server oleg244-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9bbb50f76000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9bbb50f76000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.230s user 0m0.000s sys 0m0.190s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg244-server: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg244-server: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg244-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg244-server oleg244-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9bbb50f76000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9bbb50f76000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.174s user 0m0.000s sys 0m0.183s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg244-server: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg244-server: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg244-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg244-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9bbb50f76000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9bbb50f76000.ost_server_uuid in FULL state after 0 sec PASS 33b (97s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 22:18:29 (1789265909) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 22:18:32 (1789265912) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 22:18:35 (1789265915) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 22:18:38 (1789265918) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 22:18:45 (1789265925) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.50328 s, 25.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 22:18:57 (1789265937) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282804 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330598 s, 31.7 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 22:19:05 (1789265945) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 22:19:14 (1789265954) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00174925 s, 585 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00136689 s, 749 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00190554 s, 537 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00201977 s, 507 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 22:19:25 (1789265965) create test files with size 15752340 .. e125d0549d24134abc472e692fecfb17 - f7ed9973407f5d3d9a40b333f0ebdffc - 64577a2d02485df2064a98b511ae3663 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg244-client.virtnet /mnt/lustre (opts:) Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 22:19:53 (1789265993) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (20s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 22:20:13 (1789266013) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 22:20:20 (1789266020) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126798 s, 8.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xe0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xe1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xbb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706241 s, 14.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xe2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 22:20:31 (1789266031) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.24786 s, 8.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.546795 s, 7.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.208999 s, 10.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 22:20:49 (1789266049) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 22:20:52 (1789266052) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 22:20:56 (1789266056) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038051 s, 27.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 22:21:06 (1789266066) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 22:21:09 (1789266069) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 22:21:19 (1789266079) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.416878 s, 25.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg244-client.virtnet /mnt/lustre (opts:) Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 22:21:31 (1789266091) llite.lustre-ffff9bbb58afb000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.271549 s, 38.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg244-client.virtnet /mnt/lustre (opts:) Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9bbb532a1800.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 22:21:45 (1789266105) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 22:22:23 (1789266143) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 22:23:05 (1789266185) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 22:23:09 (1789266189) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.205254 s, 15.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.134311 s, 23.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xe2fd1015 (nosync,stale) Mirror 2: 0xe2fd1015 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 4874b58fc6f64a689f9012f690e140dc - mirror 1 checksum: 6cf40ca7e74be6d36f6b4c3f1b288f1d - mirror 2 checksum: 4874b58fc6f64a689f9012f690e140dc - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-12T22:23:11-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xfe:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-12T22:23:11-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xff:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 4874b58fc6f64a689f9012f690e140dc - mirror 1 checksum: 6cf40ca7e74be6d36f6b4c3f1b288f1d - mirror 2 checksum: 6cf40ca7e74be6d36f6b4c3f1b288f1d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-12T22:23:11-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xfe:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-12T22:23:11-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xff:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 22:23:21 (1789266201) Failing mds1 on oleg244-server Stopping /mnt/lustre-mds1 (opts:) on oleg244-server 22:23:28 (1789266208) shut down facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server Failover mds1 to oleg244-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg244-server: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-MDT0000 22:23:52 (1789266232) targets are mounted 22:23:52 (1789266232) facet_failover done oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473441 s, 22.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x100:0x0] } PASS 50A (47s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 22:24:08 (1789266248) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 22:24:12 (1789266252) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 22:24:16 (1789266256) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00889374 s, 1.8 MB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 22:24:26 (1789266266) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 22:24:30 (1789266270) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00505888 s, 810 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.169558 s, 24.2 kB/s PASS 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 22:24:40 (1789266280) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 22:25:02 (1789266302) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 22:25:32 (1789266332) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 22:25:56 (1789266356) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 22:26:19 (1789266379) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 22:26:26 (1789266386) mirror create pid 54329 mirror split pid 54330 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54329 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54330 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 22:27:34 (1789266454) Seed set to 6455 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero length punch hole skipping zero size read truncating to largest ever: 0x95ed 000008 1789266456.116428 trunc from 00000000 to 0x0095ec (0x95ed bytes) 000009 1789266456.127595 write 0x020bfa thru 0x02e25e (0xd665 bytes) 000010 1789266456.129327 read 0x020a78 thru 0x02e25e (0xd7e7 bytes) 000011 1789266456.130667 read 0x02219e thru 0x02e25e (0xc0c1 bytes) 000012 1789266456.131104 mapread 0x00d0f9 thru 0x016f34 (0x9e3c bytes) 000013 1789266456.139175 read 0x027b91 thru 0x0299a5 (0x1e15 bytes) 000014 1789266456.139767 punch from 0x01ac62 to 0x01f2f4 (0x4693 bytes) 000016 1789266456.152444 mapwrite 0x03c02e thru 0x03ffff (0x3fd2 bytes) 000018 1789266456.255375 MIRROR_EXTEND to 2 mirrors 000019 1789266456.884889 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) 000020 1789266456.987246 punch from 0x0360ad to 0x03ffff (0x9f53 bytes) 000021 1789266456.998198 punch from 0x01a800 to 0x020141 (0x5942 bytes) 000022 1789266457.003959 write 0x0329f1 thru 0x03574c (0x2d5c bytes) truncating to largest ever: 0x393dc 000024 1789266457.014530 trunc from 0x040000 to 0x0393db (0x6c24 bytes) 000025 1789266457.067849 punch from 0x00e9e2 to 0x00f373 (0x992 bytes) 000026 1789266457.086872 mapwrite 0x024b77 thru 0x026432 (0x18bc bytes) 000027 1789266457.152516 write 0x01df43 thru 0x029af6 (0xbbb4 bytes) 000028 1789266457.163727 MIRROR_EXTEND to 2 mirrors 000030 1789266458.204533 read 0x01958e thru 0x01c6c5 (0x3138 bytes) 000031 1789266458.226789 mapread 0x024cee thru 0x033818 (0xeb2b bytes) 000032 1789266458.321519 punch from 0x01fb2c to 0x028ffe (0x94d3 bytes) 000033 1789266458.471306 trunc from 0x0393dc to 0x001590 (0x37e4b bytes) 000034 1789266458.555076 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1591): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1591): Mirror 2: 0x9874ad74 (stale) CRC-32 checksum value for chunk [0, 0x1591): Mirror 1: 0x9874ad74 Mirror 3: 0x9874ad74 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1591: skipped 000037 1789266460.118698 read 0x000027 thru 0x001590 (0x156a bytes) 000039 1789266460.133274 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1591): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1591): Mirror 2: 0x9874ad74 (stale) CRC-32 checksum value for chunk [0, 0x1591): Mirror 1: 0x9874ad74 Mirror 3: 0x9874ad74 Mirror 4: 0x9874ad74 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1591: skipped 000040 1789266461.080898 write 0x025904 thru 0x0317c0 (0xbebd bytes) 000041 1789266461.166922 read 0x024c46 thru 0x0317c0 (0xcb7b bytes) 000043 1789266461.174868 punch from 0x02235c to 0x0236a3 (0x1348 bytes) 000044 1789266461.191610 trunc from 0x0317c1 to 0x005c9c (0x2bb24 bytes) 000045 1789266461.232432 read 0x0010a1 thru 0x005c9c (0x4bfc bytes) 000046 1789266461.248404 punch from 0x0039b1 to 0x005c9c (0x22ec bytes) 000047 1789266461.255510 punch from 0x005553 to 0x005c9c (0x74a bytes) 000048 1789266461.260242 punch from 0x00538c to 0x005c9c (0x911 bytes) 000050 1789266461.266853 punch from 0x00026e to 0x004a6d (0x4800 bytes) 000051 1789266461.273474 read 0x001949 thru 0x002bad (0x1265 bytes) 000052 1789266461.287910 write 0x01af3c thru 0x01da9a (0x2b5f bytes) 000053 1789266461.290246 mapread 0x002b55 thru 0x00d144 (0xa5f0 bytes) 000054 1789266461.298862 punch from 0x009b96 to 0x00b7b0 (0x1c1b bytes) 000056 1789266461.311680 mapread 0x017f48 thru 0x01cccb (0x4d84 bytes) 000057 1789266461.312436 punch from 0x0152ed to 0x01da9a (0x87ae bytes) 000062 1789266461.322072 read 0x00239e thru 0x011249 (0xeeac bytes) 000063 1789266461.328897 write 0x00e8e5 thru 0x010f8c (0x26a8 bytes) 000064 1789266461.329833 write 0x00470a thru 0x012987 (0xe27e bytes) 000068 1789266461.335113 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1da9b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1da9b): Mirror 2: 0x20d024a7 (stale) Mirror 3: 0x20d024a7 (stale) Mirror 4: 0x20d024a7 (stale) CRC-32 checksum value for chunk [0, 0x1da9b): Mirror 1: 0xe1b680c5 Mirror 5: 0xe1b680c5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1da9b: skipped 000070 1789266462.316404 MIRROR_RESYNC 5 mirrors 000071 1789266463.560363 mapwrite 0x008fa5 thru 0x00cde6 (0x3e42 bytes) 000072 1789266463.836544 punch from 0x0108e9 to 0x01b5a9 (0xacc1 bytes) 000077 1789266463.875312 read 0x00c848 thru 0x01296f (0x6128 bytes) 000078 1789266463.886707 mapread 0x017897 thru 0x01932f (0x1a99 bytes) 000079 1789266463.918142 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1da9b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1da9b): Mirror 2: 0xe1b680c5 (stale) Mirror 3: 0xe1b680c5 (stale) Mirror 4: 0xe1b680c5 (stale) Mirror 5: 0xe1b680c5 (stale) CRC-32 checksum value for chunk [0, 0x1da9b): Mirror 1: 0x2f42c32e Mirror 6: 0x2f42c32e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1da9b: skipped 000080 1789266465.908966 read 0x005e9b thru 0x00e825 (0x898b bytes) 000082 1789266465.972515 punch from 0x00867f to 0x011d07 (0x9689 bytes) 000083 1789266466.191493 MIRROR_RESYNC 6 mirrors 000084 1789266467.517102 punch from 0x00fb14 to 0x019579 (0x9a66 bytes) 000086 1789266467.600322 write 0x03631c thru 0x03f1fe (0x8ee3 bytes) 000087 1789266467.603600 write 0x007a1b thru 0x010589 (0x8b6f bytes) 000088 1789266467.621608 MIRROR_RESYNC 6 mirrors 000090 1789266469.219735 punch from 0x01a467 to 0x01b839 (0x13d3 bytes) 000091 1789266469.285640 write 0x02c845 thru 0x02e704 (0x1ec0 bytes) 000092 1789266469.293715 trunc from 0x03f1ff to 0x017b4f (0x276af bytes) 000093 1789266469.315755 write 0x00e899 thru 0x01610b (0x7873 bytes) 000095 1789266469.327507 mapwrite 0x0303ea thru 0x034a73 (0x468a bytes) 000096 1789266469.446383 MIRROR_RESYNC 6 mirrors 000097 1789266471.387145 trunc from 0x034a74 to 0x032e16 (0x1c5d bytes) 000098 1789266471.603302 trunc from 0x032e17 to 0x00f585 (0x23891 bytes) 000099 1789266471.640343 trunc from 0x00f586 to 0x006d8f (0x87f6 bytes) 000100 1789266471.665365 mapwrite 0x03b3b9 thru 0x03ffff (0x4c47 bytes) 000102 1789266471.839703 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9844eab8 (stale) Mirror 3: 0x9844eab8 (stale) Mirror 4: 0x9844eab8 (stale) Mirror 5: 0x9844eab8 (stale) Mirror 6: 0x9844eab8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7110612e Mirror 7: 0x7110612e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000106 1789266473.312451 mapread 0x0020cc thru 0x00c583 (0xa4b8 bytes) 000108 1789266473.361917 mapread 0x0164cc thru 0x02271f (0xc254 bytes) 000110 1789266473.411566 mapread 0x030319 thru 0x03f4ef (0xf1d7 bytes) truncating to largest ever: 0x3ca3b 000111 1789266473.451581 trunc from 0x040000 to 0x03ca3a (0x35c5 bytes) 000112 1789266473.554396 mapread 0x01f7f5 thru 0x02f170 (0xf97c bytes) 000113 1789266473.581660 trunc from 0x03ca3b to 0x00b249 (0x317f1 bytes) 000115 1789266473.608654 write 0x0332e0 thru 0x03ff36 (0xcc57 bytes) 000117 1789266473.616118 read 0x0119a7 thru 0x01d0de (0xb738 bytes) 000118 1789266473.633801 write 0x000ca9 thru 0x00135f (0x6b7 bytes) 000119 1789266473.648735 mapwrite 0x00a57a thru 0x014438 (0x9ebf bytes) 000121 1789266473.720051 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ff37): Mirror 3: 0x369af7db (stale) Mirror 4: 0x369af7db (stale) Mirror 5: 0x369af7db (stale) Mirror 6: 0x369af7db (stale) Mirror 7: 0xf5a9fe92 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ff37): Mirror 3: 0x369af7db (stale) Mirror 4: 0x369af7db (stale) Mirror 5: 0x369af7db (stale) Mirror 6: 0x369af7db (stale) Mirror 7: 0xf5a9fe92 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ff37: skipped 000122 1789266474.174932 MIRROR_RESYNC 6 mirrors 000124 1789266475.492750 punch from 0x01576f to 0x016055 (0x8e7 bytes) 000125 1789266475.569968 read 0x01c589 thru 0x024a94 (0x850c bytes) 000127 1789266475.613486 read 0x01e5b5 thru 0x02ae61 (0xc8ad bytes) 000128 1789266475.679994 write 0x019af4 thru 0x01c89a (0x2da7 bytes) 000129 1789266475.681308 trunc from 0x03ff37 to 0x02de7c (0x120ba bytes) 000130 1789266475.725667 mapwrite 0x00d9ec thru 0x01c1f8 (0xe80d bytes) 000131 1789266475.858986 read 0x01b145 thru 0x02276d (0x7629 bytes) 000132 1789266475.859701 trunc from 0x02de7d to 0x021ea6 (0xbfd6 bytes) 000133 1789266475.883347 write 0x028725 thru 0x031636 (0x8f12 bytes) 000134 1789266475.886299 trunc from 0x031637 to 0x007d17 (0x2991f bytes) 000135 1789266475.917749 mapwrite 0x00ac97 thru 0x00d10f (0x2479 bytes) 000136 1789266476.080016 read 0x004b93 thru 0x008eaf (0x431d bytes) 000137 1789266476.081374 read 0x003da1 thru 0x00d10f (0x936f bytes) 000138 1789266476.082786 write 0x0088e3 thru 0x00d6f7 (0x4e15 bytes) 000139 1789266476.086679 mapread 0x00cd63 thru 0x00d6f7 (0x995 bytes) 000141 1789266476.093869 write 0x0082aa thru 0x0161c3 (0xdf1a bytes) 000142 1789266476.096517 mapread 0x001973 thru 0x0075b7 (0x5c45 bytes) 000145 1789266476.117637 punch from 0x010506 to 0x0136e7 (0x31e2 bytes) 000148 1789266476.162027 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x161c4): Mirror 3: 0x2f5578d2 (stale) Mirror 4: 0x2f5578d2 (stale) Mirror 6: 0x2f5578d2 (stale) Mirror 7: 0x2f5578d2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x161c4): Mirror 3: 0x2f5578d2 (stale) Mirror 4: 0x2f5578d2 (stale) Mirror 6: 0x2f5578d2 (stale) Mirror 7: 0x2f5578d2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x161c4: skipped 000150 1789266476.630451 mapread 0x006bf7 thru 0x0158e8 (0xecf2 bytes) 000151 1789266476.750948 trunc from 0x0161c4 to 0x00768a (0xeb39 bytes) 000153 1789266476.838633 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x768b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x768b): Mirror 3: 0x13cfd262 (stale) Mirror 4: 0x13cfd262 (stale) Mirror 6: 0x13cfd262 (stale) Mirror 7: 0x13cfd262 (stale) CRC-32 checksum value for chunk [0, 0x768b): Mirror 1: 0x13cfd262 Mirror 8: 0x13cfd262 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x768b: skipped 000154 1789266478.452240 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x768b): Mirror 3: 0x13cfd262 (stale) Mirror 4: 0x13cfd262 (stale) Mirror 7: 0x13cfd262 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x768b): Mirror 3: 0x13cfd262 (stale) Mirror 4: 0x13cfd262 (stale) Mirror 7: 0x13cfd262 (stale) CRC-32 checksum value for chunk [0, 0x768b): Mirror 1: 0x13cfd262 Mirror 8: 0x13cfd262 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x768b: skipped 000155 1789266479.277957 mapwrite 0x00fc82 thru 0x01492b (0x4caa bytes) 000156 1789266479.744668 write 0x013a00 thru 0x0215e7 (0xdbe8 bytes) 000157 1789266479.756159 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x215e8): Mirror 3: 0x37e4b4e3 (stale) Mirror 7: 0x37e4b4e3 (stale) Mirror 8: 0xcce28e71 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x215e8): Mirror 3: 0x37e4b4e3 (stale) Mirror 7: 0x37e4b4e3 (stale) Mirror 8: 0xcce28e71 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x215e8: skipped 000158 1789266480.358392 mapwrite 0x030d0b thru 0x03edab (0xe0a1 bytes) 000159 1789266480.515495 punch from 0x03c6e3 to 0x03edab (0x26c9 bytes) 000160 1789266480.530507 trunc from 0x03edac to 0x018ed6 (0x25ed5 bytes) 000161 1789266480.579030 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x18ed7): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18ed7): Mirror 3: 0x48eeade3 (stale) Mirror 7: 0x48eeade3 (stale) Mirror 8: 0xaba7b988 (stale) CRC-32 checksum value for chunk [0, 0x18ed7): Mirror 1: 0x31e130a2 Mirror 9: 0x31e130a2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18ed7: skipped 000163 1789266482.090602 read 0x00cd86 thru 0x018ed6 (0xc151 bytes) 000164 1789266482.110723 trunc from 0x018ed7 to 0x02e7c7 (0x158f1 bytes) 000168 1789266482.212225 MIRROR_RESYNC 5 mirrors 000169 1789266483.607595 mapwrite 0x00258e thru 0x00c08b (0x9afe bytes) 000170 1789266483.811254 write 0x0049d4 thru 0x01470d (0xfd3a bytes) 000171 1789266483.820216 read 0x00d3af thru 0x00f4d3 (0x2125 bytes) 000172 1789266483.820961 write 0x009028 thru 0x0101a9 (0x7182 bytes) 000173 1789266483.824696 punch from 0x00a5e7 to 0x00b008 (0xa22 bytes) 000174 1789266483.847885 mapwrite 0x004734 thru 0x012535 (0xde02 bytes) 000175 1789266483.909516 MIRROR_RESYNC 5 mirrors 000176 1789266485.264927 trunc from 0x02e7c8 to 0x03c170 (0xd9a9 bytes) 000180 1789266485.346618 write 0x02db6c thru 0x03a3e6 (0xc87b bytes) truncating to largest ever: 0x3f760 000181 1789266485.357635 trunc from 0x03c171 to 0x03f75f (0x35ef bytes) 000182 1789266485.390347 write 0x025549 thru 0x027441 (0x1ef9 bytes) 000183 1789266485.402591 mapwrite 0x019f1d thru 0x01c134 (0x2218 bytes) 000185 1789266485.503970 write 0x01ebe7 thru 0x020bba (0x1fd4 bytes) 000186 1789266485.505388 trunc from 0x03f760 to 0x016e3e (0x28921 bytes) 000187 1789266485.528388 punch from 0x013c52 to 0x016e3e (0x31ed bytes) 000188 1789266485.539006 mapread 0x00f0a5 thru 0x0119fe (0x295a bytes) 000189 1789266485.547685 write 0x011597 thru 0x012e50 (0x18ba bytes) 000190 1789266485.548956 trunc from 0x016e3f to 0x0150fe (0x1d40 bytes) 000192 1789266485.570850 read 0x01468f thru 0x0150fe (0xa70 bytes) 000194 1789266485.571963 trunc from 0x0150ff to 0x03d8ae (0x287b0 bytes) 000195 1789266485.602133 trunc from 0x03d8af to 0x028712 (0x1519c bytes) 000196 1789266485.628985 punch from 0x00ce06 to 0x018f44 (0xc13f bytes) 000199 1789266485.645013 mapwrite 0x02beb0 thru 0x034542 (0x8693 bytes) 000200 1789266485.793637 read 0x00313c thru 0x006fa4 (0x3e69 bytes) 000202 1789266485.823404 mapread 0x0008fd thru 0x010878 (0xff7c bytes) 000203 1789266485.825993 mapread 0x02e00c thru 0x02e2e3 (0x2d8 bytes) 000204 1789266485.827544 read 0x001806 thru 0x008621 (0x6e1c bytes) 000206 1789266485.834992 mapwrite 0x0296ee thru 0x032eb7 (0x97ca bytes) 000208 1789266485.974190 mapwrite 0x0252a0 thru 0x02e6cb (0x942c bytes) 000210 1789266486.109205 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x34543): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34543): Mirror 3: 0xad6734eb (stale) Mirror 7: 0xad6734eb (stale) Mirror 8: 0xad6734eb (stale) Mirror 9: 0xad6734eb (stale) CRC-32 checksum value for chunk [0, 0x34543): Mirror 1: 0x6f1eb0dc Mirror 10: 0x6f1eb0dc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34543: skipped 000212 1789266487.546053 read 0x028346 thru 0x032b04 (0xa7bf bytes) 000216 1789266487.604224 mapread 0x026661 thru 0x02bd8b (0x572b bytes) 000217 1789266487.663258 mapwrite 0x0302a0 thru 0x03fe26 (0xfb87 bytes) 000218 1789266487.858155 read 0x01dd15 thru 0x0251b4 (0x74a0 bytes) 000219 1789266487.872691 punch from 0x01e80e to 0x021b57 (0x334a bytes) 000220 1789266487.880686 read 0x012e56 thru 0x018444 (0x55ef bytes) 000222 1789266487.882392 mapwrite 0x0234e0 thru 0x0244ee (0x100f bytes) 000224 1789266487.959674 write 0x000e40 thru 0x003649 (0x280a bytes) 000225 1789266487.960680 trunc from 0x03fe27 to 0x019a1f (0x26407 bytes) 000227 1789266488.002015 trunc from 0x019a20 to 0x003fd5 (0x15a4a bytes) 000230 1789266488.047047 trunc from 0x003fd6 to 0x0316c4 (0x2d6ef bytes) 000231 1789266488.103384 read 0x00eb84 thru 0x011855 (0x2cd2 bytes) 000233 1789266488.116315 punch from 0x008404 to 0x017b58 (0xf755 bytes) 000236 1789266488.143243 trunc from 0x0316c5 to 0x03a569 (0x8ea5 bytes) 000237 1789266488.172010 punch from 0x0339a9 to 0x03a569 (0x6bc1 bytes) 000239 1789266488.178480 mapwrite 0x020871 thru 0x0212af (0xa3f bytes) 000240 1789266488.256577 read 0x02f7b9 thru 0x03166a (0x1eb2 bytes) 000241 1789266488.263641 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a56a): Mirror 3: 0x88719976 (stale) Mirror 8: 0x88719976 (stale) Mirror 9: 0x88719976 (stale) Mirror 10: 0x2719bee6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a56a): Mirror 3: 0x88719976 (stale) Mirror 8: 0x88719976 (stale) Mirror 9: 0x88719976 (stale) Mirror 10: 0x2719bee6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a56a: skipped 000243 1789266488.712148 punch from 0x032ed0 to 0x0340b0 (0x11e1 bytes) 000245 1789266488.719257 write 0x0222a4 thru 0x02d414 (0xb171 bytes) 000247 1789266488.730443 write 0x017a25 thru 0x02312c (0xb708 bytes) 000248 1789266488.740900 read 0x0301da thru 0x034fb2 (0x4dd9 bytes) 000249 1789266488.750277 read 0x02dd1d thru 0x03a569 (0xc84d bytes) 000251 1789266488.765092 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a56a): Mirror 3: 0x88719976 (stale) Mirror 9: 0x88719976 (stale) Mirror 10: 0x2719bee6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a56a): Mirror 3: 0x88719976 (stale) Mirror 9: 0x88719976 (stale) Mirror 10: 0x2719bee6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a56a: skipped 000252 1789266489.181686 read 0x01c1f8 thru 0x0230a4 (0x6ead bytes) 000253 1789266489.193026 mapwrite 0x0262d3 thru 0x026622 (0x350 bytes) 000254 1789266489.283902 punch from 0x03290a to 0x036ed9 (0x45d0 bytes) 000255 1789266489.320905 trunc from 0x03a56a to 0x01860c (0x21f5d bytes) 000256 1789266489.368911 write 0x03d381 thru 0x03ffff (0x2c7f bytes) 000257 1789266489.370647 mapwrite 0x01df4d thru 0x02775c (0x9810 bytes) 000259 1789266489.552451 mapwrite 0x0366bf thru 0x03679d (0xdf bytes) 000260 1789266489.675841 read 0x007bdd thru 0x013f4b (0xc36f bytes) 000261 1789266489.687297 trunc from 0x040000 to 0x01d3f0 (0x22c0f bytes) 000262 1789266489.710878 mapwrite 0x0123b4 thru 0x020a42 (0xe68f bytes) 000265 1789266489.822711 read 0x00b9ba thru 0x00c727 (0xd6e bytes) 000266 1789266489.823303 trunc from 0x020a43 to 0x0390df (0x1869d bytes) 000269 1789266489.848912 read 0x034718 thru 0x0387b7 (0x40a0 bytes) 000272 1789266489.862360 write 0x022feb thru 0x02804c (0x5062 bytes) 000274 1789266489.864604 write 0x02d2b5 thru 0x02e417 (0x1163 bytes) 000277 1789266489.867797 trunc from 0x0390e0 to 0x033f1c (0x51c3 bytes) 000278 1789266489.892753 punch from 0x00a851 to 0x015876 (0xb026 bytes) 000279 1789266489.906887 mapwrite 0x019dc7 thru 0x02714b (0xd385 bytes) 000282 1789266489.958399 mapwrite 0x025254 thru 0x02bebc (0x6c69 bytes) 000284 1789266489.984355 read 0x02a25d thru 0x02d64d (0x33f1 bytes) 000285 1789266489.984862 mapwrite 0x023e07 thru 0x033a7f (0xfc79 bytes) 000286 1789266490.032392 write 0x00ab23 thru 0x012be0 (0x80be bytes) 000287 1789266490.051556 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33f1d): Mirror 9: 0x15fc32a8 (stale) Mirror 10: 0xf403dcc3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33f1d): Mirror 9: 0x15fc32a8 (stale) Mirror 10: 0xf403dcc3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33f1d: skipped 000290 1789266490.437851 write 0x01b305 thru 0x01c68b (0x1387 bytes) 000291 1789266490.452144 read 0x02bcc9 thru 0x032b92 (0x6eca bytes) 000293 1789266490.466253 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33f1d): Mirror 9: 0x15fc32a8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33f1d): Mirror 9: 0x15fc32a8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33f1d: skipped 000294 1789266490.875577 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x33f1d): Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33f1d): Mirror 9: 0x15fc32a8 (stale) CRC-32 checksum value for chunk [0, 0x33f1d): Mirror 1: 0xfeb97e58 Mirror 10: 0xfeb97e58 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33f1d: skipped 000295 1789266492.569301 mapwrite 0x0366a3 thru 0x03ffff (0x995d bytes) 000296 1789266492.752387 trunc from 0x040000 to 0x01559e (0x2aa61 bytes) 000298 1789266492.778726 trunc from 0x01559f to 0x00ccd4 (0x88ca bytes) 000299 1789266492.797519 mapwrite 0x00218d thru 0x00869a (0x650e bytes) 000304 1789266492.855139 read 0x004179 thru 0x00ccd4 (0x8b5c bytes) 000305 1789266492.855801 mapread 0x007142 thru 0x00ccd4 (0x5b93 bytes) 000309 1789266492.859663 read 0x007be3 thru 0x00ccd4 (0x50f2 bytes) 000311 1789266492.861600 mapwrite 0x02b817 thru 0x03277c (0x6f66 bytes) 000316 1789266492.941233 punch from 0x031826 to 0x03277c (0xf57 bytes) 000317 1789266492.947496 trunc from 0x03277d to 0x00ad10 (0x27a6c bytes) 000318 1789266492.968245 trunc from 0x00ad11 to 0x017816 (0xcb06 bytes) 000319 1789266492.996011 mapwrite 0x01570a thru 0x019027 (0x391e bytes) 000320 1789266493.110090 mapwrite 0x00d16f thru 0x01b07b (0xdf0d bytes) 000321 1789266493.333507 read 0x0174ad thru 0x01b07b (0x3bcf bytes) 000322 1789266493.342610 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b07c): Mirror 9: 0xe06e36a3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b07c): Mirror 9: 0xe06e36a3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b07c: skipped 000323 1789266493.608456 mapread 0x018e5a thru 0x01b07b (0x2222 bytes) 000324 1789266493.651619 mapread 0x0136c7 thru 0x015286 (0x1bc0 bytes) 000326 1789266493.653131 mapread 0x0011fd thru 0x007942 (0x6746 bytes) 000327 1789266493.660730 write 0x03b51c thru 0x03ffff (0x4ae4 bytes) 000329 1789266493.662076 mapread 0x00e205 thru 0x01db4f (0xf94b bytes) 000331 1789266493.681248 mapread 0x0273df thru 0x02e37b (0x6f9d bytes) 000332 1789266493.690185 mapwrite 0x001dff thru 0x00f3ae (0xd5b0 bytes) 000333 1789266493.750016 write 0x035aa1 thru 0x0397aa (0x3d0a bytes) 000335 1789266493.764207 mapread 0x01c85e thru 0x024527 (0x7cca bytes) 000336 1789266493.767760 punch from 0x006553 to 0x00cf17 (0x69c5 bytes) 000337 1789266493.781474 mapwrite 0x02d953 thru 0x02f55f (0x1c0d bytes) 000339 1789266493.827393 trunc from 0x040000 to 0x03d4a4 (0x2b5b bytes) 000340 1789266493.842816 punch from 0x023582 to 0x02b839 (0x82b8 bytes) 000341 1789266493.847366 trunc from 0x03d4a5 to 0x00aad4 (0x329d0 bytes) 000342 1789266493.868847 mapread 0x006c11 thru 0x00aad4 (0x3ec4 bytes) 000343 1789266493.879479 trunc from 0x00aad5 to 0x0356be (0x2abea bytes) 000344 1789266493.910384 mapread 0x033c47 thru 0x0356be (0x1a78 bytes) 000345 1789266493.924625 MIRROR_RESYNC 2 mirrors 000346 1789266494.565647 MIRROR_RESYNC 2 mirrors 000347 1789266494.921649 trunc from 0x0356bf to 0x0194d0 (0x1c1ee bytes) 000348 1789266494.999350 mapwrite 0x0182aa thru 0x020358 (0x80af bytes) 000350 1789266495.109576 MIRROR_RESYNC 2 mirrors 000351 1789266495.581213 write 0x03b5ac thru 0x03ffff (0x4a54 bytes) 000352 1789266495.627751 punch from 0x02d064 to 0x0381f9 (0xb196 bytes) 000355 1789266495.641086 write 0x01834c thru 0x021ecf (0x9b84 bytes) 000356 1789266495.650651 write 0x023987 thru 0x02969b (0x5d15 bytes) 000357 1789266495.658950 mapwrite 0x03c6e6 thru 0x03ffff (0x391a bytes) 000358 1789266495.703124 mapread 0x01e80d thru 0x022ad5 (0x42c9 bytes) 000359 1789266495.712310 MIRROR_RESYNC 2 mirrors 000360 1789266496.478919 trunc from 0x040000 to 0x01d1a8 (0x22e57 bytes) 000364 1789266496.593769 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) 000366 1789266496.789138 read 0x00adc5 thru 0x0134ea (0x8726 bytes) 000367 1789266496.810081 read 0x010950 thru 0x01aae5 (0xa196 bytes) 000368 1789266496.825083 read 0x0168cd thru 0x01d1a8 (0x68dc bytes) 000369 1789266496.826627 read 0x004cbd thru 0x00dd41 (0x9085 bytes) 000370 1789266496.828688 read 0x012e77 thru 0x01c155 (0x92df bytes) 000371 1789266496.829475 punch from 0x01c7c6 to 0x01d1a8 (0x9e3 bytes) 000373 1789266496.836096 write 0x0114c9 thru 0x016eeb (0x5a23 bytes) 000374 1789266496.837216 mapwrite 0x037281 thru 0x039e10 (0x2b90 bytes) 000377 1789266496.933439 trunc from 0x039e11 to 0x0278e3 (0x1252d bytes) 000378 1789266496.956218 read 0x01ca53 thru 0x0278e3 (0xae91 bytes) 000379 1789266496.963594 MIRROR_EXTEND to 2 mirrors 000380 1789266497.860880 write 0x02017c thru 0x026fa3 (0x6e28 bytes) 000381 1789266497.957208 write 0x03de2d thru 0x03ffff (0x21d3 bytes) 000382 1789266497.958527 read 0x0288fd thru 0x0342f3 (0xb9f7 bytes) 000383 1789266497.965449 punch from 0x014792 to 0x017eb8 (0x3727 bytes) 000384 1789266497.987510 write 0x012286 thru 0x0153fe (0x3179 bytes) 000385 1789266497.996908 punch from 0x034fb7 to 0x03683f (0x1889 bytes) 000386 1789266498.006810 trunc from 0x040000 to 0x03077f (0xf880 bytes) 000387 1789266498.032548 read 0x02ccc5 thru 0x03077f (0x3abb bytes) 000388 1789266498.033344 punch from 0x0254c2 to 0x02bc59 (0x6798 bytes) 000389 1789266498.038524 punch from 0x01c33f to 0x01ec8e (0x2950 bytes) 000391 1789266498.044091 trunc from 0x030780 to 0x03bde4 (0xb665 bytes) 000393 1789266498.091234 punch from 0x011dd6 to 0x0149fb (0x2c26 bytes) 000395 1789266498.104654 mapwrite 0x02aaf0 thru 0x02da7d (0x2f8e bytes) 000396 1789266498.195404 mapread 0x01a98d thru 0x01ef20 (0x4594 bytes) 000397 1789266498.197785 write 0x0239f1 thru 0x028d31 (0x5341 bytes) 000398 1789266498.198776 mapread 0x0295d2 thru 0x02c2d5 (0x2d04 bytes) 000399 1789266498.209659 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3bde5): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bde5): Mirror 2: 0xd18c682a (stale) CRC-32 checksum value for chunk [0, 0x3bde5): Mirror 1: 0x1eeb42b7 Mirror 3: 0x1eeb42b7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bde5: skipped 000400 1789266499.264787 mapread 0x0353c3 thru 0x036d5e (0x199c bytes) 000402 1789266499.277920 read 0x0340e8 thru 0x0396a8 (0x55c1 bytes) 000403 1789266499.292608 mapwrite 0x02938d thru 0x035f0a (0xcb7e bytes) 000405 1789266499.489147 write 0x005014 thru 0x0126d1 (0xd6be bytes) 000406 1789266499.504874 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3bde5): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bde5): Mirror 2: 0xd18c682a (stale) Mirror 3: 0x1eeb42b7 (stale) CRC-32 checksum value for chunk [0, 0x3bde5): Mirror 1: 0x3b7bda9 Mirror 4: 0x3b7bda9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bde5: skipped 000407 1789266500.501392 mapwrite 0x03fb0e thru 0x03ffff (0x4f2 bytes) 000410 1789266500.632684 mapread 0x01311a thru 0x01bb57 (0x8a3e bytes) 000411 1789266500.656386 mapwrite 0x00cadb thru 0x0165c8 (0x9aee bytes) 000412 1789266500.731819 mapwrite 0x001f79 thru 0x00652c (0x45b4 bytes) 000413 1789266500.812041 mapread 0x030e60 thru 0x03ffff (0xf1a0 bytes) 000414 1789266500.841324 trunc from 0x040000 to 0x00da98 (0x32567 bytes) 000415 1789266500.867748 mapread 0x00a9c2 thru 0x00b3ac (0x9eb bytes) 000416 1789266500.869493 mapread 0x002ed5 thru 0x007c7e (0x4daa bytes) 000418 1789266500.872748 mapwrite 0x037279 thru 0x03f982 (0x870a bytes) 000420 1789266500.947017 punch from 0x017f56 to 0x0263e9 (0xe494 bytes) 000422 1789266500.952569 read 0x00cdfa thru 0x016892 (0x9a99 bytes) 000423 1789266500.962062 punch from 0x017a21 to 0x0270a6 (0xf686 bytes) 000424 1789266500.965634 read 0x032b11 thru 0x03d1e9 (0xa6d9 bytes) 000426 1789266500.966364 mapread 0x02e9b0 thru 0x0392b8 (0xa909 bytes) 000428 1789266500.986611 punch from 0x024142 to 0x02facf (0xb98e bytes) 000432 1789266500.994484 mapread 0x014c07 thru 0x0168d0 (0x1cca bytes) 000434 1789266500.996356 punch from 0x030b64 to 0x03ebf6 (0xe093 bytes) 000436 1789266501.004403 mapwrite 0x026ad4 thru 0x02ca80 (0x5fad bytes) 000437 1789266501.100570 trunc from 0x03f983 to 0x00f64a (0x30338 bytes) 000439 1789266501.128779 mapwrite 0x00445a thru 0x00f03d (0xabe4 bytes) 000440 1789266501.242069 MIRROR_RESYNC 4 mirrors 000443 1789266502.964946 write 0x01b0d1 thru 0x02aea7 (0xfdd7 bytes) 000444 1789266503.047156 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2aea8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2aea8): Mirror 2: 0xd58eff24 (stale) Mirror 3: 0xd58eff24 (stale) Mirror 4: 0xd58eff24 (stale) CRC-32 checksum value for chunk [0, 0x2aea8): Mirror 1: 0xb21199aa Mirror 5: 0xb21199aa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2aea8: skipped 000445 1789266504.935596 read 0x00808e thru 0x0160c4 (0xe037 bytes) 000446 1789266504.989609 mapwrite 0x00cc49 thru 0x00f4cd (0x2885 bytes) 000448 1789266505.228838 write 0x031d7e thru 0x03a50a (0x878d bytes) 000449 1789266505.239079 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3a50b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a50b): Mirror 2: 0x5e3ae527 (stale) Mirror 3: 0x5e3ae527 (stale) Mirror 4: 0x5e3ae527 (stale) Mirror 5: 0xf85ec355 (stale) CRC-32 checksum value for chunk [0, 0x3a50b): Mirror 1: 0x22b5fb4b Mirror 6: 0x22b5fb4b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a50b: skipped 000450 1789266506.394361 punch from 0x01c4b9 to 0x028bce (0xc716 bytes) 000451 1789266506.477457 trunc from 0x03a50b to 0x01fcbc (0x1a84e bytes) 000452 1789266506.504598 read 0x00cd47 thru 0x01c99f (0xfc59 bytes) 000453 1789266506.526297 MIRROR_RESYNC 6 mirrors 000454 1789266507.520255 punch from 0x00dbc4 to 0x00fc3e (0x207b bytes) 000455 1789266507.594747 write 0x02340f thru 0x02855c (0x514e bytes) 000456 1789266507.597222 mapread 0x000944 thru 0x005873 (0x4f30 bytes) 000457 1789266507.629642 punch from 0x00ebb4 to 0x016f78 (0x83c5 bytes) 000459 1789266507.644124 mapwrite 0x026e3c thru 0x027a5d (0xc22 bytes) 000461 1789266507.694188 read 0x010874 thru 0x01c6b2 (0xbe3f bytes) 000462 1789266507.706893 punch from 0x00ac33 to 0x00b860 (0xc2e bytes) 000463 1789266507.717463 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2855d): Mirror 3: 0x2770d433 (stale) Mirror 4: 0x2770d433 (stale) Mirror 5: 0x2770d433 (stale) Mirror 6: 0x2770d433 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2855d): Mirror 3: 0x2770d433 (stale) Mirror 4: 0x2770d433 (stale) Mirror 5: 0x2770d433 (stale) Mirror 6: 0x2770d433 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2855d: skipped 000465 1789266508.221684 mapread 0x00ada7 thru 0x014a04 (0x9c5e bytes) 000466 1789266508.262941 trunc from 0x02855d to 0x006110 (0x2244c bytes) 000467 1789266508.280500 punch from 0x00244d to 0x002f44 (0xaf8 bytes) 000468 1789266508.285924 trunc from 0x006111 to 0x0107be (0xa6ae bytes) 000469 1789266508.302373 write 0x0359e5 thru 0x03a1c8 (0x47e4 bytes) 000470 1789266508.303861 mapwrite 0x03fc60 thru 0x03ffff (0x3a0 bytes) 000471 1789266508.358825 mapread 0x025f8b thru 0x030c3c (0xacb2 bytes) 000472 1789266508.376004 trunc from 0x040000 to 0x01330d (0x2ccf2 bytes) 000473 1789266508.405838 mapread 0x00b604 thru 0x00e01d (0x2a1a bytes) 000475 1789266508.417260 write 0x021df1 thru 0x029211 (0x7421 bytes) 000476 1789266508.423265 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29212): Mirror 3: 0x4d8c6711 (stale) Mirror 4: 0x4d8c6711 (stale) Mirror 5: 0x4d8c6711 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29212): Mirror 3: 0x4d8c6711 (stale) Mirror 4: 0x4d8c6711 (stale) Mirror 5: 0x4d8c6711 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29212: skipped 000478 1789266508.807204 mapread 0x021376 thru 0x029211 (0x7e9c bytes) 000479 1789266508.834326 write 0x017178 thru 0x020b54 (0x99dd bytes) 000480 1789266508.847218 read 0x018c3e thru 0x01a43b (0x17fe bytes) 000482 1789266508.852687 punch from 0x008175 to 0x00be6a (0x3cf6 bytes) 000483 1789266508.879629 write 0x00eb2f thru 0x01501a (0x64ec bytes) 000485 1789266508.903612 read 0x003c5a thru 0x0108f7 (0xcc9e bytes) 000486 1789266508.928584 read 0x018989 thru 0x021cd3 (0x934b bytes) 000488 1789266508.937614 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x29212): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29212): Mirror 3: 0x4d8c6711 (stale) Mirror 4: 0x4d8c6711 (stale) Mirror 5: 0x4d8c6711 (stale) CRC-32 checksum value for chunk [0, 0x29212): Mirror 1: 0xf3bc5d92 Mirror 6: 0xf3bc5d92 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29212: skipped 000489 1789266510.251029 punch from 0x002493 to 0x0108af (0xe41d bytes) 000492 1789266510.317078 read 0x019c86 thru 0x028796 (0xeb11 bytes) 000495 1789266510.329948 mapwrite 0x01d17f thru 0x02a8d5 (0xd757 bytes) 000496 1789266510.428211 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2a8d6): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a8d6): Mirror 3: 0x19062699 (stale) Mirror 4: 0x19062699 (stale) Mirror 5: 0x19062699 (stale) Mirror 6: 0x5998f7d8 (stale) CRC-32 checksum value for chunk [0, 0x2a8d6): Mirror 1: 0x786ddc88 Mirror 7: 0x786ddc88 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a8d6: skipped 000497 1789266511.389170 trunc from 0x02a8d6 to 0x0198fd (0x10fd8 bytes) 000498 1789266511.614664 punch from 0x0019f2 to 0x00af2a (0x9539 bytes) 000499 1789266511.619163 write 0x03ff41 thru 0x03ffff (0xbf bytes) 000500 1789266511.620222 mapwrite 0x006562 thru 0x00a989 (0x4428 bytes) 000502 1789266511.767553 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf14ab89d (stale) Mirror 4: 0xf14ab89d (stale) Mirror 5: 0xf14ab89d (stale) Mirror 6: 0xa1179a3d (stale) Mirror 7: 0xb4628917 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xead6bbac Mirror 8: 0xead6bbac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000503 1789266513.299871 mapwrite 0x005085 thru 0x011fc4 (0xcf40 bytes) 000504 1789266513.712202 MIRROR_RESYNC 7 mirrors 000505 1789266516.007394 mapwrite 0x032583 thru 0x03e085 (0xbb03 bytes) 000506 1789266516.293931 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xaf7915fc (stale) Mirror 4: 0xaf7915fc (stale) Mirror 5: 0xaf7915fc (stale) Mirror 6: 0xaf7915fc (stale) Mirror 7: 0xaf7915fc (stale) Mirror 8: 0xaf7915fc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd122ca14 Mirror 9: 0xd122ca14 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000508 1789266518.601790 punch from 0x003e69 to 0x007e56 (0x3fee bytes) 000510 1789266518.731659 mapwrite 0x02c816 thru 0x0331e3 (0x69ce bytes) 000511 1789266518.857729 read 0x02a134 thru 0x02fe38 (0x5d05 bytes) 000512 1789266518.880512 MIRROR_RESYNC 8 mirrors 000514 1789266520.824993 write 0x02ee04 thru 0x03bd3f (0xcf3c bytes) 000515 1789266521.015284 write 0x020b10 thru 0x0300ab (0xf59c bytes) 000516 1789266521.024261 punch from 0x027520 to 0x02ce9d (0x597e bytes) 000517 1789266521.055032 read 0x036d38 thru 0x03fe06 (0x90cf bytes) 000518 1789266521.082328 mapwrite 0x01676b thru 0x0232db (0xcb71 bytes) 000519 1789266521.351850 MIRROR_RESYNC 8 mirrors 000520 1789266524.007084 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) 000521 1789266524.613197 mapread 0x03a111 thru 0x03ffff (0x5eef bytes) 000523 1789266524.653158 trunc from 0x040000 to 0x01e9f3 (0x2160c bytes) 000524 1789266524.857587 read 0x016a95 thru 0x01e9f3 (0x7f5f bytes) 000525 1789266524.895221 write 0x02dca1 thru 0x03c98a (0xecea bytes) 000526 1789266524.898105 punch from 0x036c44 to 0x03c98a (0x5d47 bytes) 000527 1789266524.941344 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3c98b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c98b): Mirror 3: 0x3a6c9d0f (stale) Mirror 4: 0x3a6c9d0f (stale) Mirror 5: 0x3a6c9d0f (stale) Mirror 6: 0x3a6c9d0f (stale) Mirror 8: 0x3a6c9d0f (stale) Mirror 9: 0x3a6c9d0f (stale) CRC-32 checksum value for chunk [0, 0x3c98b): Mirror 1: 0x5a8fc7b3 Mirror 10: 0x5a8fc7b3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c98b: skipped 000528 1789266526.950978 MIRROR_RESYNC 8 mirrors 000529 1789266528.777527 MIRROR_RESYNC 8 mirrors 000531 1789266529.517521 punch from 0x0336c9 to 0x03c98a (0x92c2 bytes) 000534 1789266529.605868 MIRROR_RESYNC 8 mirrors 000537 1789266531.527205 read 0x038cd2 thru 0x03c98a (0x3cb9 bytes) 000538 1789266531.552580 mapwrite 0x0099d7 thru 0x016be9 (0xd213 bytes) 000539 1789266531.866093 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3c98b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c98b): Mirror 3: 0x446c30a7 (stale) Mirror 4: 0x446c30a7 (stale) Mirror 5: 0x446c30a7 (stale) Mirror 6: 0x446c30a7 (stale) Mirror 8: 0x446c30a7 (stale) Mirror 9: 0x446c30a7 (stale) Mirror 10: 0x446c30a7 (stale) CRC-32 checksum value for chunk [0, 0x3c98b): Mirror 1: 0x491deb2 Mirror 11: 0x491deb2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c98b: skipped 000540 1789266533.179835 mapread 0x00106e thru 0x00ad92 (0x9d25 bytes) 000541 1789266533.204453 MIRROR_RESYNC 9 mirrors 000543 1789266534.332086 punch from 0x03b939 to 0x03c98a (0x1052 bytes) 000544 1789266534.394635 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c98b): Mirror 4: 0x491deb2 (stale) Mirror 5: 0x491deb2 (stale) Mirror 6: 0x491deb2 (stale) Mirror 8: 0x491deb2 (stale) Mirror 9: 0x491deb2 (stale) Mirror 10: 0x491deb2 (stale) Mirror 11: 0x491deb2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c98b): Mirror 4: 0x491deb2 (stale) Mirror 5: 0x491deb2 (stale) Mirror 6: 0x491deb2 (stale) Mirror 8: 0x491deb2 (stale) Mirror 9: 0x491deb2 (stale) Mirror 10: 0x491deb2 (stale) Mirror 11: 0x491deb2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c98b: skipped 000547 1789266534.949112 punch from 0x039e2f to 0x03c98a (0x2b5c bytes) 000550 1789266534.958183 mapwrite 0x020f3b thru 0x027c87 (0x6d4d bytes) 000551 1789266535.042296 mapwrite 0x034b96 thru 0x03ffff (0xb46a bytes) 000552 1789266535.130288 mapread 0x03a063 thru 0x03cf61 (0x2eff bytes) 000553 1789266535.132210 mapread 0x03d047 thru 0x03ffff (0x2fb9 bytes) 000554 1789266535.133752 mapwrite 0x0328b7 thru 0x03ffff (0xd749 bytes) 000555 1789266535.181704 MIRROR_RESYNC 8 mirrors 000556 1789266537.477453 mapwrite 0x00fd71 thru 0x01476a (0x49fa bytes) 000557 1789266537.769196 mapwrite 0x03747e thru 0x03ffff (0x8b82 bytes) 000558 1789266537.838403 trunc from 0x040000 to 0x03a896 (0x5769 bytes) 000559 1789266537.871994 punch from 0x01461c to 0x01efe7 (0xa9cc bytes) 000560 1789266537.886173 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3a897): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a897): Mirror 4: 0x7687eb31 (stale) Mirror 5: 0x7687eb31 (stale) Mirror 6: 0x7687eb31 (stale) Mirror 8: 0x7687eb31 (stale) Mirror 9: 0x7687eb31 (stale) Mirror 10: 0x7687eb31 (stale) Mirror 11: 0x7687eb31 (stale) CRC-32 checksum value for chunk [0, 0x3a897): Mirror 1: 0x7ee7d4e7 Mirror 12: 0x7ee7d4e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a897: skipped 000561 1789266539.293252 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3a897): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a897): Mirror 4: 0x7687eb31 (stale) Mirror 5: 0x7687eb31 (stale) Mirror 6: 0x7687eb31 (stale) Mirror 8: 0x7687eb31 (stale) Mirror 9: 0x7687eb31 (stale) Mirror 10: 0x7687eb31 (stale) Mirror 11: 0x7687eb31 (stale) CRC-32 checksum value for chunk [0, 0x3a897): Mirror 1: 0x7ee7d4e7 Mirror 12: 0x7ee7d4e7 Mirror 13: 0x7ee7d4e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a897: skipped 000563 1789266540.478985 write 0x02efed thru 0x032ffc (0x4010 bytes) 000565 1789266540.570204 write 0x0039f6 thru 0x00886a (0x4e75 bytes) 000567 1789266540.585375 read 0x0143af thru 0x019fbf (0x5c11 bytes) 000568 1789266540.595827 read 0x01553a thru 0x0188ec (0x33b3 bytes) 000569 1789266540.596475 punch from 0x014d98 to 0x01c679 (0x78e2 bytes) 000570 1789266540.607150 mapread 0x027adb thru 0x02e1ac (0x66d2 bytes) 000571 1789266540.620054 punch from 0x022556 to 0x02a780 (0x822b bytes) 000574 1789266540.626804 write 0x02cb5c thru 0x02e60d (0x1ab2 bytes) 000575 1789266540.628338 mapread 0x01c93c thru 0x02b292 (0xe957 bytes) 000576 1789266540.682196 punch from 0x00db4e to 0x00e959 (0xe0c bytes) 000577 1789266540.691809 mapwrite 0x0162a7 thru 0x019d69 (0x3ac3 bytes) 000578 1789266540.741574 trunc from 0x03a897 to 0x006e56 (0x33a40 bytes) 000579 1789266540.767032 trunc from 0x006e57 to 0x02568c (0x1e836 bytes) 000580 1789266540.785094 mapwrite 0x02a74c thru 0x02d8af (0x3164 bytes) 000581 1789266540.889125 MIRROR_RESYNC 10 mirrors 000582 1789266543.042483 punch from 0x01f8e5 to 0x021e5f (0x257b bytes) 000583 1789266543.107074 mapread 0x0268e1 thru 0x02d8af (0x6fcf bytes) 000584 1789266543.191848 read 0x00d930 thru 0x01a5ca (0xcc9b bytes) 000587 1789266543.221644 read 0x00185e thru 0x008896 (0x7039 bytes) 000588 1789266543.250900 mapread 0x002ab4 thru 0x01236f (0xf8bc bytes) 000589 1789266543.252303 trunc from 0x02d8b0 to 0x01e72e (0xf181 bytes) 000590 1789266543.316300 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e72f): Mirror 4: 0x6681db8e (stale) Mirror 5: 0x6681db8e (stale) Mirror 6: 0x6681db8e (stale) Mirror 8: 0x6681db8e (stale) Mirror 9: 0x6681db8e (stale) Mirror 11: 0x6681db8e (stale) Mirror 12: 0x6681db8e (stale) Mirror 13: 0x6681db8e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e72f): Mirror 4: 0x6681db8e (stale) Mirror 5: 0x6681db8e (stale) Mirror 6: 0x6681db8e (stale) Mirror 8: 0x6681db8e (stale) Mirror 9: 0x6681db8e (stale) Mirror 11: 0x6681db8e (stale) Mirror 12: 0x6681db8e (stale) Mirror 13: 0x6681db8e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e72f: skipped 000591 1789266544.150825 trunc from 0x01e72f to 0x021afb (0x33cd bytes) 000593 1789266544.183598 mapread 0x0144ec thru 0x01d831 (0x9346 bytes) 000594 1789266544.227816 mapread 0x014e98 thru 0x021afb (0xcc64 bytes) 000595 1789266544.240096 punch from 0x018c0a to 0x021afb (0x8ef2 bytes) 000596 1789266544.251160 mapread 0x003522 thru 0x009936 (0x6415 bytes) 000598 1789266544.269230 read 0x012013 thru 0x01849c (0x648a bytes) 000599 1789266544.284045 punch from 0x00d0b5 to 0x01c1bb (0xf107 bytes) 000601 1789266544.295342 write 0x02aaab thru 0x02c4da (0x1a30 bytes) 000602 1789266544.296643 punch from 0x012a19 to 0x020412 (0xd9fa bytes) 000603 1789266544.335907 MIRROR_RESYNC 9 mirrors 000604 1789266547.130658 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) 000605 1789266547.688728 trunc from 0x02c4db to 0x00c04a (0x20490 bytes) 000606 1789266547.846640 punch from 0x0051b0 to 0x00c04a (0x6e9b bytes) 000608 1789266547.862785 write 0x02a62f thru 0x02e078 (0x3a4a bytes) 000610 1789266547.876125 trunc from 0x02e079 to 0x02ee9d (0xe25 bytes) 000611 1789266547.934403 mapwrite 0x010ffc thru 0x01788c (0x6891 bytes) 000613 1789266547.999011 mapread 0x014327 thru 0x0214da (0xd1b4 bytes) 000615 1789266548.021021 mapread 0x01563e thru 0x019a1e (0x43e1 bytes) 000617 1789266548.026878 read 0x0123d2 thru 0x018070 (0x5c9f bytes) 000618 1789266548.028773 mapread 0x00aeac thru 0x00c1bf (0x1314 bytes) 000620 1789266548.045721 read 0x0182b9 thru 0x023e23 (0xbb6b bytes) 000621 1789266548.046753 write 0x022456 thru 0x02ed76 (0xc921 bytes) 000622 1789266548.052881 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ee9e): Mirror 4: 0x87c1b8ce (stale) Mirror 5: 0x87c1b8ce (stale) Mirror 9: 0x87c1b8ce (stale) Mirror 11: 0x87c1b8ce (stale) Mirror 12: 0x87c1b8ce (stale) Mirror 13: 0x87c1b8ce (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ee9e): Mirror 4: 0x87c1b8ce (stale) Mirror 5: 0x87c1b8ce (stale) Mirror 9: 0x87c1b8ce (stale) Mirror 11: 0x87c1b8ce (stale) Mirror 12: 0x87c1b8ce (stale) Mirror 13: 0x87c1b8ce (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ee9e: skipped 000624 1789266548.916040 mapread 0x01bee9 thru 0x01fac1 (0x3bd9 bytes) 000625 1789266548.950260 write 0x032221 thru 0x03b605 (0x93e5 bytes) 000626 1789266548.960743 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b606): Mirror 4: 0x540c7303 (stale) Mirror 5: 0x540c7303 (stale) Mirror 9: 0x540c7303 (stale) Mirror 11: 0x540c7303 (stale) Mirror 13: 0x540c7303 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b606): Mirror 4: 0x540c7303 (stale) Mirror 5: 0x540c7303 (stale) Mirror 9: 0x540c7303 (stale) Mirror 11: 0x540c7303 (stale) Mirror 13: 0x540c7303 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b606: skipped 000627 1789266549.587738 punch from 0x01fa3a to 0x0217eb (0x1db2 bytes) 000628 1789266549.600779 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b606): Mirror 5: 0x540c7303 (stale) Mirror 9: 0x540c7303 (stale) Mirror 11: 0x540c7303 (stale) Mirror 13: 0x540c7303 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b606): Mirror 5: 0x540c7303 (stale) Mirror 9: 0x540c7303 (stale) Mirror 11: 0x540c7303 (stale) Mirror 13: 0x540c7303 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b606: skipped 000629 1789266550.362908 punch from 0x015fa9 to 0x025c82 (0xfcda bytes) 000630 1789266550.373103 punch from 0x00c7f4 to 0x01a9a2 (0xe1af bytes) 000632 1789266550.394319 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b606): Mirror 5: 0x540c7303 (stale) Mirror 9: 0x540c7303 (stale) Mirror 11: 0x540c7303 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b606): Mirror 5: 0x540c7303 (stale) Mirror 9: 0x540c7303 (stale) Mirror 11: 0x540c7303 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b606: skipped 000633 1789266551.140812 punch from 0x007028 to 0x00d50f (0x64e8 bytes) 000634 1789266551.156085 punch from 0x032066 to 0x034c71 (0x2c0c bytes) 000636 1789266551.176123 mapread 0x022374 thru 0x02df1c (0xbba9 bytes) 000637 1789266551.295876 trunc from 0x03b606 to 0x00787c (0x33d89 bytes) 000638 1789266551.334010 trunc from 0x00787d to 0x00d6aa (0x5e2e bytes) 000640 1789266551.382564 mapread 0x00a463 thru 0x00d6aa (0x3248 bytes) 000641 1789266551.403643 write 0x008a91 thru 0x0113e1 (0x8951 bytes) 000644 1789266551.421818 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x113e2): Mirror 5: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x113e2): Mirror 5: 0x3640da2d (stale) Mirror 9: 0x3640da2d (stale) Mirror 11: 0x3640da2d (stale) CRC-32 checksum value for chunk [0, 0x113e2): Mirror 1: 0xcfd894d5 Mirror 12: 0xcfd894d5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x113e2: skipped 000647 1789266553.039710 MIRROR_RESYNC 5 mirrors 000648 1789266554.586702 MIRROR_EXTEND to 6 mirrors 000650 1789266555.465949 read 0x002537 thru 0x00825a (0x5d24 bytes) 000651 1789266555.473497 mapwrite 0x0178bc thru 0x01d22a (0x596f bytes) 000652 1789266555.680990 read 0x0003c0 thru 0x004e10 (0x4a51 bytes) 000654 1789266555.694725 trunc from 0x01d22b to 0x00a963 (0x128c7 bytes) 000656 1789266555.720596 trunc from 0x00a964 to 0x0254d4 (0x1ab71 bytes) 000658 1789266555.738300 read 0x0243a8 thru 0x0254d4 (0x112d bytes) 000659 1789266555.751294 mapwrite 0x006371 thru 0x00c2f7 (0x5f87 bytes) 000660 1789266555.811230 MIRROR_RESYNC 6 mirrors 000661 1789266557.842212 read 0x016944 thru 0x01996f (0x302c bytes) 000663 1789266557.851046 mapwrite 0x01121b thru 0x01ebec (0xd9d2 bytes) 000664 1789266558.197092 trunc from 0x0254d5 to 0x018390 (0xd144 bytes) 000666 1789266558.236005 write 0x039c88 thru 0x03b0ed (0x1466 bytes) 000667 1789266558.241692 mapwrite 0x000570 thru 0x009678 (0x9109 bytes) 000668 1789266558.340793 read 0x039e12 thru 0x03b0ed (0x12dc bytes) 000670 1789266558.341964 read 0x014fff thru 0x0232be (0xe2c0 bytes) 000672 1789266558.376150 trunc from 0x03b0ee to 0x03d7fa (0x270d bytes) 000673 1789266558.403402 mapwrite 0x01c999 thru 0x025e52 (0x94ba bytes) 000675 1789266558.542175 MIRROR_RESYNC 6 mirrors 000676 1789266560.423770 mapwrite 0x03134d thru 0x03c301 (0xafb5 bytes) 000678 1789266560.782919 MIRROR_RESYNC 6 mirrors 000679 1789266563.124741 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) 000680 1789266563.624666 punch from 0x02c25f to 0x0326d8 (0x647a bytes) 000681 1789266563.765212 mapread 0x03c08f thru 0x03d7fa (0x176c bytes) 000682 1789266563.795598 mapwrite 0x024d01 thru 0x02ad38 (0x6038 bytes) 000684 1789266563.941877 mapread 0x01c544 thru 0x01cee3 (0x9a0 bytes) 000685 1789266563.959383 MIRROR_RESYNC 5 mirrors 000686 1789266565.043465 read 0x036f65 thru 0x03d7fa (0x6896 bytes) 000688 1789266565.050062 trunc from 0x03d7fb to 0x02d571 (0x10289 bytes) 000690 1789266565.110280 write 0x02beb9 thru 0x039ea5 (0xdfed bytes) 000691 1789266565.118745 read 0x038592 thru 0x039ea5 (0x1914 bytes) 000692 1789266565.119842 mapread 0x0160ad thru 0x01ecd8 (0x8c2c bytes) 000693 1789266565.130749 mapwrite 0x03d527 thru 0x03db51 (0x62b bytes) 000695 1789266565.198126 mapread 0x01519d thru 0x021ffa (0xce5e bytes) 000696 1789266565.199316 mapwrite 0x00fa64 thru 0x01e336 (0xe8d3 bytes) 000697 1789266565.258767 trunc from 0x03db52 to 0x0296db (0x14476 bytes) 000698 1789266565.315005 trunc from 0x0296dc to 0x006833 (0x22ea8 bytes) 000700 1789266565.348342 write 0x02cf03 thru 0x03467d (0x777b bytes) 000701 1789266565.354950 read 0x01f2f5 thru 0x02b8a2 (0xc5ae bytes) 000702 1789266565.384398 mapwrite 0x01eaf6 thru 0x029efb (0xb406 bytes) 000703 1789266565.552585 MIRROR_RESYNC 5 mirrors 000704 1789266567.135676 mapwrite 0x01b0ba thru 0x01e2b7 (0x31fe bytes) 000705 1789266567.361841 mapread 0x009167 thru 0x017206 (0xe0a0 bytes) 000708 1789266567.455112 read 0x0018ab thru 0x00ad41 (0x9497 bytes) 000709 1789266567.485528 punch from 0x01ce6f to 0x025ffd (0x918f bytes) 000710 1789266567.515839 trunc from 0x03467e to 0x0190ce (0x1b5af bytes) 000712 1789266567.562873 mapwrite 0x02ce75 thru 0x039ee7 (0xd073 bytes) 000713 1789266567.771257 punch from 0x028a57 to 0x033832 (0xaddc bytes) 000714 1789266567.788163 read 0x026f7c thru 0x0367db (0xf860 bytes) 000716 1789266567.808626 read 0x0135d7 thru 0x019efb (0x6925 bytes) 000717 1789266567.809917 mapread 0x02dba5 thru 0x02edc5 (0x1221 bytes) 000718 1789266567.817842 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39ee8): Mirror 5: 0x3ffe539b (stale) Mirror 9: 0x3ffe539b (stale) Mirror 13: 0x3ffe539b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39ee8): Mirror 5: 0x3ffe539b (stale) Mirror 9: 0x3ffe539b (stale) Mirror 13: 0x3ffe539b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39ee8: skipped 000719 1789266568.292265 mapread 0x02ba9b thru 0x039ee7 (0xe44d bytes) 000721 1789266568.322665 read 0x008b01 thru 0x010086 (0x7586 bytes) 000723 1789266568.333924 trunc from 0x039ee8 to 0x002f26 (0x36fc1 bytes) 000724 1789266568.361548 MIRROR_RESYNC 4 mirrors 000725 1789266569.154951 mapwrite 0x039b16 thru 0x03ffff (0x64ea bytes) 000726 1789266569.354034 punch from 0x0213b6 to 0x027836 (0x6481 bytes) 000727 1789266569.364081 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 9: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x68403778 (stale) Mirror 9: 0x68403778 (stale) Mirror 13: 0x68403778 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5617265e Mirror 14: 0x5617265e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000729 1789266570.596830 mapread 0x021f8c thru 0x02b818 (0x988d bytes) 000732 1789266570.637359 punch from 0x01e7cc to 0x02816a (0x999f bytes) 000733 1789266570.703385 read 0x02ab36 thru 0x03055c (0x5a27 bytes) 000734 1789266570.712202 write 0x00e545 thru 0x019d86 (0xb842 bytes) 000736 1789266570.723194 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x68403778 (stale) Mirror 9: 0x68403778 (stale) Mirror 13: 0x68403778 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x68403778 (stale) Mirror 9: 0x68403778 (stale) Mirror 13: 0x68403778 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000737 1789266571.063581 write 0x012d04 thru 0x01e05d (0xb35a bytes) 000740 1789266571.086179 mapwrite 0x02f9c5 thru 0x032f20 (0x355c bytes) 000741 1789266571.176063 mapwrite 0x01d1b1 thru 0x01f43b (0x228b bytes) 000742 1789266571.260284 mapread 0x003fb8 thru 0x00e133 (0xa17c bytes) 000745 1789266571.269645 mapwrite 0x022b7d thru 0x02416c (0x15f0 bytes) 000746 1789266571.318527 punch from 0x03ff3c to 0x03ffff (0xc4 bytes) 000747 1789266571.326054 mapwrite 0x014a7b thru 0x01ed43 (0xa2c9 bytes) 000748 1789266571.513321 read 0x012765 thru 0x0153f2 (0x2c8e bytes) 000751 1789266571.515053 read 0x010197 thru 0x0168cd (0x6737 bytes) 000752 1789266571.547397 mapread 0x000743 thru 0x00b200 (0xaabe bytes) 000753 1789266571.548568 write 0x01764b thru 0x0237e1 (0xc197 bytes) 000755 1789266571.581828 punch from 0x03df47 to 0x03ffff (0x20b9 bytes) 000759 1789266571.613531 trunc from 0x040000 to 0x015cb4 (0x2a34b bytes) 000764 1789266571.713640 read 0x0065e6 thru 0x015cb4 (0xf6cf bytes) 000765 1789266571.717020 read 0x001d1f thru 0x0068ba (0x4b9c bytes) 000766 1789266571.722742 read 0x00c5d6 thru 0x015cb4 (0x96df bytes) 000767 1789266571.724468 write 0x039d3d thru 0x03ffff (0x62c3 bytes) 000768 1789266571.741078 mapwrite 0x017243 thru 0x026eaa (0xfc68 bytes) 000769 1789266571.976154 mapwrite 0x02ef77 thru 0x038314 (0x939e bytes) 000772 1789266572.086369 read 0x0198cd thru 0x01e9fe (0x5132 bytes) 000775 1789266572.088283 punch from 0x0206d0 to 0x020ea9 (0x7da bytes) 000776 1789266572.093586 read 0x033f83 thru 0x03fcbf (0xbd3d bytes) 000781 1789266572.102612 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x68403778 (stale) Mirror 13: 0x68403778 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x68403778 (stale) Mirror 13: 0x68403778 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000783 1789266572.487317 trunc from 0x040000 to 0x01b444 (0x24bbb bytes) 000785 1789266572.511101 MIRROR_RESYNC 3 mirrors 000786 1789266574.274922 write 0x01f351 thru 0x028a94 (0x9744 bytes) 000789 1789266574.345780 trunc from 0x028a95 to 0x028832 (0x262 bytes) 000790 1789266574.384174 punch from 0x01f6c2 to 0x0265e5 (0x6f24 bytes) 000791 1789266574.405098 trunc from 0x028833 to 0x02a857 (0x2025 bytes) 000792 1789266574.428346 write 0x01824a thru 0x01a4ec (0x22a3 bytes) 000793 1789266574.448359 punch from 0x01adb5 to 0x01b088 (0x2d4 bytes) 000794 1789266574.468924 trunc from 0x02a858 to 0x028c54 (0x1c03 bytes) 000795 1789266574.499792 trunc from 0x028c55 to 0x029f73 (0x131f bytes) 000796 1789266574.530022 write 0x01715c thru 0x017c02 (0xaa7 bytes) 000798 1789266574.544819 MIRROR_RESYNC 3 mirrors 000799 1789266575.444940 mapread 0x00301f thru 0x01204f (0xf031 bytes) 000801 1789266575.480883 punch from 0x00fc70 to 0x012ec6 (0x3257 bytes) 000802 1789266575.571555 read 0x00abe7 thru 0x00d30f (0x2729 bytes) 000804 1789266575.580577 punch from 0x01ef08 to 0x02475d (0x5856 bytes) 000805 1789266575.591804 mapread 0x024b43 thru 0x027dfd (0x32bb bytes) 000806 1789266575.609616 write 0x000667 thru 0x00a38f (0x9d29 bytes) 000807 1789266575.618842 read 0x0257c1 thru 0x029f73 (0x47b3 bytes) 000808 1789266575.634697 write 0x039821 thru 0x03b423 (0x1c03 bytes) 000811 1789266575.647335 mapwrite 0x03eb00 thru 0x03ffff (0x1500 bytes) 000812 1789266575.828166 mapread 0x03336c thru 0x03f963 (0xc5f8 bytes) 000813 1789266575.832808 trunc from 0x040000 to 0x01a90a (0x256f5 bytes) 000814 1789266575.910591 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1a90b): Mirror 9: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a90b): Mirror 9: 0x42be2c2f (stale) Mirror 13: 0x42be2c2f (stale) CRC-32 checksum value for chunk [0, 0x1a90b): Mirror 1: 0x47f9baa2 Mirror 14: 0x47f9baa2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a90b: skipped 000815 1789266577.344591 mapread 0x0106f6 thru 0x01a90a (0xa215 bytes) 000818 1789266577.522073 mapread 0x015b2c thru 0x0191dd (0x36b2 bytes) 000819 1789266577.572159 read 0x007b88 thru 0x00bfe1 (0x445a bytes) 000820 1789266577.614163 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a90b): Mirror 9: 0x42be2c2f (stale) Mirror 13: 0x42be2c2f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a90b): Mirror 9: 0x42be2c2f (stale) Mirror 13: 0x42be2c2f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a90b: skipped 000822 1789266578.311564 read 0x001dd3 thru 0x00e9ea (0xcc18 bytes) 000823 1789266578.328567 MIRROR_RESYNC 3 mirrors 000824 1789266579.666034 punch from 0x00728e to 0x009c58 (0x29cb bytes) 000825 1789266579.808662 mapwrite 0x0126e8 thru 0x014141 (0x1a5a bytes) 000826 1789266579.917168 punch from 0x00401e to 0x00f705 (0xb6e8 bytes) 000827 1789266579.926944 MIRROR_RESYNC 3 mirrors 000828 1789266581.208805 trunc from 0x01a90b to 0x030ead (0x165a3 bytes) 000829 1789266581.425560 read 0x00b2a2 thru 0x0103f8 (0x5157 bytes) 000830 1789266581.448559 trunc from 0x030eae to 0x00b489 (0x25a24 bytes) 000831 1789266581.474475 trunc from 0x00b48a to 0x020d9b (0x15912 bytes) 000832 1789266581.499838 mapwrite 0x03101f thru 0x03712f (0x6111 bytes) 000834 1789266581.597315 read 0x023ae0 thru 0x02c021 (0x8542 bytes) 000835 1789266581.615969 mapwrite 0x008779 thru 0x01140f (0x8c97 bytes) 000836 1789266581.658983 write 0x0279df thru 0x034bc5 (0xd1e7 bytes) 000837 1789266581.660531 read 0x01f687 thru 0x023c67 (0x45e1 bytes) 000838 1789266581.666492 MIRROR_RESYNC 3 mirrors 000841 1789266582.745095 trunc from 0x037130 to 0x0195c3 (0x1db6c bytes) 000844 1789266582.914783 read 0x0074f2 thru 0x008936 (0x1445 bytes) 000845 1789266582.929592 mapread 0x011809 thru 0x0195c3 (0x7dbb bytes) 000847 1789266582.960536 mapwrite 0x0007f3 thru 0x006ca0 (0x64ae bytes) 000848 1789266583.180899 trunc from 0x0195c4 to 0x020729 (0x7166 bytes) 000849 1789266583.214473 trunc from 0x02072a to 0x01e68c (0x209d bytes) 000850 1789266583.266695 mapread 0x01c062 thru 0x01c639 (0x5d8 bytes) 000852 1789266583.289534 mapread 0x01e0b8 thru 0x01e68c (0x5d5 bytes) 000853 1789266583.295063 MIRROR_RESYNC 3 mirrors 000854 1789266584.180650 punch from 0x00e1b9 to 0x01c816 (0xe65e bytes) 000856 1789266584.226811 punch from 0x016128 to 0x01e68c (0x8565 bytes) 000857 1789266584.232263 read 0x01bd5e thru 0x01e68c (0x292f bytes) 000858 1789266584.240467 read 0x00d9ce thru 0x00fde7 (0x241a bytes) 000860 1789266584.251320 mapread 0x00eb4f thru 0x0158f9 (0x6dab bytes) 000861 1789266584.253113 read 0x009dd9 thru 0x018fc0 (0xf1e8 bytes) 000863 1789266584.253982 read 0x000fc5 thru 0x00b392 (0xa3ce bytes) 000864 1789266584.254703 read 0x0195cd thru 0x01e68c (0x50c0 bytes) 000866 1789266584.256030 mapwrite 0x00ce50 thru 0x01522c (0x83dd bytes) 000868 1789266584.317012 trunc from 0x01e68d to 0x03ee6b (0x207df bytes) 000869 1789266584.347980 write 0x03dcad thru 0x03ffff (0x2353 bytes) 000870 1789266584.360623 mapread 0x00c3c2 thru 0x0189e6 (0xc625 bytes) 000872 1789266584.365616 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x75354110 (stale) Mirror 13: 0x75354110 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x74e39bff Mirror 14: 0x74e39bff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000873 1789266585.921127 mapread 0x001542 thru 0x00f1bf (0xdc7e bytes) 000875 1789266586.043726 read 0x00193f thru 0x0093f4 (0x7ab6 bytes) 000876 1789266586.051938 punch from 0x01697a to 0x01f158 (0x87df bytes) 000877 1789266586.161701 mapwrite 0x02860f thru 0x02ca2e (0x4420 bytes) 000879 1789266586.219723 mapread 0x01b403 thru 0x0287cd (0xd3cb bytes) 000880 1789266586.234788 mapwrite 0x0134cb thru 0x01c4c7 (0x8ffd bytes) 000882 1789266586.326101 mapwrite 0x03b812 thru 0x03ffff (0x47ee bytes) 000883 1789266586.405963 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x75354110 (stale) Mirror 14: 0x74e39bff (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x75354110 (stale) Mirror 14: 0x74e39bff (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000885 1789266587.014346 punch from 0x0028a3 to 0x00bc4c (0x93aa bytes) 000886 1789266587.030388 mapwrite 0x039895 thru 0x03ffff (0x676b bytes) 000888 1789266587.189086 trunc from 0x040000 to 0x0109fa (0x2f605 bytes) 000891 1789266587.249539 mapwrite 0x009322 thru 0x00bb5a (0x2839 bytes) 000893 1789266587.332666 write 0x0317e3 thru 0x032013 (0x831 bytes) 000894 1789266587.335609 write 0x00f747 thru 0x011e4f (0x2709 bytes) 000895 1789266587.352674 mapwrite 0x00c0ac thru 0x0155bc (0x9511 bytes) 000896 1789266587.439532 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x32014): Mirror 9: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32014): Mirror 9: 0x7d61e5fd (stale) Mirror 14: 0xdda98486 (stale) CRC-32 checksum value for chunk [0, 0x32014): Mirror 1: 0xaf405faa Mirror 15: 0xaf405faa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32014: skipped 000898 1789266588.814227 mapwrite 0x00da31 thru 0x019898 (0xbe68 bytes) 000899 1789266589.145198 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32014): Mirror 9: 0x7d61e5fd (stale) Mirror 14: 0xdda98486 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32014): Mirror 9: 0x7d61e5fd (stale) Mirror 14: 0xdda98486 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32014: skipped 000900 1789266589.733254 mapread 0x0071bf thru 0x012724 (0xb566 bytes) 000901 1789266589.773737 write 0x02611a thru 0x032a37 (0xc91e bytes) 000902 1789266589.786541 read 0x0308ab thru 0x032a37 (0x218d bytes) 000904 1789266589.792123 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32a38): Mirror 9: 0x9acb5fab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32a38): Mirror 9: 0x9acb5fab (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32a38: skipped 000905 1789266590.172228 punch from 0x00c179 to 0x0121db (0x6063 bytes) 000906 1789266590.178561 trunc from 0x032a38 to 0x00b3c1 (0x27676 bytes) 000907 1789266590.244617 read 0x00aeb9 thru 0x00b3c1 (0x509 bytes) 000908 1789266590.253650 write 0x036441 thru 0x03eede (0x8a9e bytes) 000909 1789266590.256736 trunc from 0x03eedf to 0x00227f (0x3cc5f bytes) 000910 1789266590.282265 punch from 0x000c8c to 0x00227f (0x15f4 bytes) 000911 1789266590.287992 mapwrite 0x010c6e thru 0x01bf1d (0xb2b0 bytes) 000912 1789266590.396579 read 0x0064a8 thru 0x00c783 (0x62dc bytes) 000913 1789266590.410746 mapread 0x01575c thru 0x0195ae (0x3e53 bytes) 000914 1789266590.412044 write 0x000c4e thru 0x00bdbc (0xb16f bytes) 000915 1789266590.414384 write 0x03f0a7 thru 0x03ffff (0xf59 bytes) 000916 1789266590.415747 mapwrite 0x01fd18 thru 0x029d9a (0xa083 bytes) 000918 1789266590.491654 read 0x03e3e2 thru 0x03ffff (0x1c1e bytes) 000920 1789266590.493467 read 0x013254 thru 0x01f467 (0xc214 bytes) 000921 1789266590.495137 mapwrite 0x027573 thru 0x02855f (0xfed bytes) 000922 1789266590.564986 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) 000923 1789266590.913534 mapread 0x003ce7 thru 0x00920f (0x5529 bytes) 000925 1789266590.957149 read 0x035b76 thru 0x03f2aa (0x9735 bytes) 000927 1789266591.021364 MIRROR_EXTEND to 2 mirrors 000928 1789266592.095701 MIRROR_RESYNC 2 mirrors 000929 1789266592.620571 punch from 0x03f184 to 0x03ffff (0xe7c bytes) 000930 1789266592.745856 punch from 0x021cdd to 0x02d81d (0xbb41 bytes) 000931 1789266592.754521 write 0x001f1d thru 0x005398 (0x347c bytes) 000932 1789266592.771618 mapwrite 0x03f844 thru 0x03ffff (0x7bc bytes) 000933 1789266592.848397 write 0x030896 thru 0x0324af (0x1c1a bytes) 000934 1789266592.859491 punch from 0x014f08 to 0x0233df (0xe4d8 bytes) 000936 1789266592.877988 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) 000938 1789266593.168781 write 0x008609 thru 0x010bcb (0x85c3 bytes) 000939 1789266593.190741 mapread 0x038e52 thru 0x03f79f (0x694e bytes) 000941 1789266593.214652 read 0x010ef7 thru 0x01fd5c (0xee66 bytes) 000942 1789266593.226124 trunc from 0x040000 to 0x02847d (0x17b82 bytes) 000943 1789266593.242445 write 0x02c116 thru 0x037a93 (0xb97e bytes) 000947 1789266593.250354 trunc from 0x037a94 to 0x017de0 (0x1fcb3 bytes) 000948 1789266593.276217 read 0x016d80 thru 0x017de0 (0x1061 bytes) 000949 1789266593.276811 mapwrite 0x02c674 thru 0x0324a5 (0x5e32 bytes) 000950 1789266593.420461 trunc from 0x0324a6 to 0x01c34c (0x16159 bytes) 000953 1789266593.452509 mapwrite 0x0192fd thru 0x020837 (0x753b bytes) 000955 1789266593.653053 read 0x0145b4 thru 0x01b63c (0x7089 bytes) 000960 1789266593.671516 MIRROR_EXTEND to 2 mirrors 000962 1789266594.840543 write 0x03dbbe thru 0x03ffff (0x2442 bytes) 000964 1789266594.947922 punch from 0x00d14f to 0x0107de (0x3690 bytes) 000966 1789266594.960824 write 0x00beca thru 0x00d02f (0x1166 bytes) 000967 1789266594.975316 read 0x0016a3 thru 0x00759e (0x5efc bytes) 000969 1789266594.987093 read 0x038d30 thru 0x03ffff (0x72d0 bytes) 000971 1789266595.019113 punch from 0x03b8e3 to 0x03ffff (0x471d bytes) 000972 1789266595.054699 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x42fd8175 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x29376fa1 Mirror 3: 0x29376fa1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000975 1789266596.316131 write 0x0055ff thru 0x00f7ba (0xa1bc bytes) 000978 1789266596.399221 trunc from 0x040000 to 0x006f32 (0x390cd bytes) 000979 1789266596.415044 trunc from 0x006f33 to 0x006afc (0x436 bytes) 000980 1789266596.426493 write 0x02a318 thru 0x02f5f7 (0x52e0 bytes) 000984 1789266596.436041 MIRROR_RESYNC 3 mirrors 000985 1789266597.540402 write 0x02c6d0 thru 0x035f2d (0x985e bytes) 000986 1789266597.600021 punch from 0x02ff96 to 0x035f2d (0x5f98 bytes) 000988 1789266597.614355 punch from 0x01ece8 to 0x0209f7 (0x1d10 bytes) 000990 1789266597.618376 write 0x03fa30 thru 0x03ffff (0x5d0 bytes) 000992 1789266597.620318 mapwrite 0x028e9e thru 0x0385fb (0xf75e bytes) 000993 1789266597.695127 read 0x024e65 thru 0x029aa8 (0x4c44 bytes) 000994 1789266597.696346 write 0x0063cc thru 0x0159c4 (0xf5f9 bytes) 000995 1789266597.711530 mapwrite 0x023842 thru 0x027058 (0x3817 bytes) 000997 1789266597.780902 mapwrite 0x03946e thru 0x03ffff (0x6b92 bytes) 000998 1789266597.851775 write 0x0375ab thru 0x03ffff (0x8a55 bytes) 000999 1789266597.855561 mapread 0x002cf2 thru 0x006e8d (0x419c bytes) 001000 1789266597.875291 read 0x00c7cf thru 0x014db6 (0x85e8 bytes) All operations completed A-OK! PASS 70a (150s) == sanity-flr test 71: check mirror extend parent fid ==== 22:30:03 (1789266603) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 22:30:07 (1789266607) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ed:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.198382 s, 10.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ed:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14 s, 7.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ee:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ed:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ee:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ed:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cb:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 22:30:16 (1789266616) Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre2 Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5786336 .. Extending file size to 5469696 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6858272 .. Extending file size to 7308288 .. Extending file size to 7753984 .. Extending file size to 5963392 .. Extending file size to 8690272 .. Extending file size to 7985408 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8207328 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9066656 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 8240640 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8553056 .. Waiting 57468 57469 57470 57471 57472 done 192.168.202.144@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg244-client.virtnet /mnt/lustre2 (opts:) 192.168.202.144@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg244-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 22:31:37 (1789266697) Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre2 Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4721856 .. Extending file size to 1503232 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5251424 .. Extending file size to 6954240 .. Extending file size to 7232352 .. Extending file size to 7834944 .. Extending file size to 9098976 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8709952 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 58817 58818 58821 58822 58823 failed 192.168.202.144@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg244-client.virtnet /mnt/lustre2 (opts:) 192.168.202.144@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg244-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 22:32:56 (1789266776) Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.128343 s, 16.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.347252 s, 6.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x88846d6e (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.144@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg244-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 22:33:19 (1789266799) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.241437 s, 8.7 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 22:33:28 (1789266808) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f3:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x304:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x304:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x305:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 22:33:36 (1789266816) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.265483 s, 7.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x306:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 22:33:47 (1789266827) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.261257 s, 8.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x307:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x307:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } PASS 204b (11s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 22:33:58 (1789266838) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.222016 s, 9.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x309:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x309:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 22:34:33 (1789266873) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg244-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272146 s, 3.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30b:0x0] } PASS 204d (63s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 22:35:37 (1789266937) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg244-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.309716 s, 6.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (63s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 22:36:40 (1789267000) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg244-server mds-ost sync done. Creating new pool test_204f oleg244-server: Pool lustre.test_204f created Adding targets to pool oleg244-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg244-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.255531 s, 8.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg244-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg244-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg244-server: Pool lustre.test_204f destroyed PASS 204f (76s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 22:37:56 (1789267076) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 22:38:04 (1789267084) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 22:38:11 (1789267091) oleg244-server: Pool lustre.test_206 created Adding targets to pool oleg244-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg244-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg244-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg244-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg244-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 22:38:36 (1789267116) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702175 s, 14.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342583 s, 30.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x11ff3126 Mirror 3: 0x11ff3126 PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 22:38:46 (1789267126) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 22:38:50 (1789267130) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 22:38:56 (1789267136) Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.160762 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1a50000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x7b0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x7b0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x7b0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x6a0000 Read this (4 bytes): BBBB 192.168.202.144@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg244-client.virtnet /mnt/lustre2 (opts:) PASS 209a (45s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 22:39:41 (1789267181) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.015358 s, 533 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 22:39:48 (1789267188) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310524 s, 33.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x218 lmm_fid: [0x200000404:0x218:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x218 lmm_fid: [0x200000404:0x218:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x318:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834456 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-12 22:39:49.000000000 -0400 Modify: 2026-09-12 22:39:49.000000000 -0400 Change: 2026-09-12 22:39:51.000000000 -0400 Birth: 2026-09-12 22:39:48.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 22:39:56 (1789267196) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 22:40:00 (1789267200) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.68622 s, 59.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0939492 s, 43.6 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 12 22:40 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21a:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 12 22:40 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 22:40:10 (1789267210) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 22:40:19 (1789267219) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1883 sec =========== 22:40:28 (1789267228) === sanity-flr: start cleanup 22:40:30 (1789267230) === === sanity-flr: finish cleanup 22:40:37 (1789267237) ===