-----============= acceptance-small: sanity-flr ============----- Sun Aug 9 15:41:41 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg110-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 15:41:59 (1786304519) === oleg110-client.virtnet: executing check_config_client /mnt/lustre oleg110-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg110-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff908843964000.idle_timeout=debug osc.lustre-OST0001-osc-ffff908843964000.idle_timeout=debug disable quota as required oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 15:42:36 (1786304556) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7631] == sanity-flr test 0a: lfs mirror create with -N option == 15:42:39 (1786304559) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:42:53 (1786304573) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:42:56 (1786304576) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 15:42:59 (1786304579) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:43:22 (1786304602) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:43:25 (1786304605) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 15:43:29 (1786304609) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff908843964000.stats=clear osc.lustre-OST0001-osc-ffff908843964000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:43:37 (1786304617) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:43:47 (1786304627) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 15:43:55 (1786304635) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 15:43:58 (1786304638) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 15:44:06 (1786304646) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 15:44:15 (1786304655) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:44:22 (1786304662) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:44:25 (1786304665) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:44:33 (1786304673) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 15:44:44 (1786304684) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:44:51 (1786304691) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.287967 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193862 s, 5.4 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:45:02 (1786304702) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.290177 s, 25.3 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 15:45:10 (1786304710) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:45:22 (1786304722) Stopping /mnt/lustre-ost1 (opts:) on oleg110-server oleg110-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff908843964000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff908843964000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg110-server: oleg110-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg110-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg110-server oleg110-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff908843964000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff908843964000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg110-server: oleg110-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg110-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (74s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 15:46:36 (1786304796) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.299681 s, 10.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg110-server oleg110-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff908843964000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff908843964000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg110-server: oleg110-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg110-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:47:21 (1786304841) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg110-server oleg110-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff908843964000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff908843964000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg110-server: oleg110-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg110-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg110-server oleg110-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff908843964000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff908843964000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg110-server: oleg110-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg110-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (76s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:48:37 (1786304917) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.86927 s, 10.8 MB/s -rw-r--r-- 1 root root 52428800 Aug 9 15:48 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.078s user 0m0.001s sys 0m0.179s Stopping /mnt/lustre-ost1 (opts:) on oleg110-server oleg110-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff908843964000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff908843964000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.180s user 0m0.000s sys 0m0.138s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg110-server: oleg110-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg110-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg110-server oleg110-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff908843964000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff908843964000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.244s user 0m0.001s sys 0m0.191s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg110-server: oleg110-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg110-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg110-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff908843964000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff908843964000.ost_server_uuid in FULL state after 0 sec PASS 33b (88s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:50:05 (1786305005) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:50:08 (1786305008) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:50:12 (1786305012) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 15:50:16 (1786305016) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 15:50:22 (1786305022) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.463492 s, 27.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:50:33 (1786305033) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336463 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523922 s, 20.0 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:50:40 (1786305040) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:50:49 (1786305049) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000873512 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000677662 s, 1.5 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0022877 s, 448 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000888005 s, 1.2 MB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 15:50:59 (1786305059) create test files with size 15736211 .. 8ce893e6c7aa6e8558f5fb595dfa0731 - 80edcdcbe4591eb292d2b98e956bdcdd - 948d133bf35f7c1c5663dd3a5f1bd506 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg110-client.virtnet /mnt/lustre (opts:) Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 15:51:25 (1786305085) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 15:51:40 (1786305100) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 15:51:46 (1786305106) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768343 s, 13.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295716 s, 35.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xe0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (7s) == sanity-flr test 41: lfs mirror resync check =========== 15:51:53 (1786305113) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.110699 s, 18.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.210892 s, 19.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.173297 s, 12.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (12s) == sanity-flr test 42: lfs mirror verify ================= 15:52:05 (1786305125) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 15:52:08 (1786305128) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 15:52:11 (1786305131) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449512 s, 23.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 44a: lfs mirror split check =========== 15:52:17 (1786305137) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 15:52:20 (1786305140) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:52:29 (1786305149) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.231206 s, 45.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg110-client.virtnet /mnt/lustre (opts:) Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:52:39 (1786305159) llite.lustre-ffff908843ca1800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.231373 s, 45.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg110-client.virtnet /mnt/lustre (opts:) Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff908846023000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 15:52:52 (1786305172) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (37s) == sanity-flr test 46: Verify setstripe --copy option ==== 15:53:29 (1786305209) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (33s) == sanity-flr test 47: Verify mirror obj alloc =========== 15:54:02 (1786305242) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 15:54:05 (1786305245) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.163432 s, 19.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.14037 s, 22.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xfa34fa35 (nosync,stale) Mirror 2: 0xfa34fa35 (nosync,stale) ** verify mirror 2 doesn't change original checksum: ae9c0e9ffdcfae65044beb2fb73a2e11 - mirror 1 checksum: 92b5d079800647dbea40aea86028e816 - mirror 2 checksum: ae9c0e9ffdcfae65044beb2fb73a2e11 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-09T15:54:09-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-09T15:54:09-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: ae9c0e9ffdcfae65044beb2fb73a2e11 - mirror 1 checksum: 92b5d079800647dbea40aea86028e816 - mirror 2 checksum: 92b5d079800647dbea40aea86028e816 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-09T15:54:09-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-09T15:54:09-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 15:54:17 (1786305257) Failing mds1 on oleg110-server Stopping /mnt/lustre-mds1 (opts:) on oleg110-server 15:54:23 (1786305263) shut down facet: mds1 facet_host: oleg110-server facet_failover_host: oleg110-server Failover mds1 to oleg110-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-MDT0000 15:54:56 (1786305296) targets are mounted 15:54:56 (1786305296) facet_failover done oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438568 s, 23.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } PASS 50A (64s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 15:55:22 (1786305322) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 15:55:25 (1786305325) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 15:55:29 (1786305329) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00598319 s, 2.7 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 15:55:36 (1786305336) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 15:55:39 (1786305339) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00777867 s, 527 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0640577 s, 63.9 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 15:55:46 (1786305346) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 15:56:08 (1786305368) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (25s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 15:56:33 (1786305393) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 15:56:56 (1786305416) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 15:57:19 (1786305439) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 15:57:27 (1786305447) mirror create pid 54311 mirror split pid 54312 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54311 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54312 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 15:58:35 (1786305515) Seed set to 5517 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1786305518.129916 mapwrite 0x00c9c8 thru 0x0189fc (0xc035 bytes) 000002 1786305518.402449 write 0x0000aa thru 0x00230b (0x2262 bytes) 000003 1786305518.435497 write 0x00e344 thru 0x011474 (0x3131 bytes) truncating to largest ever: 0x1caaf 000004 1786305518.440544 trunc from 0x0189fd to 0x01caae (0x40b2 bytes) 000005 1786305518.480940 mapwrite 0x024e62 thru 0x032d5f (0xdefe bytes) 000006 1786305518.694567 write 0x00f885 thru 0x01bcb8 (0xc434 bytes) 000008 1786305518.703268 write 0x035880 thru 0x0399d6 (0x4157 bytes) 000009 1786305518.704466 read 0x01d7da thru 0x01fe78 (0x269f bytes) 000010 1786305518.705894 mapread 0x015d5f thru 0x019915 (0x3bb7 bytes) 000023 1786305518.727591 MIRROR_EXTEND to 2 mirrors 000024 1786305520.096340 MIRROR_RESYNC 2 mirrors 000025 1786305520.502959 read 0x02d9d9 thru 0x031162 (0x378a bytes) 000026 1786305520.520512 mapread 0x02c72b thru 0x03544f (0x8d25 bytes) 000027 1786305520.559564 mapwrite 0x03ccd8 thru 0x03ffff (0x3328 bytes) truncating to largest ever: 0x1d981 000028 1786305520.797367 trunc from 0x040000 to 0x01d980 (0x2267f bytes) 000029 1786305520.832640 mapwrite 0x013420 thru 0x0207f3 (0xd3d4 bytes) 000030 1786305520.976318 read 0x01d53a thru 0x0207f3 (0x32ba bytes) truncating to largest ever: 0x3191a 000032 1786305520.977548 trunc from 0x0207f4 to 0x031919 (0x11126 bytes) 000035 1786305521.008650 trunc from 0x03191a to 0x010349 (0x215d0 bytes) 000037 1786305521.041325 mapwrite 0x03a1e1 thru 0x03ec0a (0x4a2a bytes) 000041 1786305521.155896 mapwrite 0x02c24a thru 0x02f798 (0x354f bytes) 000042 1786305521.220512 mapread 0x00fe84 thru 0x01ebe1 (0xed5e bytes) 000043 1786305521.222095 mapread 0x01c7dd thru 0x022fc2 (0x67e6 bytes) 000044 1786305521.227195 mapwrite 0x03e762 thru 0x03ffff (0x189e bytes) 000046 1786305521.304960 read 0x019553 thru 0x0203b4 (0x6e62 bytes) 000047 1786305521.309955 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x89e9ff00 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf86390b1 Mirror 3: 0xf86390b1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped truncating to largest ever: 0x35cc2 000049 1786305522.156260 trunc from 0x040000 to 0x035cc1 (0xa33e bytes) 000050 1786305522.264858 mapread 0x0245fc thru 0x02b4a0 (0x6ea5 bytes) 000051 1786305522.291560 mapread 0x0242c8 thru 0x030c55 (0xc98e bytes) 000053 1786305522.301915 trunc from 0x035cc2 to 0x034e13 (0xeae bytes) 000055 1786305522.330229 write 0x03eaf7 thru 0x03ffff (0x1509 bytes) 000057 1786305522.332494 read 0x0262f1 thru 0x02ed69 (0x8a79 bytes) 000059 1786305522.335095 mapread 0x02872d thru 0x02d8fa (0x51ce bytes) 000063 1786305522.339939 write 0x01eae2 thru 0x024390 (0x58af bytes) 000064 1786305522.345440 read 0x028fee thru 0x035851 (0xc864 bytes) 000065 1786305522.361269 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf86390b1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf86390b1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000068 1786305522.629458 write 0x02d2a2 thru 0x032523 (0x5282 bytes) 000070 1786305522.643850 MIRROR_RESYNC 2 mirrors 000071 1786305523.356651 read 0x033485 thru 0x03c8c2 (0x943e bytes) truncating to largest ever: 0x372f3 000072 1786305523.367504 trunc from 0x040000 to 0x0372f2 (0x8d0d bytes) 000073 1786305523.441860 mapread 0x034863 thru 0x0372f2 (0x2a90 bytes) 000078 1786305523.463566 trunc from 0x0372f3 to 0x003f9b (0x33357 bytes) 000080 1786305523.487079 write 0x025ecc thru 0x02ea07 (0x8b3c bytes) 000081 1786305523.494109 MIRROR_RESYNC 2 mirrors 000083 1786305524.292151 MIRROR_EXTEND to 3 mirrors 000084 1786305525.284772 trunc from 0x02ea08 to 0x00cdf8 (0x21c0f bytes) 000087 1786305525.370611 read 0x00b723 thru 0x00cdf8 (0x16d6 bytes) 000091 1786305525.377212 trunc from 0x00cdf9 to 0x018a45 (0xbc4d bytes) 000094 1786305525.402477 read 0x0060c5 thru 0x014f09 (0xee45 bytes) 000095 1786305525.413299 trunc from 0x018a46 to 0x0222cc (0x9887 bytes) 000097 1786305525.435169 mapread 0x0214fa thru 0x0222cc (0xdd3 bytes) 000098 1786305525.457599 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x222cd): Mirror 3: 0x28ce4eae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x222cd): Mirror 3: 0x28ce4eae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x222cd: skipped 000099 1786305525.711088 mapread 0x01a216 thru 0x0219d5 (0x77c0 bytes) 000104 1786305525.735182 read 0x0219f6 thru 0x0222cc (0x8d7 bytes) 000105 1786305525.736445 mapwrite 0x017a07 thru 0x019a26 (0x2020 bytes) 000107 1786305525.798687 read 0x0215be thru 0x0222cc (0xd0f bytes) 000109 1786305525.800877 mapwrite 0x0061d3 thru 0x013009 (0xce37 bytes) 000111 1786305525.889931 write 0x028b33 thru 0x029c47 (0x1115 bytes) 000114 1786305525.894156 mapread 0x00dc17 thru 0x011d84 (0x416e bytes) 000116 1786305525.895923 trunc from 0x029c48 to 0x01046b (0x197dc bytes) 000117 1786305525.924336 write 0x013aa5 thru 0x01668e (0x2bea bytes) 000119 1786305525.927206 mapwrite 0x00a90c thru 0x00fd54 (0x5449 bytes) 000120 1786305525.992405 mapread 0x000583 thru 0x00d3e3 (0xce61 bytes) 000121 1786305525.993631 read 0x00cdd0 thru 0x01668e (0x98bf bytes) 000124 1786305526.012023 mapread 0x00603c thru 0x00e0bc (0x8081 bytes) 000126 1786305526.017240 read 0x0055c5 thru 0x007b47 (0x2583 bytes) 000127 1786305526.020214 mapwrite 0x00d875 thru 0x012de6 (0x5572 bytes) 000128 1786305526.134080 mapwrite 0x03444b thru 0x03ffff (0xbbb5 bytes) 000130 1786305526.246540 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) 000132 1786305526.385796 trunc from 0x040000 to 0x000417 (0x3fbe8 bytes) 000133 1786305526.410429 write 0x01ecc9 thru 0x02a69f (0xb9d7 bytes) 000134 1786305526.412827 write 0x033eee thru 0x03ffff (0xc112 bytes) 000135 1786305526.419421 mapwrite 0x01e8d8 thru 0x02e4d6 (0xfbff bytes) 000139 1786305526.479190 mapread 0x0055cc thru 0x0141cd (0xec02 bytes) 000140 1786305526.514928 MIRROR_EXTEND to 2 mirrors 000142 1786305527.210024 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) 000143 1786305527.332137 mapread 0x034cb1 thru 0x0362d3 (0x1623 bytes) 000145 1786305527.351011 mapwrite 0x001fa8 thru 0x0118bf (0xf918 bytes) 000149 1786305527.473006 trunc from 0x040000 to 0x018aa4 (0x2755b bytes) 000151 1786305527.554553 mapwrite 0x00e0eb thru 0x01a89e (0xc7b4 bytes) 000152 1786305527.683729 mapwrite 0x0091bd thru 0x00df7f (0x4dc3 bytes) 000153 1786305527.743277 read 0x014934 thru 0x01a89e (0x5f6b bytes) 000154 1786305527.744013 trunc from 0x01a89f to 0x0342b1 (0x19a13 bytes) 000155 1786305527.772043 read 0x00990c thru 0x00eea6 (0x559b bytes) 000158 1786305527.773759 mapread 0x029dcd thru 0x032d43 (0x8f77 bytes) 000159 1786305527.790337 trunc from 0x0342b2 to 0x002092 (0x3221f bytes) 000160 1786305527.807849 read 0x00150e thru 0x002092 (0xb85 bytes) 000161 1786305527.808554 mapread 0x000d52 thru 0x002092 (0x1341 bytes) 000162 1786305527.815597 read 0x001ece thru 0x002092 (0x1c5 bytes) 000163 1786305527.816314 read 0x0007b6 thru 0x002092 (0x18dd bytes) 000172 1786305527.838804 mapwrite 0x0381a0 thru 0x03ffff (0x7e60 bytes) 000174 1786305527.936146 trunc from 0x040000 to 0x02154d (0x1eab2 bytes) truncating to largest ever: 0x3d3ce 000176 1786305527.958684 trunc from 0x02154e to 0x03d3cd (0x1be80 bytes) 000178 1786305527.976394 mapread 0x02e5f2 thru 0x036504 (0x7f13 bytes) 000179 1786305528.049989 mapread 0x03812b thru 0x03b593 (0x3469 bytes) 000180 1786305528.096189 write 0x03977a thru 0x03cc54 (0x34db bytes) 000181 1786305528.113178 read 0x036435 thru 0x03d3cd (0x6f99 bytes) 000182 1786305528.126570 read 0x02540d thru 0x030588 (0xb17c bytes) 000183 1786305528.128117 write 0x00b940 thru 0x01633c (0xa9fd bytes) 000184 1786305528.130658 write 0x006fad thru 0x00ed46 (0x7d9a bytes) 000185 1786305528.134530 mapwrite 0x03c98a thru 0x03ffff (0x3676 bytes) 000186 1786305528.216671 read 0x0181c2 thru 0x026296 (0xe0d5 bytes) 000187 1786305528.219409 write 0x01f5ab thru 0x024769 (0x51bf bytes) 000189 1786305528.228993 mapread 0x0026db thru 0x00cdcc (0xa6f2 bytes) 000193 1786305528.234121 mapwrite 0x01d36e thru 0x021b68 (0x47fb bytes) 000194 1786305528.280565 write 0x01301e thru 0x01398e (0x971 bytes) 000196 1786305528.282179 write 0x0105a9 thru 0x01aa1a (0xa472 bytes) 000197 1786305528.289409 MIRROR_EXTEND to 2 mirrors 000199 1786305528.895462 MIRROR_EXTEND to 3 mirrors 000200 1786305529.432827 mapwrite 0x018b78 thru 0x0271a0 (0xe629 bytes) 000202 1786305529.648973 mapwrite 0x020c5d thru 0x02f1ef (0xe593 bytes) 000203 1786305529.715163 mapread 0x0288ed thru 0x032187 (0x989b bytes) 000204 1786305529.716320 write 0x03f79c thru 0x03ffff (0x864 bytes) 000205 1786305529.721949 mapread 0x03dee8 thru 0x03ffff (0x2118 bytes) 000208 1786305529.753016 mapwrite 0x02afd6 thru 0x03abc4 (0xfbef bytes) 000211 1786305529.878491 trunc from 0x040000 to 0x00d8ab (0x32754 bytes) 000212 1786305529.906558 trunc from 0x00d8ac to 0x03909c (0x2b7f1 bytes) 000213 1786305529.933559 trunc from 0x03909d to 0x02a74d (0xe94f bytes) 000216 1786305529.962415 read 0x024bf6 thru 0x02828a (0x3695 bytes) 000217 1786305529.973276 mapread 0x007e85 thru 0x0159e0 (0xdb5c bytes) 000219 1786305530.006151 trunc from 0x02a74e to 0x00f642 (0x1b10b bytes) 000222 1786305530.026526 write 0x030f71 thru 0x03c29d (0xb32d bytes) 000223 1786305530.029420 mapwrite 0x0013d9 thru 0x004adc (0x3704 bytes) 000226 1786305530.083357 read 0x006a1c thru 0x01283b (0xbe20 bytes) 000227 1786305530.099985 mapwrite 0x00bba6 thru 0x01a616 (0xea71 bytes) 000230 1786305530.170251 mapwrite 0x03c6b4 thru 0x03ffff (0x394c bytes) 000232 1786305530.263727 mapwrite 0x00f713 thru 0x01026f (0xb5d bytes) 000233 1786305530.318516 write 0x021433 thru 0x02fb45 (0xe713 bytes) 000234 1786305530.320442 write 0x00545d thru 0x0119bf (0xc563 bytes) 000235 1786305530.322111 mapwrite 0x00e286 thru 0x01b0fb (0xce76 bytes) 000238 1786305530.379514 mapread 0x0277bd thru 0x027aa5 (0x2e9 bytes) 000239 1786305530.381220 read 0x033c8f thru 0x03b581 (0x78f3 bytes) 000240 1786305530.382655 mapwrite 0x03e0ad thru 0x03ffff (0x1f53 bytes) 000241 1786305530.430247 write 0x00007c thru 0x009f15 (0x9e9a bytes) 000245 1786305530.437785 mapread 0x03366a thru 0x03842d (0x4dc4 bytes) 000246 1786305530.439466 write 0x014b32 thru 0x01d9ff (0x8ece bytes) 000247 1786305530.441442 mapwrite 0x0000e9 thru 0x00b859 (0xb771 bytes) 000248 1786305530.514167 mapwrite 0x01de4b thru 0x02817c (0xa332 bytes) 000249 1786305530.564261 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8ada0981 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8ada0981 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000250 1786305530.936542 MIRROR_RESYNC 2 mirrors 000251 1786305531.554947 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) 000252 1786305531.678041 write 0x0167ce thru 0x01d9c4 (0x71f7 bytes) truncating to largest ever: 0x3f465 000253 1786305531.699972 trunc from 0x040000 to 0x03f464 (0xb9b bytes) 000254 1786305531.716978 trunc from 0x03f465 to 0x0370c1 (0x83a3 bytes) 000257 1786305531.749442 trunc from 0x0370c2 to 0x03c451 (0x5390 bytes) 000258 1786305531.768536 write 0x007c1c thru 0x0159a9 (0xdd8e bytes) 000259 1786305531.799077 MIRROR_EXTEND to 2 mirrors 000260 1786305532.476149 mapwrite 0x033bce thru 0x03ec73 (0xb0a6 bytes) 000262 1786305532.660077 trunc from 0x03ec74 to 0x03642e (0x8845 bytes) 000265 1786305532.693086 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) 000267 1786305532.882961 mapwrite 0x02af31 thru 0x03ab0d (0xfbdd bytes) 000268 1786305532.991551 trunc from 0x03ab0e to 0x02b24a (0xf8c3 bytes) 000272 1786305533.013154 trunc from 0x02b24b to 0x0390ba (0xde70 bytes) 000274 1786305533.044911 mapread 0x02764f thru 0x03313a (0xbaec bytes) 000275 1786305533.059680 trunc from 0x0390bb to 0x002c9e (0x3641c bytes) 000276 1786305533.083047 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x3fdc6 000278 1786305533.697000 trunc from 0x002c9f to 0x03fdc5 (0x3d127 bytes) 000281 1786305533.794473 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) 000283 1786305533.947114 mapwrite 0x01a34f thru 0x01bf08 (0x1bba bytes) 000284 1786305534.025137 trunc from 0x03fdc6 to 0x01061e (0x2f7a7 bytes) 000288 1786305534.051915 read 0x00f867 thru 0x01061e (0xdb8 bytes) 000289 1786305534.057276 trunc from 0x01061f to 0x02b0f0 (0x1aad2 bytes) 000290 1786305534.080003 read 0x004786 thru 0x010a66 (0xc2e1 bytes) 000292 1786305534.131159 MIRROR_EXTEND to 2 mirrors 000293 1786305534.947060 read 0x02a9f3 thru 0x02b0f0 (0x6fe bytes) 000294 1786305534.952876 write 0x014b36 thru 0x01d505 (0x89d0 bytes) 000295 1786305535.021153 mapwrite 0x00939e thru 0x015dfb (0xca5e bytes) 000302 1786305535.090666 mapwrite 0x03834d thru 0x0386f1 (0x3a5 bytes) 000303 1786305535.174278 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) 000304 1786305535.294116 trunc from 0x0386f2 to 0x00e0a7 (0x2a64a bytes) 000305 1786305535.312086 write 0x002e3b thru 0x00ee9c (0xc062 bytes) 000306 1786305535.322309 mapwrite 0x01b7d2 thru 0x021e27 (0x6656 bytes) 000307 1786305535.396012 mapread 0x012625 thru 0x02081c (0xe1f8 bytes) 000308 1786305535.404886 mapwrite 0x009663 thru 0x0166c2 (0xd060 bytes) 000309 1786305535.443519 trunc from 0x021e28 to 0x034fb1 (0x1318a bytes) 000312 1786305535.458116 read 0x00f490 thru 0x013cc3 (0x4834 bytes) 000313 1786305535.458687 read 0x01f2c0 thru 0x02966a (0xa3ab bytes) 000315 1786305535.469130 trunc from 0x034fb2 to 0x03c174 (0x71c3 bytes) 000323 1786305535.486626 read 0x001be2 thru 0x004edc (0x32fb bytes) 000324 1786305535.487815 read 0x027e86 thru 0x036f30 (0xf0ab bytes) 000325 1786305535.495143 mapread 0x023588 thru 0x02c5f8 (0x9071 bytes) 000326 1786305535.499426 MIRROR_EXTEND to 2 mirrors 000327 1786305536.438892 mapwrite 0x019ee6 thru 0x029769 (0xf884 bytes) 000331 1786305536.606167 read 0x017a15 thru 0x01f19e (0x778a bytes) 000333 1786305536.608635 trunc from 0x03c175 to 0x02a2dd (0x11e97 bytes) 000335 1786305536.630804 read 0x0160a6 thru 0x01c2c5 (0x6220 bytes) 000337 1786305536.632100 mapread 0x018d7b thru 0x01b531 (0x27b7 bytes) 000338 1786305536.636098 mapread 0x00a175 thru 0x012736 (0x85c2 bytes) 000340 1786305536.650503 mapwrite 0x02f176 thru 0x035708 (0x6593 bytes) 000341 1786305536.725579 mapread 0x020ee6 thru 0x024915 (0x3a30 bytes) 000342 1786305536.728358 trunc from 0x035709 to 0x01b696 (0x1a072 bytes) 000348 1786305536.746417 read 0x001518 thru 0x011460 (0xff49 bytes) 000349 1786305536.753382 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1b697): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b697): Mirror 2: 0x4237fe0 (stale) CRC-32 checksum value for chunk [0, 0x1b697): Mirror 1: 0xf996c2ec Mirror 3: 0xf996c2ec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b697: skipped 000350 1786305537.476937 write 0x025da0 thru 0x02d3f5 (0x7656 bytes) 000351 1786305537.538632 read 0x003587 thru 0x00eef3 (0xb96d bytes) 000352 1786305537.548140 read 0x025a03 thru 0x02d3f5 (0x79f3 bytes) 000355 1786305537.549992 write 0x021599 thru 0x028cfe (0x7766 bytes) 000357 1786305537.560212 write 0x018d99 thru 0x0219eb (0x8c53 bytes) 000361 1786305537.564908 mapread 0x006e7c thru 0x0107a6 (0x992b bytes) 000362 1786305537.566938 mapwrite 0x034091 thru 0x03ffff (0xbf6f bytes) 000363 1786305537.669356 mapwrite 0x03519a thru 0x0361ed (0x1054 bytes) 000364 1786305537.758333 mapwrite 0x03bbce thru 0x03ffff (0x4432 bytes) 000365 1786305537.832111 write 0x0117d7 thru 0x020d34 (0xf55e bytes) 000366 1786305537.833687 mapread 0x01196a thru 0x01626b (0x4902 bytes) 000368 1786305537.843386 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x11448d68 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x11448d68 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000369 1786305538.192576 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x11448d68 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa020688c Mirror 4: 0xa020688c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000370 1786305539.066579 mapread 0x00bfb5 thru 0x00df0e (0x1f5a bytes) 000372 1786305539.110962 write 0x020178 thru 0x027ea4 (0x7d2d bytes) 000373 1786305539.197785 trunc from 0x040000 to 0x02e1fd (0x11e02 bytes) 000374 1786305539.218500 read 0x0280c3 thru 0x02e1fd (0x613b bytes) 000375 1786305539.224834 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e1fe): Mirror 4: 0xf5b0852c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e1fe): Mirror 4: 0xf5b0852c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e1fe: skipped 000376 1786305539.577069 mapread 0x019ef5 thru 0x01c597 (0x26a3 bytes) 000377 1786305539.605128 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2e1fe): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e1fe): Mirror 4: 0xf5b0852c (stale) CRC-32 checksum value for chunk [0, 0x2e1fe): Mirror 1: 0x56de3b4d Mirror 5: 0x56de3b4d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e1fe: skipped 000380 1786305540.459968 read 0x011ba0 thru 0x0204d5 (0xe936 bytes) 000381 1786305540.471835 read 0x00a4e4 thru 0x013dfe (0x991b bytes) 000383 1786305540.487215 trunc from 0x02e1fe to 0x020d11 (0xd4ec bytes) 000384 1786305540.604839 write 0x038c87 thru 0x03e87b (0x5bf5 bytes) 000387 1786305540.617255 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e87c): Mirror 5: 0x791b1176 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e87c): Mirror 5: 0x791b1176 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e87c: skipped 000388 1786305541.013466 write 0x025018 thru 0x02d448 (0x8431 bytes) 000389 1786305541.025374 read 0x01efd3 thru 0x020dd3 (0x1e01 bytes) 000391 1786305541.036363 MIRROR_RESYNC 2 mirrors 000392 1786305541.743464 trunc from 0x03e87c to 0x01b31d (0x2355e bytes) 000394 1786305541.851515 write 0x0079ed thru 0x00d09e (0x56b2 bytes) 000395 1786305541.866179 mapwrite 0x0182d1 thru 0x027987 (0xf6b7 bytes) 000396 1786305541.969526 trunc from 0x027988 to 0x038212 (0x1088b bytes) 000397 1786305541.986925 write 0x016190 thru 0x01c0ed (0x5f5e bytes) 000403 1786305541.992972 mapread 0x03764a thru 0x038212 (0xbc9 bytes) 000404 1786305542.002897 read 0x032440 thru 0x037341 (0x4f02 bytes) 000406 1786305542.004161 trunc from 0x038213 to 0x01fbaf (0x18663 bytes) 000408 1786305542.025482 write 0x02d2b6 thru 0x02ebe9 (0x1934 bytes) 000409 1786305542.027856 mapwrite 0x00abbb thru 0x0160d4 (0xb51a bytes) 000410 1786305542.086019 read 0x01b995 thru 0x02822f (0xc89b bytes) 000411 1786305542.102134 read 0x015419 thru 0x016d5f (0x1947 bytes) 000412 1786305542.106499 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2ebea): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ebea): Mirror 5: 0x48ee4b16 (stale) CRC-32 checksum value for chunk [0, 0x2ebea): Mirror 1: 0xf3742925 Mirror 6: 0xf3742925 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ebea: skipped 000413 1786305542.789305 mapread 0x01602c thru 0x025b2e (0xfb03 bytes) 000414 1786305542.834168 mapread 0x005bfd thru 0x00903e (0x3442 bytes) 000416 1786305542.872883 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2ebea): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ebea): Mirror 5: 0x48ee4b16 (stale) CRC-32 checksum value for chunk [0, 0x2ebea): Mirror 1: 0xf3742925 Mirror 6: 0xf3742925 Mirror 7: 0xf3742925 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ebea: skipped 000418 1786305543.592579 mapread 0x026e39 thru 0x02ebe9 (0x7db1 bytes) 000421 1786305543.618349 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2ebea): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ebea): Mirror 5: 0x48ee4b16 (stale) CRC-32 checksum value for chunk [0, 0x2ebea): Mirror 1: 0xf3742925 Mirror 6: 0xf3742925 Mirror 7: 0xf3742925 Mirror 8: 0xf3742925 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ebea: skipped 000428 1786305544.467973 trunc from 0x02ebea to 0x021840 (0xd3a9 bytes) 000429 1786305544.594560 read 0x0050d3 thru 0x008238 (0x3166 bytes) 000431 1786305544.609438 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x21841): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21841): Mirror 5: 0xb6a454b6 (stale) Mirror 6: 0x4ee18d14 (stale) Mirror 7: 0x4ee18d14 (stale) Mirror 8: 0x4ee18d14 (stale) CRC-32 checksum value for chunk [0, 0x21841): Mirror 1: 0x4ee18d14 Mirror 9: 0x4ee18d14 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21841: skipped 000434 1786305545.489142 mapwrite 0x03cf24 thru 0x03cfc0 (0x9d bytes) 000435 1786305545.702456 MIRROR_RESYNC 6 mirrors 000436 1786305546.959146 trunc from 0x03cfc1 to 0x032c9b (0xa325 bytes) 000438 1786305547.062092 write 0x013232 thru 0x015f3a (0x2d09 bytes) 000439 1786305547.084107 write 0x039a3d thru 0x03ffff (0x65c3 bytes) 000441 1786305547.093392 mapwrite 0x02b1d9 thru 0x03075d (0x5585 bytes) 000443 1786305547.213955 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xa8c89b42 (stale) Mirror 7: 0xa8c89b42 (stale) Mirror 8: 0xa8c89b42 (stale) Mirror 9: 0xa8c89b42 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xa8c89b42 (stale) Mirror 7: 0xa8c89b42 (stale) Mirror 8: 0xa8c89b42 (stale) Mirror 9: 0xa8c89b42 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000444 1786305547.724965 write 0x02831f thru 0x033ba6 (0xb888 bytes) 000445 1786305547.735467 read 0x018be5 thru 0x01d6a8 (0x4ac4 bytes) 000448 1786305547.752337 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xa8c89b42 (stale) Mirror 7: 0xa8c89b42 (stale) Mirror 9: 0xa8c89b42 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xa8c89b42 (stale) Mirror 7: 0xa8c89b42 (stale) Mirror 9: 0xa8c89b42 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000449 1786305548.076957 mapwrite 0x02bc99 thru 0x035d55 (0xa0bd bytes) 000450 1786305548.181972 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xa8c89b42 (stale) Mirror 7: 0xa8c89b42 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xa8c89b42 (stale) Mirror 7: 0xa8c89b42 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000451 1786305548.542834 read 0x013102 thru 0x01bf0d (0x8e0c bytes) 000456 1786305548.557962 write 0x00c08c thru 0x01bced (0xfc62 bytes) 000458 1786305548.562927 trunc from 0x040000 to 0x000b98 (0x3f467 bytes) 000459 1786305548.578401 trunc from 0x000b99 to 0x027a60 (0x26ec8 bytes) 000460 1786305548.602423 MIRROR_RESYNC 3 mirrors 000461 1786305549.379557 write 0x0105c7 thru 0x01a53f (0x9f79 bytes) 000462 1786305549.467262 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27a61): Mirror 7: 0xfd8922fa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27a61): Mirror 7: 0xfd8922fa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27a61: skipped 000463 1786305549.771420 write 0x0208b2 thru 0x025482 (0x4bd1 bytes) 000468 1786305549.780975 read 0x010e8b thru 0x014cae (0x3e24 bytes) 000469 1786305549.786779 mapread 0x00135d thru 0x00976b (0x840f bytes) 000470 1786305549.817192 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x27a61): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27a61): Mirror 7: 0xfd8922fa (stale) CRC-32 checksum value for chunk [0, 0x27a61): Mirror 1: 0x7a718070 Mirror 8: 0x7a718070 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27a61: skipped 000471 1786305550.579492 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) 000472 1786305550.898226 mapwrite 0x01dac4 thru 0x01ef69 (0x14a6 bytes) 000474 1786305551.150520 mapread 0x019dda thru 0x01d66b (0x3892 bytes) 000475 1786305551.154480 write 0x0074af thru 0x00c447 (0x4f99 bytes) 000476 1786305551.177552 read 0x02304d thru 0x023666 (0x61a bytes) 000478 1786305551.180013 mapread 0x00f781 thru 0x01a795 (0xb015 bytes) 000479 1786305551.200304 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x27a61): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27a61): Mirror 8: 0x7a718070 (stale) CRC-32 checksum value for chunk [0, 0x27a61): Mirror 1: 0xfcab6bb Mirror 9: 0xfcab6bb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27a61: skipped 000480 1786305552.084809 mapwrite 0x00c761 thru 0x00df91 (0x1831 bytes) 000481 1786305552.277829 MIRROR_RESYNC 3 mirrors 000485 1786305553.014546 write 0x005f7b thru 0x010f43 (0xafc9 bytes) 000486 1786305553.062137 read 0x0054d3 thru 0x00e27d (0x8dab bytes) 000487 1786305553.063862 write 0x031a9d thru 0x039b4a (0x80ae bytes) 000491 1786305553.074555 mapwrite 0x01ccff thru 0x02a3da (0xd6dc bytes) 000492 1786305553.191619 write 0x00e916 thru 0x019dca (0xb4b5 bytes) 000495 1786305553.195961 mapread 0x00963e thru 0x00da68 (0x442b bytes) 000496 1786305553.197830 mapwrite 0x01a983 thru 0x020fc0 (0x663e bytes) 000497 1786305553.278300 read 0x0147eb thru 0x02358c (0xeda2 bytes) 000498 1786305553.280742 mapread 0x03870d thru 0x039b4a (0x143e bytes) 000499 1786305553.283022 trunc from 0x039b4b to 0x026249 (0x13901 bytes) 000500 1786305553.306015 mapread 0x018d7a thru 0x01e328 (0x55af bytes) 000502 1786305553.309470 mapwrite 0x030bc5 thru 0x0322c2 (0x16fe bytes) 000503 1786305553.380332 mapwrite 0x03315c thru 0x03ee6b (0xbd10 bytes) 000509 1786305553.473756 write 0x03947a thru 0x03cdbc (0x3943 bytes) 000510 1786305553.475910 trunc from 0x03ee6c to 0x0254b9 (0x199b2 bytes) 000513 1786305553.503010 mapread 0x005a87 thru 0x01130a (0xb884 bytes) 000514 1786305553.505927 read 0x005070 thru 0x00d493 (0x8424 bytes) 000519 1786305553.512178 trunc from 0x0254ba to 0x036533 (0x1107a bytes) 000520 1786305553.529362 write 0x01d24b thru 0x020d5a (0x3b10 bytes) 000521 1786305553.531835 trunc from 0x036534 to 0x01325c (0x232d7 bytes) 000522 1786305553.559004 MIRROR_RESYNC 3 mirrors 000523 1786305554.161096 read 0x00e024 thru 0x01325c (0x5239 bytes) 000524 1786305554.174778 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) 000526 1786305554.462977 write 0x01a71a thru 0x029975 (0xf25c bytes) 000527 1786305554.556105 mapwrite 0x00080d thru 0x0102de (0xfad2 bytes) 000529 1786305554.649476 read 0x0060c1 thru 0x0107d4 (0xa714 bytes) 000530 1786305554.650247 trunc from 0x029976 to 0x022b20 (0x6e55 bytes) 000532 1786305554.670479 mapwrite 0x03482c thru 0x03ffff (0xb7d4 bytes) 000533 1786305554.749325 MIRROR_RESYNC 2 mirrors 000534 1786305555.277414 mapwrite 0x03f30e thru 0x03ffff (0xcf2 bytes) 000535 1786305555.459124 mapwrite 0x02ac2f thru 0x02acb2 (0x84 bytes) 000536 1786305555.567073 read 0x007c3c thru 0x00cb43 (0x4f08 bytes) 000537 1786305555.588810 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) 000539 1786305555.803403 write 0x006c2a thru 0x00bdd5 (0x51ac bytes) 000540 1786305555.822390 mapwrite 0x02ff22 thru 0x038c10 (0x8cef bytes) 000541 1786305555.910640 mapread 0x00f959 thru 0x01a168 (0xa810 bytes) 000542 1786305555.951045 MIRROR_EXTEND to 2 mirrors 000543 1786305556.639933 trunc from 0x040000 to 0x024725 (0x1b8da bytes) 000544 1786305556.737637 trunc from 0x024726 to 0x01e35b (0x63ca bytes) 000546 1786305556.766570 read 0x00816f thru 0x009a06 (0x1898 bytes) 000547 1786305556.772146 mapwrite 0x0070ef thru 0x0124fd (0xb40f bytes) 000548 1786305556.849389 trunc from 0x01e35c to 0x02eff7 (0x10c9c bytes) 000549 1786305556.870893 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) 000550 1786305557.009609 trunc from 0x02eff8 to 0x0016a9 (0x2d94e bytes) 000551 1786305557.036560 read 0x0009d0 thru 0x0016a9 (0xcda bytes) 000552 1786305557.042166 read 0x000afc thru 0x0016a9 (0xbae bytes) 000553 1786305557.043404 write 0x0062cd thru 0x00a625 (0x4359 bytes) 000555 1786305557.046459 write 0x036205 thru 0x03ffff (0x9dfb bytes) 000556 1786305557.048994 mapwrite 0x031141 thru 0x038104 (0x6fc4 bytes) 000557 1786305557.097910 mapwrite 0x00aea7 thru 0x00bb06 (0xc60 bytes) 000559 1786305557.127533 read 0x037968 thru 0x03acb4 (0x334d bytes) 000560 1786305557.128278 read 0x03ac27 thru 0x03ffff (0x53d9 bytes) 000564 1786305557.132118 trunc from 0x040000 to 0x03b4eb (0x4b14 bytes) 000565 1786305557.150677 read 0x03604f thru 0x037a49 (0x19fb bytes) 000566 1786305557.151771 read 0x0377a5 thru 0x03b4eb (0x3d47 bytes) 000567 1786305557.153223 trunc from 0x03b4ec to 0x03429d (0x724e bytes) 000568 1786305557.166061 write 0x03a579 thru 0x03ffff (0x5a87 bytes) 000569 1786305557.167062 trunc from 0x040000 to 0x033b85 (0xc47a bytes) 000571 1786305557.185668 trunc from 0x033b86 to 0x0349ce (0xe49 bytes) 000572 1786305557.202326 mapwrite 0x02bc83 thru 0x036774 (0xaaf2 bytes) 000573 1786305557.275338 mapwrite 0x0064e8 thru 0x00bb6a (0x5683 bytes) 000577 1786305557.324871 read 0x009f35 thru 0x012df8 (0x8ec4 bytes) 000578 1786305557.325890 trunc from 0x036775 to 0x01b450 (0x1b324 bytes) 000579 1786305557.342273 write 0x037e5a thru 0x03ca52 (0x4bf9 bytes) 000585 1786305557.349596 mapwrite 0x0033e3 thru 0x00601d (0x2c3b bytes) 000586 1786305557.396851 mapread 0x036573 thru 0x03ca52 (0x64e0 bytes) 000587 1786305557.409867 trunc from 0x03ca53 to 0x01adb6 (0x21c9c bytes) 000589 1786305557.424468 read 0x0060af thru 0x007c3f (0x1b91 bytes) 000590 1786305557.425599 mapwrite 0x00dca9 thru 0x012e57 (0x51af bytes) 000591 1786305557.480972 mapread 0x001e88 thru 0x0091a6 (0x731f bytes) 000594 1786305557.484481 mapwrite 0x00db8e thru 0x00fe17 (0x228a bytes) 000596 1786305557.523342 write 0x0126d3 thru 0x01d8a0 (0xb1ce bytes) 000597 1786305557.526171 mapwrite 0x0222a3 thru 0x02a003 (0x7d61 bytes) 000598 1786305557.622918 mapread 0x01d168 thru 0x024aa7 (0x7940 bytes) 000601 1786305557.627095 write 0x03d2ca thru 0x03ffff (0x2d36 bytes) 000602 1786305557.628460 mapwrite 0x021228 thru 0x021af1 (0x8ca bytes) 000603 1786305557.668558 trunc from 0x040000 to 0x03c81f (0x37e0 bytes) 000606 1786305557.684652 mapread 0x0328f9 thru 0x0374e6 (0x4bee bytes) 000607 1786305557.694025 mapread 0x01c89e thru 0x02a700 (0xde63 bytes) 000611 1786305557.698882 read 0x034f7d thru 0x039754 (0x47d8 bytes) 000613 1786305557.700574 trunc from 0x03c820 to 0x026804 (0x1601b bytes) 000618 1786305557.720677 mapread 0x017467 thru 0x0226fe (0xb298 bytes) 000619 1786305557.726910 MIRROR_EXTEND to 2 mirrors 000621 1786305558.468568 mapwrite 0x00d522 thru 0x016ac9 (0x95a8 bytes) 000623 1786305558.640820 mapread 0x00ce2e thru 0x01c516 (0xf6e9 bytes) 000625 1786305558.663578 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) 000627 1786305558.817516 mapwrite 0x0253c6 thru 0x034a78 (0xf6b3 bytes) 000629 1786305558.958146 trunc from 0x034a79 to 0x03d393 (0x891b bytes) 000630 1786305558.974571 write 0x025534 thru 0x02ed6a (0x9837 bytes) 000631 1786305558.981794 MIRROR_EXTEND to 2 mirrors 000632 1786305559.774420 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) 000633 1786305559.938247 mapread 0x017973 thru 0x01dc73 (0x6301 bytes) 000634 1786305559.972839 trunc from 0x03d394 to 0x00e3a9 (0x2efea bytes) 000635 1786305560.005869 MIRROR_EXTEND to 2 mirrors 000636 1786305561.017455 mapwrite 0x001cdc thru 0x00b8fc (0x9c21 bytes) 000638 1786305561.328052 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xe3aa): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe3aa): Mirror 2: 0xb99e9a8f (stale) CRC-32 checksum value for chunk [0, 0xe3aa): Mirror 1: 0x7cf8e7d Mirror 3: 0x7cf8e7d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe3aa: skipped 000639 1786305562.385872 trunc from 0x00e3aa to 0x029bc9 (0x1b820 bytes) 000640 1786305562.474666 mapread 0x004514 thru 0x00b553 (0x7040 bytes) 000641 1786305562.501475 write 0x021b9b thru 0x02e7c6 (0xcc2c bytes) 000643 1786305562.509619 mapread 0x029359 thru 0x02e7c6 (0x546e bytes) 000644 1786305562.511565 mapread 0x028409 thru 0x02c95f (0x4557 bytes) 000645 1786305562.513870 mapread 0x024598 thru 0x02e3da (0x9e43 bytes) 000647 1786305562.516989 write 0x0072c8 thru 0x00ec7a (0x79b3 bytes) 000648 1786305562.519835 mapread 0x00436b thru 0x0100ae (0xbd44 bytes) 000651 1786305562.528548 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2e7c7): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e7c7): Mirror 2: 0x65ffa963 (stale) Mirror 3: 0x1d58926 (stale) CRC-32 checksum value for chunk [0, 0x2e7c7): Mirror 1: 0x42f6f13b Mirror 4: 0x42f6f13b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e7c7: skipped 000652 1786305563.660317 mapwrite 0x0109e7 thru 0x01a4da (0x9af4 bytes) 000655 1786305563.856655 mapwrite 0x000343 thru 0x001f9d (0x1c5b bytes) 000656 1786305563.926177 mapwrite 0x01990e thru 0x01de56 (0x4549 bytes) 000658 1786305564.036049 trunc from 0x02e7c7 to 0x03b720 (0xcf5a bytes) 000661 1786305564.061157 mapread 0x01b27f thru 0x01b787 (0x509 bytes) 000662 1786305564.063122 trunc from 0x03b721 to 0x0124b0 (0x29270 bytes) 000665 1786305564.087039 trunc from 0x0124b1 to 0x01330a (0xe5a bytes) 000669 1786305564.112772 read 0x008cb8 thru 0x01330a (0xa653 bytes) 000673 1786305564.120903 mapwrite 0x025da7 thru 0x02b7e6 (0x5a40 bytes) 000674 1786305564.205083 mapread 0x01295c thru 0x01f8ed (0xcf92 bytes) 000675 1786305564.236031 read 0x0095d9 thru 0x00dbd9 (0x4601 bytes) 000676 1786305564.236958 mapwrite 0x0199cf thru 0x028de1 (0xf413 bytes) 000677 1786305564.295559 trunc from 0x02b7e7 to 0x0205d8 (0xb20e bytes) 000678 1786305564.315453 write 0x01f00b thru 0x0254ce (0x64c4 bytes) 000679 1786305564.318122 write 0x02341d thru 0x02b797 (0x837b bytes) 000682 1786305564.322626 trunc from 0x02b798 to 0x001586 (0x2a211 bytes) 000684 1786305564.339399 trunc from 0x001587 to 0x018d36 (0x177b0 bytes) 000685 1786305564.354467 write 0x035982 thru 0x03e236 (0x88b5 bytes) 000686 1786305564.355959 mapwrite 0x03ef06 thru 0x03ffff (0x10fa bytes) 000689 1786305564.430625 read 0x03ff5e thru 0x03ffff (0xa2 bytes) 000694 1786305564.435011 trunc from 0x040000 to 0x015d37 (0x2a2c8 bytes) 000695 1786305564.465520 read 0x00b7e9 thru 0x00d60f (0x1e27 bytes) 000698 1786305564.470472 trunc from 0x015d38 to 0x00e7de (0x7559 bytes) 000699 1786305564.490468 write 0x02e48f thru 0x039a53 (0xb5c5 bytes) 000700 1786305564.493567 write 0x0206d2 thru 0x023b92 (0x34c1 bytes) 000703 1786305564.507477 read 0x0391cb thru 0x039a53 (0x889 bytes) 000704 1786305564.509049 trunc from 0x039a54 to 0x0262b6 (0x1379d bytes) 000705 1786305564.536486 trunc from 0x0262b7 to 0x02a294 (0x3fde bytes) 000706 1786305564.564826 write 0x01d63d thru 0x02b394 (0xdd58 bytes) 000707 1786305564.568160 write 0x0253ab thru 0x02a1bb (0x4e11 bytes) 000710 1786305564.571303 write 0x002685 thru 0x0091d7 (0x6b53 bytes) 000711 1786305564.573447 trunc from 0x02b395 to 0x00c95b (0x1ea39 bytes) 000713 1786305564.595009 mapwrite 0x020c97 thru 0x021c52 (0xfbc bytes) 000714 1786305564.697507 trunc from 0x021c53 to 0x018903 (0x934f bytes) 000716 1786305564.741539 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18904): Mirror 2: 0x897b4879 (stale) Mirror 3: 0xede2e111 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18904): Mirror 2: 0x897b4879 (stale) Mirror 3: 0xede2e111 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18904: skipped 000717 1786305565.040062 trunc from 0x018904 to 0x03ddca (0x254c7 bytes) 000718 1786305565.056163 read 0x039948 thru 0x03ddca (0x4483 bytes) 000719 1786305565.068215 write 0x02c6aa thru 0x03765f (0xafb6 bytes) 000720 1786305565.096749 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3ddcb): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ddcb): Mirror 2: 0x9d87600d (stale) Mirror 3: 0xb4ed254a (stale) CRC-32 checksum value for chunk [0, 0x3ddcb): Mirror 1: 0x25939bfa Mirror 4: 0x25939bfa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ddcb: skipped 000722 1786305566.067807 mapwrite 0x03c4bb thru 0x03d4a0 (0xfe6 bytes) 000723 1786305566.355593 write 0x029f78 thru 0x02dcff (0x3d88 bytes) 000725 1786305566.382360 trunc from 0x03ddcb to 0x03d5a2 (0x828 bytes) 000728 1786305566.446439 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d5a3): Mirror 3: 0x8ee4cb0a (stale) Mirror 4: 0xeef9e276 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d5a3): Mirror 3: 0x8ee4cb0a (stale) Mirror 4: 0xeef9e276 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d5a3: skipped 000729 1786305566.797876 read 0x02af02 thru 0x037a49 (0xcb48 bytes) 000730 1786305566.807125 read 0x00e344 thru 0x011654 (0x3311 bytes) 000732 1786305566.814993 mapwrite 0x02a74d thru 0x0368d3 (0xc187 bytes) 000733 1786305566.961405 mapread 0x0233f5 thru 0x02cd27 (0x9933 bytes) 000736 1786305566.972077 trunc from 0x03d5a3 to 0x016f77 (0x2662b bytes) 000740 1786305567.013643 MIRROR_RESYNC 3 mirrors 000742 1786305567.937139 trunc from 0x016f78 to 0x0013ea (0x15b8d bytes) 000743 1786305568.105790 read 0x000211 thru 0x0013ea (0x11da bytes) 000748 1786305568.128587 MIRROR_RESYNC 3 mirrors 000749 1786305569.026415 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) 000751 1786305569.310751 mapread 0x000a4d thru 0x0013ea (0x99e bytes) 000752 1786305569.324129 read 0x0003c3 thru 0x0013ea (0x1028 bytes) 000753 1786305569.324887 write 0x00f65e thru 0x012caf (0x3652 bytes) 000755 1786305569.393931 mapread 0x00058b thru 0x009fe3 (0x9a59 bytes) 000758 1786305569.408565 mapwrite 0x0353aa thru 0x039687 (0x42de bytes) 000759 1786305569.531223 mapread 0x0393b9 thru 0x039687 (0x2cf bytes) 000760 1786305569.532159 read 0x0259d3 thru 0x028d6f (0x339d bytes) 000762 1786305569.545488 trunc from 0x039688 to 0x014bc0 (0x24ac7 bytes) 000763 1786305569.560572 trunc from 0x014bc1 to 0x00f0aa (0x5b16 bytes) 000764 1786305569.585415 mapread 0x00a33e thru 0x00f0aa (0x4d6d bytes) 000766 1786305569.588032 mapwrite 0x025039 thru 0x02eb9b (0x9b63 bytes) 000767 1786305569.694515 read 0x00aa1e thru 0x00f268 (0x484b bytes) 000769 1786305569.700833 MIRROR_RESYNC 2 mirrors 000771 1786305570.457547 MIRROR_EXTEND to 3 mirrors 000772 1786305571.231819 MIRROR_RESYNC 3 mirrors 000774 1786305571.581343 mapread 0x014ee0 thru 0x01f7a2 (0xa8c3 bytes) 000775 1786305571.635763 trunc from 0x02eb9c to 0x0212ee (0xd8ad bytes) 000776 1786305571.709662 mapread 0x009280 thru 0x00c43a (0x31bb bytes) 000777 1786305571.723348 mapread 0x002313 thru 0x00c61b (0xa309 bytes) 000778 1786305571.730318 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x212ef): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x212ef): Mirror 4: 0x206ea079 (stale) Mirror 5: 0x206ea079 (stale) CRC-32 checksum value for chunk [0, 0x212ef): Mirror 1: 0x206ea079 Mirror 6: 0x206ea079 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x212ef: skipped 000780 1786305572.812119 read 0x01b9d7 thru 0x01f0c3 (0x36ed bytes) 000781 1786305572.838477 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x212ef): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x212ef): Mirror 4: 0x206ea079 (stale) Mirror 5: 0x206ea079 (stale) CRC-32 checksum value for chunk [0, 0x212ef): Mirror 1: 0x206ea079 Mirror 6: 0x206ea079 Mirror 7: 0x206ea079 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x212ef: skipped 000783 1786305574.175621 read 0x00f308 thru 0x019c4c (0xa945 bytes) 000784 1786305574.184646 trunc from 0x0212ef to 0x021351 (0x63 bytes) 000785 1786305574.285994 write 0x01793b thru 0x01ee04 (0x74ca bytes) 000786 1786305574.297180 mapwrite 0x01c79e thru 0x01cf20 (0x783 bytes) 000787 1786305574.355248 mapread 0x01ef7f thru 0x021351 (0x23d3 bytes) 000788 1786305574.366797 write 0x001f2f thru 0x004123 (0x21f5 bytes) 000790 1786305574.378236 write 0x00a3c3 thru 0x013221 (0x8e5f bytes) 000791 1786305574.384215 read 0x0059c7 thru 0x007fe6 (0x2620 bytes) 000792 1786305574.385318 read 0x01bcd9 thru 0x021351 (0x5679 bytes) 000793 1786305574.390943 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x21352): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21352): Mirror 4: 0x8e52832f (stale) Mirror 5: 0x8e52832f (stale) Mirror 6: 0x8e52832f (stale) Mirror 7: 0x8e52832f (stale) CRC-32 checksum value for chunk [0, 0x21352): Mirror 1: 0xd5e4dea0 Mirror 8: 0xd5e4dea0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21352: skipped 000795 1786305575.259374 mapwrite 0x0355da thru 0x0380b1 (0x2ad8 bytes) 000796 1786305575.419030 read 0x00ad23 thru 0x00d1e3 (0x24c1 bytes) 000799 1786305575.427647 mapwrite 0x033317 thru 0x03987d (0x6567 bytes) 000800 1786305575.488244 write 0x03222c thru 0x039b59 (0x792e bytes) 000801 1786305575.489472 write 0x00ceea thru 0x00e2bc (0x13d3 bytes) 000802 1786305575.493807 mapwrite 0x020ab3 thru 0x0231df (0x272d bytes) 000803 1786305575.535386 read 0x02650e thru 0x03506d (0xeb60 bytes) 000804 1786305575.553645 mapwrite 0x01fbd1 thru 0x02bfa8 (0xc3d8 bytes) 000808 1786305575.644829 trunc from 0x039b5a to 0x032d54 (0x6e05 bytes) 000810 1786305575.666585 trunc from 0x032d55 to 0x0372b2 (0x455e bytes) 000811 1786305575.692835 write 0x03e005 thru 0x03ffff (0x1ffb bytes) 000813 1786305575.700807 MIRROR_RESYNC 6 mirrors 000814 1786305576.768132 write 0x02d4c8 thru 0x02e64a (0x1183 bytes) 000815 1786305576.816339 write 0x020479 thru 0x026630 (0x61b8 bytes) 000820 1786305576.826485 read 0x02fdca thru 0x034c8e (0x4ec5 bytes) 000821 1786305576.836601 trunc from 0x040000 to 0x0304bb (0xfb44 bytes) 000822 1786305576.866516 write 0x00a76c thru 0x00f7d5 (0x506a bytes) 000823 1786305576.888132 MIRROR_RESYNC 6 mirrors 000825 1786305577.989419 write 0x0085f7 thru 0x00f5fb (0x7005 bytes) 000826 1786305578.061356 read 0x0056f1 thru 0x00b6d5 (0x5fe5 bytes) 000827 1786305578.068142 write 0x029bac thru 0x032a4c (0x8ea1 bytes) 000830 1786305578.078262 mapwrite 0x004e49 thru 0x00a3e5 (0x559d bytes) 000831 1786305578.140386 read 0x00f5f8 thru 0x016371 (0x6d7a bytes) 000833 1786305578.159223 trunc from 0x032a4d to 0x0155a3 (0x1d4a9 bytes) 000834 1786305578.184167 MIRROR_RESYNC 6 mirrors 000835 1786305579.126287 trunc from 0x0155a4 to 0x0333eb (0x1de48 bytes) 000836 1786305579.262411 mapread 0x022efa thru 0x02d280 (0xa387 bytes) 000838 1786305579.312973 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x333ec): Mirror 4: 0x3b9344b2 (stale) Mirror 6: 0x3b9344b2 (stale) Mirror 7: 0x3b9344b2 (stale) Mirror 8: 0x3b9344b2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x333ec): Mirror 4: 0x3b9344b2 (stale) Mirror 6: 0x3b9344b2 (stale) Mirror 7: 0x3b9344b2 (stale) Mirror 8: 0x3b9344b2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x333ec: skipped 000841 1786305579.716827 MIRROR_RESYNC 5 mirrors 000842 1786305580.626137 MIRROR_RESYNC 5 mirrors 000847 1786305581.003074 read 0x0098d3 thru 0x00f647 (0x5d75 bytes) 000848 1786305581.012438 read 0x002dcd thru 0x00fa03 (0xcc37 bytes) 000850 1786305581.034337 mapwrite 0x00848c thru 0x00860f (0x184 bytes) 000851 1786305581.228986 read 0x020bc5 thru 0x022f59 (0x2395 bytes) 000854 1786305581.239754 write 0x0286d7 thru 0x030049 (0x7973 bytes) 000856 1786305581.244304 mapread 0x01940a thru 0x024c9d (0xb894 bytes) 000860 1786305581.248078 trunc from 0x0333ec to 0x00c4b8 (0x26f33 bytes) 000861 1786305581.280342 write 0x037000 thru 0x03dd78 (0x6d79 bytes) 000862 1786305581.292268 MIRROR_RESYNC 5 mirrors 000863 1786305583.109978 write 0x010f47 thru 0x02056c (0xf626 bytes) 000865 1786305583.194887 trunc from 0x03dd79 to 0x033442 (0xa936 bytes) 000866 1786305583.224602 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33443): Mirror 4: 0x46289167 (stale) Mirror 6: 0x46289167 (stale) Mirror 8: 0x46289167 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33443): Mirror 4: 0x46289167 (stale) Mirror 6: 0x46289167 (stale) Mirror 8: 0x46289167 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33443: skipped 000867 1786305583.568511 trunc from 0x033443 to 0x015ef7 (0x1d54b bytes) 000868 1786305583.600418 read 0x00bd2c thru 0x015a87 (0x9d5c bytes) 000871 1786305583.618465 write 0x03e07c thru 0x03ffff (0x1f84 bytes) truncating to largest ever: 0x3fe90 000872 1786305583.620735 trunc from 0x040000 to 0x03fe8f (0x170 bytes) 000873 1786305583.641235 mapwrite 0x036b8e thru 0x03ffff (0x9472 bytes) 000874 1786305583.807469 write 0x0096c7 thru 0x00da87 (0x43c1 bytes) 000876 1786305583.822187 trunc from 0x040000 to 0x008096 (0x37f69 bytes) 000878 1786305583.851456 trunc from 0x008097 to 0x00977e (0x16e8 bytes) 000880 1786305583.888787 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x977f): Mirror 4: 0xbfcb6291 (stale) Mirror 8: 0xbfcb6291 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x977f): Mirror 4: 0xbfcb6291 (stale) Mirror 8: 0xbfcb6291 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x977f: skipped 000881 1786305584.178727 mapwrite 0x01755f thru 0x01b411 (0x3eb3 bytes) 000882 1786305584.291767 mapread 0x003ec2 thru 0x0133b7 (0xf4f6 bytes) 000883 1786305584.323932 read 0x00456d thru 0x00aada (0x656e bytes) 000884 1786305584.325954 write 0x02f1bc thru 0x039c99 (0xaade bytes) 000886 1786305584.341424 trunc from 0x039c9a to 0x007ea4 (0x31df5 bytes) 000888 1786305584.409819 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x7ea5): Mirror 8: 0x30bd7722 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7ea5): Mirror 8: 0x30bd7722 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7ea5: skipped 000889 1786305584.801964 write 0x02c1d0 thru 0x031c0d (0x5a3e bytes) 000890 1786305584.813026 mapwrite 0x005d57 thru 0x0132e4 (0xd58e bytes) 000891 1786305584.917399 write 0x022f1a thru 0x02c3a2 (0x9489 bytes) 000894 1786305584.929302 mapwrite 0x0247cb thru 0x031ad4 (0xd30a bytes) 000896 1786305584.987575 mapwrite 0x01b071 thru 0x02823f (0xd1cf bytes) 000897 1786305585.048461 mapwrite 0x0151b1 thru 0x01ef4f (0x9d9f bytes) 000898 1786305585.112252 write 0x010aae thru 0x0117b9 (0xd0c bytes) 000899 1786305585.113252 read 0x026a33 thru 0x031c0d (0xb1db bytes) 000900 1786305585.114608 mapwrite 0x033863 thru 0x03c65d (0x8dfb bytes) 000901 1786305585.206378 MIRROR_RESYNC 2 mirrors 000902 1786305585.917442 mapread 0x015160 thru 0x01ca14 (0x78b5 bytes) 000905 1786305585.956276 write 0x01597f thru 0x01e9e1 (0x9063 bytes) 000907 1786305586.115168 write 0x000875 thru 0x0010eb (0x877 bytes) 000909 1786305586.140288 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) 000910 1786305586.311265 mapread 0x01ce47 thru 0x0281a0 (0xb35a bytes) 000911 1786305586.367799 read 0x017552 thru 0x024ad3 (0xd582 bytes) 000912 1786305586.369023 mapread 0x037c58 thru 0x03c65d (0x4a06 bytes) 000913 1786305586.377975 read 0x02e5a3 thru 0x03237a (0x3dd8 bytes) 000915 1786305586.379196 read 0x02f505 thru 0x039263 (0x9d5f bytes) 000917 1786305586.380436 mapwrite 0x026879 thru 0x028371 (0x1af9 bytes) 000919 1786305586.455160 read 0x036d95 thru 0x0377e9 (0xa55 bytes) 000924 1786305586.457285 trunc from 0x03c65e to 0x026a0a (0x15c53 bytes) 000925 1786305586.484640 trunc from 0x026a0b to 0x01d7b8 (0x9252 bytes) 000926 1786305586.505740 trunc from 0x01d7b9 to 0x01f754 (0x1f9c bytes) 000929 1786305586.519586 trunc from 0x01f755 to 0x01c2af (0x34a5 bytes) 000933 1786305586.534533 mapwrite 0x022ad3 thru 0x025326 (0x2854 bytes) 000936 1786305586.620147 trunc from 0x025327 to 0x02c968 (0x7642 bytes) 000937 1786305586.641790 write 0x022fde thru 0x025d9d (0x2dc0 bytes) 000938 1786305586.643252 trunc from 0x02c969 to 0x008faa (0x239be bytes) 000941 1786305586.669049 trunc from 0x008fab to 0x008325 (0xc85 bytes) 000942 1786305586.688534 mapread 0x0079e2 thru 0x008325 (0x944 bytes) 000943 1786305586.690670 write 0x00cc7f thru 0x01c543 (0xf8c5 bytes) 000945 1786305586.694093 mapread 0x0132c1 thru 0x01c543 (0x9283 bytes) 000946 1786305586.700807 read 0x00016c thru 0x005de5 (0x5c7a bytes) 000947 1786305586.701835 trunc from 0x01c544 to 0x0092cb (0x13278 bytes) 000948 1786305586.725412 mapread 0x00928a thru 0x0092cb (0x42 bytes) 000951 1786305586.744061 mapread 0x003cf3 thru 0x0092cb (0x55d9 bytes) 000953 1786305586.747032 trunc from 0x0092cc to 0x02b6ad (0x223e2 bytes) 000958 1786305586.769203 trunc from 0x02b6ae to 0x021855 (0x9e58 bytes) 000959 1786305586.785415 mapread 0x00662e thru 0x015877 (0xf24a bytes) 000960 1786305586.809694 read 0x00b765 thru 0x01b2f5 (0xfb91 bytes) 000964 1786305586.814264 read 0x00e277 thru 0x01c2bd (0xe047 bytes) 000972 1786305586.823902 read 0x006d57 thru 0x0111f7 (0xa4a1 bytes) 000975 1786305586.827605 mapread 0x00d416 thru 0x017f13 (0xaafe bytes) 000976 1786305586.830037 mapread 0x001caa thru 0x004e98 (0x31ef bytes) 000977 1786305586.831956 mapread 0x019e90 thru 0x01c991 (0x2b02 bytes) 000978 1786305586.833089 mapread 0x01523b thru 0x01f214 (0x9fda bytes) 000979 1786305586.835326 trunc from 0x021856 to 0x023806 (0x1fb1 bytes) 000981 1786305586.857229 mapread 0x0082df thru 0x016630 (0xe352 bytes) 000984 1786305586.863209 mapread 0x00136b thru 0x00d87a (0xc510 bytes) 000985 1786305586.866664 mapread 0x00a5fb thru 0x00f114 (0x4b1a bytes) 000986 1786305586.868968 mapread 0x001b95 thru 0x00604d (0x44b9 bytes) 000987 1786305586.871755 read 0x00a329 thru 0x00b162 (0xe3a bytes) 000989 1786305586.874449 write 0x023682 thru 0x027258 (0x3bd7 bytes) 000991 1786305586.887230 trunc from 0x027259 to 0x037b06 (0x108ae bytes) 000992 1786305586.907049 mapread 0x02b711 thru 0x02cc36 (0x1526 bytes) 000994 1786305586.920262 write 0x02dd09 thru 0x037c61 (0x9f59 bytes) 000997 1786305586.925995 mapread 0x02371f thru 0x032ba1 (0xf483 bytes) 000998 1786305586.928025 mapwrite 0x00b710 thru 0x016931 (0xb222 bytes) 000999 1786305587.002270 trunc from 0x037c62 to 0x03cc7a (0x5019 bytes) All operations completed A-OK! PASS 70a (77s) == sanity-flr test 71: check mirror extend parent fid ==== 15:59:52 (1786305592) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (5s) == sanity-flr test 72: check append on FLR file ========== 15:59:57 (1786305597) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x330:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x314:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.206015 s, 10.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x330:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x331:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x314:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167031 s, 6.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x330:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x331:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x315:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x314:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x330:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x331:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x315:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x314:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x332:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 16:00:07 (1786305607) Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre2 Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3022592 .. done Extending file size to 2535936 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5803296 .. failed Extending file size to 7463616 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7529792 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6124416 .. Extending file size to 7445344 .. Extending file size to 8733888 .. Extending file size to 7649216 .. Extending file size to 8962144 .. Extending file size to 8038848 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8342624 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8906240 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 57520 57522 57523 57525 57527 192.168.201.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg110-client.virtnet /mnt/lustre2 (opts:) 192.168.201.110@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg110-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 16:01:30 (1786305690) Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre2 Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6294080 .. Extending file size to 3457536 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6248480 .. Extending file size to 7239392 .. Extending file size to 8837280 .. Extending file size to 7877920 .. Extending file size to 8460288 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8628672 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8861120 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8884288 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8904256 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 59087 59089 59090 59091 59093 192.168.201.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg110-client.virtnet /mnt/lustre2 (opts:) 192.168.201.110@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg110-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (75s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 16:02:45 (1786305765) Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.113069 s, 18.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.26258 s, 8.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xec3e7692 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg110-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 16:03:04 (1786305784) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.119414 s, 17.6 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 16:03:11 (1786305791) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x227:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32b:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32b:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32c:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x347:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 16:03:19 (1786305799) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.239638 s, 8.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x348:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x349:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 16:03:27 (1786305807) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.224154 s, 9.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34b:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 16:03:37 (1786305817) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.24779 s, 8.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x330:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x330:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34d:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 16:04:10 (1786305850) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg110-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185986 s, 5.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x331:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x331:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x332:0x0] } PASS 204d (59s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 16:05:09 (1786305909) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg110-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x350:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x351:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.448855 s, 4.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x350:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x351:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x333:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x350:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x351:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x333:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (57s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 16:06:06 (1786305966) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg110-server mds-ost sync done. Creating new pool test_204f oleg110-server: Pool lustre.test_204f created Adding targets to pool oleg110-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg110-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.24896 s, 8.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x352:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x334:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x352:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x334:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg110-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg110-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg110-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 16:07:26 (1786306046) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 16:07:33 (1786306053) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 16:07:41 (1786306061) oleg110-server: Pool lustre.test_206 created Adding targets to pool oleg110-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg110-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg110-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg110-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg110-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 16:08:08 (1786306088) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999383 s, 10.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745852 s, 14.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xd14b9666 Mirror 3: 0xd14b9666 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 16:08:19 (1786306099) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 16:08:23 (1786306103) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 16:08:26 (1786306106) Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.145277 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1810000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xd70000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xd70000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xd70000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xb00000 Read this (4 bytes): BBBB 192.168.201.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg110-client.virtnet /mnt/lustre2 (opts:) PASS 209a (35s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 16:09:01 (1786306141) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0713854 s, 115 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 16:09:10 (1786306150) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327761 s, 32.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x24c lmm_fid: [0x200000404:0x24c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x24c lmm_fid: [0x200000404:0x24c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x33e:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834508 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-09 16:09:12.000000000 -0400 Modify: 2026-08-09 16:09:12.000000000 -0400 Change: 2026-08-09 16:09:14.000000000 -0400 Birth: 2026-08-09 16:09:13.000000000 -0400 PASS 210a (11s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 16:09:21 (1786306161) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 16:09:25 (1786306165) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.704804 s, 58.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0768341 s, 53.3 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 9 16:09 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x24e:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 9 16:09 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 16:09:33 (1786306173) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (6s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 16:09:40 (1786306180) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1687 sec =========== 16:09:50 (1786306190) === sanity-flr: start cleanup 16:09:52 (1786306192) === === sanity-flr: finish cleanup 16:09:58 (1786306198) ===