-----============= acceptance-small: sanity-flr ============----- Thu Jul 30 01:17:23 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg247-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 01:17:51 (1785388671) === oleg247-client.virtnet: executing check_config_client /mnt/lustre oleg247-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg247-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff996521645000.idle_timeout=debug osc.lustre-OST0001-osc-ffff996521645000.idle_timeout=debug disable quota as required oleg247-server: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 01:18:42 (1785388722) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7616] == sanity-flr test 0a: lfs mirror create with -N option == 01:18:46 (1785388726) PASS 0a (28s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:19:14 (1785388754) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (5s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:19:19 (1785388759) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (5s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:19:24 (1785388764) PASS 0d (39s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:20:03 (1785388803) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (7s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:20:11 (1785388811) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (6s) == sanity-flr test 0g: lfs mirror create flags support === 01:20:17 (1785388817) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff996521645000.stats=clear osc.lustre-OST0001-osc-ffff996521645000.stats=clear PASS 0g (11s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:20:28 (1785388828) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (16s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:20:45 (1785388845) PASS 0j (16s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:21:01 (1785388861) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (6s) == sanity-flr test 0m: mirror extend with -N= option ===== 01:21:08 (1785388868) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (16s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:21:24 (1785388884) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 01:21:35 (1785388895) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:21:44 (1785388904) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:21:48 (1785388908) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:21:57 (1785388917) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (9s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:22:09 (1785388929) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:22:17 (1785388937) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 2.72131 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209517 s, 5.0 MB/s PASS 21 (15s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:22:32 (1785388952) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.344092 s, 24.4 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 01:22:43 (1785388963) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:22:55 (1785388975) Stopping /mnt/lustre-ost1 (opts:) on oleg247-server oleg247-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff996521645000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff996521645000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg247-server: oleg247-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg247-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg247-server oleg247-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff996521645000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff996521645000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg247-server: oleg247-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg247-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (90s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:24:25 (1785389065) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.463808 s, 15.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg247-server oleg247-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff996521645000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff996521645000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg247-server: oleg247-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg247-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (52s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:25:17 (1785389117) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg247-server oleg247-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff996521645000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff996521645000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg247-server: oleg247-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg247-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg247-server oleg247-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff996521645000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff996521645000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg247-server: oleg247-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg247-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (109s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:27:06 (1785389226) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 6.98466 s, 7.5 MB/s -rw-r--r-- 1 root root 52428800 Jul 30 01:27 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.368s user 0m0.000s sys 0m0.135s Stopping /mnt/lustre-ost1 (opts:) on oleg247-server oleg247-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff996521645000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff996521645000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.093s user 0m0.001s sys 0m0.169s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg247-server: oleg247-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg247-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg247-server oleg247-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff996521645000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff996521645000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.322s user 0m0.001s sys 0m0.156s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg247-server: oleg247-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg247-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg247-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff996521645000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff996521645000.ost_server_uuid in FULL state after 0 sec PASS 33b (125s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 01:29:11 (1785389351) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (5s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 01:29:16 (1785389356) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 01:29:20 (1785389360) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 01:29:24 (1785389364) PASS 35 (11s) == sanity-flr test 36a: write to mirrored files ========== 01:29:35 (1785389375) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 1.2784 s, 9.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (20s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:29:56 (1785389396) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489195 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612971 s, 17.1 MB/s PASS 36b (10s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:30:07 (1785389407) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:30:17 (1785389417) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00281534 s, 364 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00099051 s, 1.0 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000990008 s, 1.0 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00100076 s, 1.0 MB/s flv=10 olv=10 PASS 36d (15s) == sanity-flr test 37: mirror I/O API verification ======= 01:30:32 (1785389432) create test files with size 15734731 .. e8f5949e7f1829f767f161d64d465ae1 - 7093a9e83635ebd213e35ad972e8261e - e59e148ea4e0fe21ae2d36c129563553 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg247-client.virtnet /mnt/lustre (opts:) Starting client: oleg247-client.virtnet: -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre PASS 37 (34s) == sanity-flr test 38: resync ============================ 01:31:07 (1785389467) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (23s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:31:29 (1785389489) PASS 39 (10s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:31:39 (1785389499) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103392 s, 10.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654634 s, 16.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (13s) == sanity-flr test 41: lfs mirror resync check =========== 01:31:52 (1785389512) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.114987 s, 18.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.438407 s, 9.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.369819 s, 5.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (24s) == sanity-flr test 42: lfs mirror verify ================= 01:32:16 (1785389536) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (5s) == sanity-flr test 43a: mirror pick on write ============= 01:32:21 (1785389541) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (5s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 01:32:26 (1785389546) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709547 s, 14.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (11s) == sanity-flr test 44a: lfs mirror split check =========== 01:32:37 (1785389557) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (5s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 01:32:42 (1785389562) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc8:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 01:32:55 (1785389575) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.406267 s, 25.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg247-client.virtnet /mnt/lustre (opts:) Starting client: oleg247-client.virtnet: -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (20s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 01:33:15 (1785389595) llite.lustre-ffff996503223800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.284704 s, 36.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg247-client.virtnet /mnt/lustre (opts:) Starting client: oleg247-client.virtnet: -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff99652edc5000.xattr_cache=1 PASS 44e (18s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 01:33:33 (1785389613) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (49s) == sanity-flr test 46: Verify setstripe --copy option ==== 01:34:22 (1785389662) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (57s) == sanity-flr test 47: Verify mirror obj alloc =========== 01:35:19 (1785389719) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (6s) == sanity-flr test 48: Verify snapshot mirror ============ 01:35:26 (1785389726) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.416054 s, 7.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.24837 s, 12.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x519f1e5f (nosync,stale) Mirror 2: 0x519f1e5f (nosync,stale) ** verify mirror 2 doesn't change original checksum: a4cce471cf163719fe18d478e133aebf - mirror 1 checksum: e0121c4262c3a5ff6be17b27ab9aa4f8 - mirror 2 checksum: a4cce471cf163719fe18d478e133aebf - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-30T01:35:30-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-30T01:35:30-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: a4cce471cf163719fe18d478e133aebf - mirror 1 checksum: e0121c4262c3a5ff6be17b27ab9aa4f8 - mirror 2 checksum: e0121c4262c3a5ff6be17b27ab9aa4f8 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-30T01:35:30-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-30T01:35:30-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } PASS 48 (16s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 01:35:44 (1785389744) Failing mds1 on oleg247-server Stopping /mnt/lustre-mds1 (opts:) on oleg247-server 01:35:54 (1785389754) shut down facet: mds1 facet_host: oleg247-server facet_failover_host: oleg247-server Failover mds1 to oleg247-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg247-server: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-MDT0000 01:36:30 (1785389790) targets are mounted 01:36:30 (1785389790) facet_failover done oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651249 s, 16.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } PASS 50A (69s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 01:36:54 (1785389814) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (5s) == sanity-flr test 50b: mirror rsync handles sparseness == 01:37:00 (1785389820) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (5s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 01:37:05 (1785389825) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00534151 s, 3.1 MB/s PASS 50c (12s) == sanity-flr test 50d: mirror rsync keep holes ========== 01:37:17 (1785389837) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 01:37:22 (1785389842) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0073558 s, 557 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.184045 s, 22.3 kB/s PASS 60a (11s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 01:37:33 (1785389853) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (26s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 01:38:00 (1785389880) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (33s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 01:38:33 (1785389913) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (28s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:39:01 (1785389941) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (28s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:39:29 (1785389969) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (10s) == sanity-flr test 70: mirror create and split race ====== 01:39:39 (1785389979) mirror create pid 54296 mirror split pid 54297 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5070: 54296 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5070: 54297 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (70s) == sanity-flr test 70a: flr mode fsx test ================ 01:40:49 (1785390049) Seed set to 52 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1785390053.237925 write 0x01ca2c thru 0x029332 (0xc907 bytes) 000003 1785390053.256619 mapwrite 0x0237ee thru 0x029bec (0x63ff bytes) truncating to largest ever: 0x1a932 000005 1785390053.370302 trunc from 0x029bed to 0x01a931 (0xf2bb bytes) 000006 1785390053.456578 MIRROR_EXTEND to 2 mirrors 000007 1785390054.232386 read 0x0100ca thru 0x016c21 (0x6b58 bytes) 000008 1785390054.275399 MIRROR_EXTEND to 3 mirrors 000017 1785390055.820143 MIRROR_RESYNC 3 mirrors 000018 1785390056.704966 read 0x005251 thru 0x00c47b (0x722b bytes) 000020 1785390056.717107 write 0x02874b thru 0x0375c6 (0xee7c bytes) 000021 1785390056.955450 read 0x01aeeb thru 0x027957 (0xca6d bytes) 000022 1785390057.088356 mapread 0x01e683 thru 0x02def6 (0xf874 bytes) 000023 1785390057.089692 mapwrite 0x00b7aa thru 0x014d4d (0x95a4 bytes) 000029 1785390057.279647 mapread 0x02c488 thru 0x034edb (0x8a54 bytes) 000030 1785390057.285215 mapwrite 0x024820 thru 0x033435 (0xec16 bytes) 000032 1785390057.430820 mapread 0x032e61 thru 0x0375c6 (0x4766 bytes) 000033 1785390057.431811 read 0x0302d7 thru 0x035c9c (0x59c6 bytes) truncating to largest ever: 0x3d08e 000036 1785390057.441739 trunc from 0x0375c7 to 0x03d08d (0x5ac7 bytes) 000037 1785390057.468093 mapread 0x031cf5 thru 0x03a2d0 (0x85dc bytes) 000038 1785390057.488407 mapwrite 0x007a51 thru 0x0152be (0xd86e bytes) 000039 1785390057.614318 MIRROR_RESYNC 3 mirrors 000042 1785390059.014236 mapread 0x00a1d8 thru 0x015d48 (0xbb71 bytes) 000043 1785390059.085316 mapwrite 0x021a78 thru 0x02e338 (0xc8c1 bytes) 000044 1785390059.397181 mapwrite 0x01e65a thru 0x027267 (0x8c0e bytes) 000046 1785390059.477069 read 0x03b2e6 thru 0x03d08d (0x1da8 bytes) 000047 1785390059.500308 MIRROR_RESYNC 3 mirrors 000048 1785390060.488118 mapread 0x00cc33 thru 0x01745c (0xa82a bytes) 000049 1785390060.511752 MIRROR_RESYNC 3 mirrors 000050 1785390060.805580 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x155:0x0]: Invalid argument (22) 000051 1785390061.037060 MIRROR_EXTEND to 3 mirrors 000053 1785390061.886554 MIRROR_RESYNC 3 mirrors 000054 1785390062.212090 mapread 0x0286ee thru 0x02ba44 (0x3357 bytes) 000057 1785390062.228524 trunc from 0x03d08e to 0x03b076 (0x2017 bytes) 000058 1785390062.303366 write 0x02411b thru 0x028cf2 (0x4bd8 bytes) 000060 1785390062.314132 mapwrite 0x03c0af thru 0x03cd3f (0xc91 bytes) 000061 1785390062.401990 mapread 0x002f0c thru 0x007526 (0x461b bytes) 000063 1785390062.459096 write 0x0046dc thru 0x00ef6c (0xa891 bytes) 000064 1785390062.467804 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3cd40): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd40): Mirror 3: 0x85c368b0 (stale) Mirror 4: 0x85c368b0 (stale) CRC-32 checksum value for chunk [0, 0x3cd40): Mirror 1: 0x87ccddd2 Mirror 5: 0x87ccddd2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd40: skipped 000065 1785390063.762532 write 0x0140a0 thru 0x01b2d0 (0x7231 bytes) 000067 1785390063.851346 mapread 0x033950 thru 0x03cd3f (0x93f0 bytes) 000069 1785390063.870066 read 0x02fd16 thru 0x03c23e (0xc529 bytes) 000070 1785390063.870804 read 0x0098a8 thru 0x0176bf (0xde18 bytes) 000072 1785390063.880490 read 0x0211c9 thru 0x02bf64 (0xad9c bytes) 000075 1785390063.886272 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x155:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cd40): Mirror 4: 0x85c368b0 (stale) Mirror 5: 0x87ccddd2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd40): Mirror 4: 0x85c368b0 (stale) Mirror 5: 0x87ccddd2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd40: skipped 000076 1785390064.143521 write 0x0292f7 thru 0x036308 (0xd012 bytes) 000078 1785390064.154859 trunc from 0x03cd40 to 0x017b8c (0x251b3 bytes) 000080 1785390064.170404 mapread 0x01083b thru 0x015df7 (0x55bd bytes) 000081 1785390064.193832 MIRROR_RESYNC 3 mirrors 000082 1785390064.890008 trunc from 0x017b8d to 0x0104db (0x76b1 bytes) 000085 1785390065.084870 read 0x00031c thru 0x006237 (0x5f1c bytes) 000086 1785390065.094905 read 0x00ba11 thru 0x0104db (0x4acb bytes) 000087 1785390065.105056 mapread 0x00821e thru 0x00b355 (0x3138 bytes) 000088 1785390065.106599 mapread 0x002d52 thru 0x0104db (0xd78a bytes) 000089 1785390065.114929 MIRROR_RESYNC 3 mirrors 000090 1785390065.875767 mapread 0x00c21b thru 0x0104db (0x42c1 bytes) 000092 1785390065.899323 read 0x00e824 thru 0x0104db (0x1cb8 bytes) 000093 1785390065.901196 mapwrite 0x00898e thru 0x008c1f (0x292 bytes) 000095 1785390066.157245 mapwrite 0x00eecd thru 0x0140c6 (0x51fa bytes) 000096 1785390066.364990 mapwrite 0x012205 thru 0x01bc9f (0x9a9b bytes) 000097 1785390066.527717 write 0x004d1e thru 0x0121c4 (0xd4a7 bytes) 000099 1785390066.546059 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1bca0): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bca0): Mirror 4: 0x60ee2e3b (stale) Mirror 5: 0x60ee2e3b (stale) CRC-32 checksum value for chunk [0, 0x1bca0): Mirror 1: 0xae5d6949 Mirror 6: 0xae5d6949 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bca0: skipped 000100 1785390068.052849 write 0x006625 thru 0x00f266 (0x8c42 bytes) 000102 1785390068.158035 mapwrite 0x03c21e thru 0x03ffff (0x3de2 bytes) 000104 1785390068.331949 trunc from 0x040000 to 0x01191b (0x2e6e4 bytes) 000105 1785390068.380878 mapread 0x002780 thru 0x0050d1 (0x2952 bytes) 000107 1785390068.392544 read 0x00dbad thru 0x01191b (0x3d6f bytes) 000110 1785390068.399904 mapwrite 0x010b6d thru 0x01b841 (0xacd5 bytes) 000113 1785390068.640051 trunc from 0x01b842 to 0x01de19 (0x25d8 bytes) 000114 1785390068.696580 mapwrite 0x012b63 thru 0x019dc5 (0x7263 bytes) 000116 1785390068.754542 write 0x03844e thru 0x039346 (0xef9 bytes) 000118 1785390068.756465 read 0x02f884 thru 0x039346 (0x9ac3 bytes) 000120 1785390068.794604 mapwrite 0x033288 thru 0x03c8ee (0x9667 bytes) 000121 1785390068.893928 read 0x02c6d7 thru 0x035ea0 (0x97ca bytes) 000122 1785390068.895913 mapread 0x03bcc5 thru 0x03c8ee (0xc2a bytes) 000123 1785390068.901219 mapread 0x002b7d thru 0x012279 (0xf6fd bytes) 000126 1785390068.906823 trunc from 0x03c8ef to 0x0191e9 (0x23705 bytes) 000127 1785390068.919616 read 0x005a51 thru 0x008a99 (0x3049 bytes) 000129 1785390068.931697 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x191ea): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x191ea): Mirror 4: 0x7b59a3b7 (stale) Mirror 5: 0x7b59a3b7 (stale) Mirror 6: 0x2d33baab (stale) CRC-32 checksum value for chunk [0, 0x191ea): Mirror 1: 0x6662cdc Mirror 7: 0x6662cdc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x191ea: skipped 000130 1785390070.051559 read 0x00c378 thru 0x0116ac (0x5335 bytes) 000132 1785390070.095066 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x155:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x191ea): Mirror 4: 0x7b59a3b7 (stale) Mirror 6: 0x2d33baab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x191ea): Mirror 4: 0x7b59a3b7 (stale) Mirror 6: 0x2d33baab (stale) CRC-32 checksum value for chunk [0, 0x191ea): Mirror 1: 0x6662cdc Mirror 7: 0x6662cdc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x191ea: skipped 000133 1785390070.824118 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x155:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x191ea): Mirror 4: 0x7b59a3b7 (stale) Mirror 6: 0x2d33baab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x191ea): Mirror 4: 0x7b59a3b7 (stale) Mirror 6: 0x2d33baab (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x191ea: skipped 000134 1785390071.157512 read 0x002b1d thru 0x00d613 (0xaaf7 bytes) 000136 1785390071.167284 trunc from 0x0191ea to 0x014b36 (0x46b3 bytes) 000138 1785390071.195797 write 0x005a5f thru 0x011c03 (0xc1a5 bytes) 000141 1785390071.207671 mapwrite 0x015d20 thru 0x0167b4 (0xa95 bytes) 000143 1785390071.303780 MIRROR_RESYNC 3 mirrors 000144 1785390072.116235 write 0x00327f thru 0x00e985 (0xb707 bytes) 000147 1785390072.323994 trunc from 0x0167b5 to 0x018eef (0x273b bytes) 000148 1785390072.408144 write 0x03d742 thru 0x03ffff (0x28be bytes) 000149 1785390072.436100 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xb844dbc3 (stale) Mirror 6: 0xb844dbc3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc8616940 Mirror 7: 0xc8616940 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000152 1785390074.295913 write 0x02943e thru 0x0381d9 (0xed9c bytes) 000154 1785390074.511465 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xb844dbc3 (stale) Mirror 6: 0xb844dbc3 (stale) Mirror 7: 0xc8616940 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdb205a31 Mirror 8: 0xdb205a31 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000155 1785390076.166237 read 0x005826 thru 0x008842 (0x301d bytes) 000159 1785390076.222722 mapwrite 0x00afab thru 0x0175c4 (0xc61a bytes) 000161 1785390076.515619 read 0x03c83a thru 0x03ffff (0x37c6 bytes) 000164 1785390076.532781 mapwrite 0x004b84 thru 0x00dac2 (0x8f3f bytes) 000168 1785390076.626654 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xb844dbc3 (stale) Mirror 6: 0xb844dbc3 (stale) Mirror 7: 0xc8616940 (stale) Mirror 8: 0xdb205a31 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa47536d1 Mirror 9: 0xa47536d1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000169 1785390078.529712 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xb844dbc3 (stale) Mirror 6: 0xb844dbc3 (stale) Mirror 7: 0xc8616940 (stale) Mirror 8: 0xdb205a31 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa47536d1 Mirror 9: 0xa47536d1 Mirror 10: 0xa47536d1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000170 1785390079.992520 write 0x01707f thru 0x0265b1 (0xf533 bytes) 000171 1785390080.062819 write 0x03aea1 thru 0x03e939 (0x3a99 bytes) 000172 1785390080.073791 mapread 0x02ab7c thru 0x02f201 (0x4686 bytes) 000173 1785390080.112033 mapread 0x01ad51 thru 0x026c19 (0xbec9 bytes) 000176 1785390080.129328 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x155:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xb844dbc3 (stale) Mirror 6: 0xb844dbc3 (stale) Mirror 7: 0xc8616940 (stale) Mirror 9: 0xa47536d1 (stale) Mirror 10: 0xa47536d1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xb844dbc3 (stale) Mirror 6: 0xb844dbc3 (stale) Mirror 7: 0xc8616940 (stale) Mirror 9: 0xa47536d1 (stale) Mirror 10: 0xa47536d1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000179 1785390080.637711 mapread 0x00c30f thru 0x00ff29 (0x3c1b bytes) 000184 1785390080.669394 mapread 0x005331 thru 0x00ecf0 (0x99c0 bytes) 000185 1785390080.691831 trunc from 0x040000 to 0x0005d0 (0x3fa2f bytes) 000187 1785390080.779058 read 0x00022f thru 0x0005d0 (0x3a2 bytes) 000188 1785390080.783950 mapread 0x000003 thru 0x0005d0 (0x5ce bytes) 000189 1785390080.785057 mapwrite 0x0177b1 thru 0x01c739 (0x4f89 bytes) 000190 1785390080.877854 mapread 0x009ccd thru 0x00e854 (0x4b88 bytes) 000191 1785390080.894935 write 0x02d9f4 thru 0x02ee15 (0x1422 bytes) 000195 1785390080.897570 trunc from 0x02ee16 to 0x02ed18 (0xfd bytes) 000196 1785390080.911113 read 0x0298cd thru 0x02bcb7 (0x23eb bytes) 000199 1785390080.924994 write 0x0096cf thru 0x00a17f (0xab1 bytes) 000203 1785390080.931775 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x155:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ed19): Mirror 4: 0xbd9b391f (stale) Mirror 6: 0xbd9b391f (stale) Mirror 7: 0x8909ce8d (stale) Mirror 10: 0x193ee376 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ed19): Mirror 4: 0xbd9b391f (stale) Mirror 6: 0xbd9b391f (stale) Mirror 7: 0x8909ce8d (stale) Mirror 10: 0x193ee376 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ed19: skipped 000205 1785390081.333661 read 0x01ef62 thru 0x0267ae (0x784d bytes) 000207 1785390081.351859 MIRROR_RESYNC 5 mirrors 000208 1785390083.369821 write 0x004907 thru 0x00e901 (0x9ffb bytes) 000209 1785390083.617701 trunc from 0x02ed19 to 0x004d12 (0x2a006 bytes) 000210 1785390083.672142 write 0x00171e thru 0x00bfce (0xa8b1 bytes) 000213 1785390083.698654 mapread 0x009dde thru 0x00bfce (0x21f1 bytes) 000214 1785390083.699786 write 0x014d5a thru 0x0194be (0x4765 bytes) 000216 1785390083.733325 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x155:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x194bf): Mirror 4: 0x7b1add33 (stale) Mirror 6: 0x7b1add33 (stale) Mirror 7: 0x7b1add33 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x194bf): Mirror 4: 0x7b1add33 (stale) Mirror 6: 0x7b1add33 (stale) Mirror 7: 0x7b1add33 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x194bf: skipped 000217 1785390084.555797 write 0x01228b thru 0x0194a4 (0x721a bytes) 000218 1785390084.610783 write 0x02d1d9 thru 0x02ecc7 (0x1aef bytes) 000219 1785390084.618099 trunc from 0x02ecc8 to 0x005130 (0x29b97 bytes) 000221 1785390084.666781 mapread 0x0028cc thru 0x003abb (0x11f0 bytes) 000225 1785390084.705882 trunc from 0x005131 to 0x001f58 (0x31d8 bytes) 000226 1785390084.772123 mapread 0x00174b thru 0x001f58 (0x80e bytes) 000228 1785390084.804580 mapwrite 0x00d913 thru 0x010578 (0x2c66 bytes) 000232 1785390085.076020 mapread 0x00f0e6 thru 0x010578 (0x1493 bytes) 000233 1785390085.077105 read 0x00dd2d thru 0x010578 (0x284c bytes) 000234 1785390085.083781 mapwrite 0x0378b2 thru 0x03c4c9 (0x4c18 bytes) 000235 1785390085.296380 MIRROR_RESYNC 4 mirrors 000237 1785390087.555174 MIRROR_EXTEND to 5 mirrors 000238 1785390089.537466 mapwrite 0x0117ca thru 0x01dc53 (0xc48a bytes) 000240 1785390090.040131 trunc from 0x03c4ca to 0x01a9eb (0x21ade bytes) 000246 1785390090.106751 write 0x03b828 thru 0x03ffff (0x47d8 bytes) 000247 1785390090.108007 write 0x03ac83 thru 0x03ffff (0x537d bytes) 000251 1785390090.127590 mapwrite 0x015e16 thru 0x016a01 (0xbec bytes) 000252 1785390090.297714 mapwrite 0x01daf3 thru 0x01e5c4 (0xad2 bytes) 000256 1785390090.432734 trunc from 0x040000 to 0x022803 (0x1d7fc bytes) 000261 1785390090.483504 write 0x012fbf thru 0x0151d2 (0x2214 bytes) 000262 1785390090.493030 MIRROR_RESYNC 5 mirrors 000263 1785390091.979574 MIRROR_RESYNC 5 mirrors 000265 1785390092.716255 write 0x01140e thru 0x013364 (0x1f57 bytes) 000266 1785390092.866967 trunc from 0x022804 to 0x00acee (0x17b15 bytes) 000268 1785390092.920875 mapread 0x00893d thru 0x00acee (0x23b2 bytes) 000270 1785390092.948722 read 0x0041c6 thru 0x00acee (0x6b29 bytes) 000272 1785390092.950002 mapwrite 0x00b158 thru 0x00ff17 (0x4dc0 bytes) 000273 1785390093.121362 write 0x03cf87 thru 0x03ffff (0x3079 bytes) 000275 1785390093.124499 trunc from 0x040000 to 0x02d55d (0x12aa2 bytes) 000276 1785390093.159540 read 0x00699c thru 0x00b913 (0x4f78 bytes) 000277 1785390093.171267 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2d55e): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d55e): Mirror 4: 0xf397bdc0 (stale) Mirror 6: 0xf397bdc0 (stale) Mirror 7: 0xf397bdc0 (stale) Mirror 8: 0xf397bdc0 (stale) CRC-32 checksum value for chunk [0, 0x2d55e): Mirror 1: 0xd5ec5314 Mirror 9: 0xd5ec5314 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d55e: skipped 000278 1785390094.235507 write 0x00e53f thru 0x0134cc (0x4f8e bytes) 000279 1785390094.295735 read 0x01e6c1 thru 0x0227fa (0x413a bytes) 000280 1785390094.301971 mapwrite 0x023447 thru 0x0271b1 (0x3d6b bytes) 000284 1785390094.353057 write 0x0050e0 thru 0x0091b2 (0x40d3 bytes) 000285 1785390094.360622 mapread 0x0015ed thru 0x007e35 (0x6849 bytes) 000287 1785390094.369336 write 0x02b731 thru 0x030a2f (0x52ff bytes) 000289 1785390094.371977 mapwrite 0x005a6d thru 0x009762 (0x3cf6 bytes) 000290 1785390094.405587 trunc from 0x030a30 to 0x00ad16 (0x25d19 bytes) 000292 1785390094.434020 read 0x00a6e9 thru 0x00ad16 (0x62e bytes) 000293 1785390094.438830 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xad17): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xad17): Mirror 4: 0x6c572379 (stale) Mirror 6: 0x6c572379 (stale) Mirror 7: 0x6c572379 (stale) Mirror 8: 0x6c572379 (stale) Mirror 9: 0x6c572379 (stale) CRC-32 checksum value for chunk [0, 0xad17): Mirror 1: 0xadbaeda4 Mirror 10: 0xadbaeda4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xad17: skipped 000294 1785390094.926602 trunc from 0x00ad17 to 0x015458 (0xa742 bytes) 000297 1785390095.002365 mapread 0x015069 thru 0x015458 (0x3f0 bytes) 000298 1785390095.010488 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x155:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15459): Mirror 4: 0xc9921ca7 (stale) Mirror 6: 0xc9921ca7 (stale) Mirror 7: 0xc9921ca7 (stale) Mirror 8: 0xc9921ca7 (stale) Mirror 9: 0xd9275567 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15459): Mirror 4: 0xc9921ca7 (stale) Mirror 6: 0xc9921ca7 (stale) Mirror 7: 0xc9921ca7 (stale) Mirror 8: 0xc9921ca7 (stale) Mirror 9: 0xd9275567 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15459: skipped 000300 1785390095.250395 trunc from 0x015459 to 0x027303 (0x11eab bytes) 000302 1785390095.261073 mapread 0x0019a3 thru 0x00c4e5 (0xab43 bytes) 000303 1785390095.276109 mapread 0x0256d6 thru 0x027303 (0x1c2e bytes) 000305 1785390095.286050 read 0x00a56e thru 0x0198f1 (0xf384 bytes) truncating to largest ever: 0x3e6e5 000309 1785390095.295658 trunc from 0x027304 to 0x03e6e4 (0x173e1 bytes) 000310 1785390095.313681 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3e6e5): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e6e5): Mirror 4: 0x8be4fc60 (stale) Mirror 6: 0x8be4fc60 (stale) Mirror 7: 0x8be4fc60 (stale) Mirror 8: 0x8be4fc60 (stale) Mirror 9: 0x2dc9dfa6 (stale) CRC-32 checksum value for chunk [0, 0x3e6e5): Mirror 1: 0xd934844b Mirror 10: 0xd934844b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e6e5: skipped 000311 1785390096.144996 mapread 0x00b3b2 thru 0x015757 (0xa3a6 bytes) 000313 1785390096.205448 mapwrite 0x00c7f6 thru 0x011019 (0x4824 bytes) 000315 1785390096.399671 write 0x03328c thru 0x03995f (0x66d4 bytes) 000317 1785390096.413517 mapread 0x00adbc thru 0x010e9a (0x60df bytes) 000320 1785390096.416235 mapwrite 0x025549 thru 0x028f5a (0x3a12 bytes) 000321 1785390096.585450 read 0x01499f thru 0x0221fa (0xd85c bytes) 000322 1785390096.601545 write 0x00c2f5 thru 0x014339 (0x8045 bytes) 000327 1785390096.617683 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3e6e5): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e6e5): Mirror 4: 0x8be4fc60 (stale) Mirror 6: 0x8be4fc60 (stale) Mirror 7: 0x8be4fc60 (stale) Mirror 8: 0x8be4fc60 (stale) Mirror 9: 0x2dc9dfa6 (stale) Mirror 10: 0xd934844b (stale) CRC-32 checksum value for chunk [0, 0x3e6e5): Mirror 1: 0xc14a91d Mirror 11: 0xc14a91d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e6e5: skipped 000328 1785390097.432433 write 0x01d0ea thru 0x0211a1 (0x40b8 bytes) 000333 1785390097.600320 mapwrite 0x03c9ec thru 0x03ffff (0x3614 bytes) 000334 1785390097.917533 read 0x01cfe4 thru 0x0229d9 (0x59f6 bytes) 000337 1785390097.971879 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xc2a0908c (stale) Mirror 6: 0xc2a0908c (stale) Mirror 7: 0xc2a0908c (stale) Mirror 8: 0xc2a0908c (stale) Mirror 9: 0xe131f51a (stale) Mirror 10: 0x3f7f396c (stale) Mirror 11: 0x23d01df1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc81c590a Mirror 12: 0xc81c590a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000339 1785390099.037524 mapwrite 0x021c5a thru 0x0247ee (0x2b95 bytes) 000341 1785390099.253714 MIRROR_RESYNC 9 mirrors 000343 1785390100.783923 write 0x02430c thru 0x02fb85 (0xb87a bytes) 000344 1785390100.944047 write 0x00f9de thru 0x01f7ca (0xfded bytes) 000345 1785390100.961477 MIRROR_RESYNC 9 mirrors 000347 1785390102.549816 read 0x02d3bc thru 0x0327a5 (0x53ea bytes) 000350 1785390102.599621 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x155:0x0]: Invalid argument (22) 000352 1785390102.933437 read 0x0068e9 thru 0x008059 (0x1771 bytes) 000353 1785390102.939939 mapwrite 0x024b6b thru 0x0346ae (0xfb44 bytes) 000354 1785390103.098631 read 0x00288f thru 0x00b0e1 (0x8853 bytes) 000355 1785390103.118941 trunc from 0x040000 to 0x0204d2 (0x1fb2d bytes) 000356 1785390103.136407 read 0x00e707 thru 0x01b8f7 (0xd1f1 bytes) 000359 1785390103.138220 write 0x02fcde thru 0x037ec3 (0x81e6 bytes) 000360 1785390103.142370 mapread 0x0121a7 thru 0x01e9aa (0xc804 bytes) 000362 1785390103.145361 trunc from 0x037ec4 to 0x038b02 (0xc3f bytes) 000364 1785390103.158656 read 0x02d1eb thru 0x02d906 (0x71c bytes) 000365 1785390103.165429 mapread 0x01da26 thru 0x02bf07 (0xe4e2 bytes) 000366 1785390103.167679 mapread 0x0199e5 thru 0x01cba4 (0x31c0 bytes) 000369 1785390103.170391 read 0x031941 thru 0x038b02 (0x71c2 bytes) 000370 1785390103.171270 mapread 0x003da1 thru 0x00fc5e (0xbebe bytes) 000371 1785390103.172510 write 0x03c18e thru 0x03ffff (0x3e72 bytes) 000372 1785390103.173954 write 0x028332 thru 0x033ade (0xb7ad bytes) 000373 1785390103.175398 write 0x01fa16 thru 0x0238d6 (0x3ec1 bytes) 000375 1785390103.178993 read 0x02c968 thru 0x030b86 (0x421f bytes) 000377 1785390103.180927 mapwrite 0x0294c9 thru 0x031c9f (0x87d7 bytes) 000378 1785390103.230790 mapread 0x03160e thru 0x03d3db (0xbdce bytes) 000380 1785390103.239774 mapwrite 0x02120c thru 0x029254 (0x8049 bytes) 000381 1785390103.283084 trunc from 0x040000 to 0x03c5e4 (0x3a1b bytes) 000382 1785390103.296267 write 0x00633d thru 0x009e36 (0x3afa bytes) 000386 1785390103.300598 trunc from 0x03c5e5 to 0x02de23 (0xe7c1 bytes) 000388 1785390103.320849 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x155:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2de24): Mirror 4: 0xc627545a (stale) Mirror 7: 0xc627545a (stale) Mirror 8: 0xc627545a (stale) Mirror 9: 0xc627545a (stale) Mirror 10: 0xc627545a (stale) Mirror 11: 0xc627545a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2de24): Mirror 4: 0xc627545a (stale) Mirror 7: 0xc627545a (stale) Mirror 8: 0xc627545a (stale) Mirror 9: 0xc627545a (stale) Mirror 10: 0xc627545a (stale) Mirror 11: 0xc627545a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2de24: skipped 000389 1785390103.853574 mapread 0x009dcc thru 0x010ae0 (0x6d15 bytes) 000391 1785390103.906588 MIRROR_RESYNC 7 mirrors 000394 1785390105.438131 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x155:0x0]: Invalid argument (22) 000395 1785390105.759573 trunc from 0x02de24 to 0x013075 (0x1adae bytes) 000397 1785390105.833598 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x155:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13076): Mirror 7: 0xc71bcc93 (stale) Mirror 8: 0xc71bcc93 (stale) Mirror 9: 0xc71bcc93 (stale) Mirror 11: 0xc71bcc93 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13076): Mirror 7: 0xc71bcc93 (stale) Mirror 8: 0xc71bcc93 (stale) Mirror 9: 0xc71bcc93 (stale) Mirror 11: 0xc71bcc93 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13076: skipped 000398 1785390106.265069 mapread 0x000bbf thru 0x000fa9 (0x3eb bytes) 000401 1785390106.281469 read 0x011629 thru 0x013075 (0x1a4d bytes) 000402 1785390106.291803 read 0x00a4b8 thru 0x013075 (0x8bbe bytes) 000403 1785390106.324915 read 0x00f519 thru 0x013075 (0x3b5d bytes) 000406 1785390106.344287 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x155:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13076): Mirror 7: 0xc71bcc93 (stale) Mirror 8: 0xc71bcc93 (stale) Mirror 9: 0xc71bcc93 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13076): Mirror 7: 0xc71bcc93 (stale) Mirror 8: 0xc71bcc93 (stale) Mirror 9: 0xc71bcc93 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13076: skipped 000409 1785390106.983721 mapwrite 0x01e112 thru 0x01e4ad (0x39c bytes) 000411 1785390107.171482 trunc from 0x01e4ae to 0x03e5e3 (0x20136 bytes) 000413 1785390107.216625 trunc from 0x03e5e4 to 0x006184 (0x3845f bytes) 000416 1785390107.238497 mapwrite 0x029a0e thru 0x035d0a (0xc2fd bytes) 000419 1785390107.433491 trunc from 0x035d0b to 0x035261 (0xaa9 bytes) 000420 1785390107.461272 trunc from 0x035262 to 0x01b71e (0x19b43 bytes) 000422 1785390107.504116 write 0x0276a8 thru 0x03065f (0x8fb8 bytes) 000425 1785390107.509854 write 0x03e3c8 thru 0x03ffff (0x1c38 bytes) 000426 1785390107.513503 mapread 0x009e88 thru 0x012551 (0x86ca bytes) 000427 1785390107.566965 trunc from 0x040000 to 0x0267fa (0x19805 bytes) 000428 1785390107.589649 mapwrite 0x034d72 thru 0x03fc21 (0xaeb0 bytes) 000429 1785390107.729756 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x155:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fc22): Mirror 7: 0x74b858a4 (stale) Mirror 8: 0x74b858a4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fc22): Mirror 7: 0x74b858a4 (stale) Mirror 8: 0x74b858a4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fc22: skipped 000430 1785390108.150410 trunc from 0x03fc22 to 0x00148d (0x3e794 bytes) 000432 1785390108.166346 MIRROR_RESYNC 3 mirrors 000434 1785390109.241865 read 0x001138 thru 0x00148d (0x356 bytes) 000436 1785390109.265890 read 0x00000f thru 0x00148d (0x147f bytes) 000438 1785390109.281257 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x155:0x0]: Invalid argument (22) 000439 1785390109.667497 trunc from 0x00148e to 0x004b87 (0x36fa bytes) 000440 1785390109.836722 mapwrite 0x007da6 thru 0x011805 (0x9a60 bytes) 000443 1785390110.090399 read 0x0078e7 thru 0x0079e8 (0x102 bytes) 000444 1785390110.093938 mapwrite 0x0048b3 thru 0x010d7d (0xc4cb bytes) 000447 1785390110.183738 read 0x004ef7 thru 0x00b763 (0x686d bytes) 000450 1785390110.187473 write 0x03a4dd thru 0x03ffff (0x5b23 bytes) 000454 1785390110.203975 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1051dca3 Mirror 9: 0x1051dca3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000455 1785390112.141883 trunc from 0x040000 to 0x01f6da (0x20925 bytes) 000456 1785390112.311365 write 0x02b328 thru 0x03147f (0x6158 bytes) 000459 1785390112.317073 read 0x00425b thru 0x012bea (0xe990 bytes) 000460 1785390112.328042 write 0x039836 thru 0x03ffff (0x67ca bytes) 000462 1785390112.331934 mapwrite 0x034c8f thru 0x03ebf8 (0x9f6a bytes) 000463 1785390112.512043 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x155:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000464 1785390113.146758 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x80b62b96 Mirror 9: 0x80b62b96 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000465 1785390114.343267 mapwrite 0x0027e9 thru 0x00b2a6 (0x8abe bytes) 000466 1785390114.866137 mapwrite 0x00779e thru 0x00d6a4 (0x5f07 bytes) 000468 1785390115.012578 mapread 0x036626 thru 0x036d8d (0x768 bytes) 000472 1785390115.039625 write 0x00b8cf thru 0x01158d (0x5cbf bytes) 000476 1785390115.116801 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x155:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000478 1785390115.821261 mapwrite 0x030e87 thru 0x034464 (0x35de bytes) 000480 1785390116.050803 mapread 0x03fc4b thru 0x03ffff (0x3b5 bytes) 000481 1785390116.095578 mapread 0x025e06 thru 0x02d381 (0x757c bytes) 000483 1785390116.129816 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x155:0x0]: Invalid argument (22) 000485 1785390116.506479 write 0x00c62a thru 0x01c5bf (0xff96 bytes) 000486 1785390116.547887 write 0x035b88 thru 0x03aa7b (0x4ef4 bytes) 000487 1785390116.557911 write 0x025161 thru 0x028f10 (0x3db0 bytes) 000488 1785390116.583201 mapwrite 0x02e6b9 thru 0x03e31f (0xfc67 bytes) 000490 1785390116.798350 mapread 0x013247 thru 0x013e73 (0xc2d bytes) 000494 1785390116.805560 mapread 0x029733 thru 0x0319d7 (0x82a5 bytes) 000495 1785390116.809555 trunc from 0x040000 to 0x02d393 (0x12c6c bytes) 000496 1785390116.876271 MIRROR_EXTEND to 2 mirrors 000497 1785390118.312982 mapwrite 0x00bc39 thru 0x014599 (0x8961 bytes) 000498 1785390118.677800 mapwrite 0x032c57 thru 0x039ade (0x6e88 bytes) 000499 1785390118.870843 read 0x024197 thru 0x0269ec (0x2856 bytes) 000500 1785390118.902834 mapwrite 0x01b902 thru 0x02b626 (0xfd25 bytes) 000501 1785390119.117771 write 0x022bc7 thru 0x0231d7 (0x611 bytes) 000502 1785390119.123330 write 0x0012e2 thru 0x004109 (0x2e28 bytes) 000503 1785390119.157620 write 0x026507 thru 0x035d95 (0xf88f bytes) 000506 1785390119.160210 mapwrite 0x007526 thru 0x01632f (0xee0a bytes) 000507 1785390119.348622 write 0x006a42 thru 0x0120ff (0xb6be bytes) 000508 1785390119.354513 read 0x014ae0 thru 0x01bc1f (0x7140 bytes) 000510 1785390119.373989 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x155:0x0]: Invalid argument (22) 000511 1785390119.597669 write 0x026d3c thru 0x032949 (0xbc0e bytes) 000512 1785390119.619404 trunc from 0x039adf to 0x015077 (0x24a67 bytes) 000514 1785390119.641460 mapread 0x011638 thru 0x015077 (0x3a40 bytes) 000517 1785390119.669695 read 0x0115a7 thru 0x015077 (0x3ad1 bytes) 000520 1785390119.672862 mapwrite 0x0159a2 thru 0x016c17 (0x1276 bytes) 000521 1785390119.752562 mapwrite 0x013b74 thru 0x018539 (0x49c6 bytes) 000522 1785390119.852512 MIRROR_EXTEND to 2 mirrors 000523 1785390120.946034 mapread 0x007d3e thru 0x016532 (0xe7f5 bytes) 000524 1785390120.992358 mapread 0x00f2b3 thru 0x014aa7 (0x57f5 bytes) 000529 1785390121.022843 mapread 0x00b4fb thru 0x01600b (0xab11 bytes) 000530 1785390121.032484 read 0x0125a5 thru 0x018539 (0x5f95 bytes) 000531 1785390121.037461 mapread 0x00f90e thru 0x011320 (0x1a13 bytes) 000533 1785390121.054809 mapwrite 0x001974 thru 0x00bfc1 (0xa64e bytes) 000534 1785390121.209759 trunc from 0x01853a to 0x000c56 (0x178e3 bytes) 000535 1785390121.245102 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x155:0x0]: Invalid argument (22) 000536 1785390121.409437 mapwrite 0x016ba5 thru 0x01a8a6 (0x3d02 bytes) 000537 1785390121.486955 read 0x0036a5 thru 0x00dd40 (0xa69c bytes) 000538 1785390121.495456 write 0x00dad2 thru 0x0181e3 (0xa712 bytes) 000540 1785390121.498451 write 0x017cce thru 0x02503a (0xd36d bytes) 000546 1785390121.505869 mapread 0x01c1c9 thru 0x02503a (0x8e72 bytes) 000549 1785390121.509241 write 0x0285cc thru 0x036081 (0xdab6 bytes) 000550 1785390121.511420 trunc from 0x036082 to 0x0312ee (0x4d93 bytes) 000554 1785390121.532201 mapwrite 0x02015b thru 0x02e71d (0xe5c3 bytes) 000559 1785390121.604630 read 0x02e8a1 thru 0x0312ee (0x2a4e bytes) 000562 1785390121.609412 trunc from 0x0312ef to 0x02fdc9 (0x1525 bytes) 000563 1785390121.646239 mapread 0x01391b thru 0x01c523 (0x8c09 bytes) 000566 1785390121.658921 mapwrite 0x021b5d thru 0x022013 (0x4b7 bytes) 000567 1785390121.732564 write 0x0263be thru 0x02b3df (0x5022 bytes) 000568 1785390121.734122 mapread 0x018711 thru 0x020e18 (0x8708 bytes) 000569 1785390121.736834 mapwrite 0x026541 thru 0x02ef9a (0x8a5a bytes) 000570 1785390121.777750 read 0x018019 thru 0x01feba (0x7ea2 bytes) 000572 1785390121.778932 trunc from 0x02fdca to 0x020cfa (0xf0cf bytes) 000574 1785390121.791472 write 0x02e950 thru 0x039792 (0xae43 bytes) 000576 1785390121.794373 read 0x01459e thru 0x0187f9 (0x425c bytes) 000577 1785390121.795573 mapwrite 0x0205e4 thru 0x0225fd (0x201a bytes) 000578 1785390121.833057 mapwrite 0x007b80 thru 0x00c874 (0x4cf5 bytes) 000581 1785390121.864626 read 0x0091c3 thru 0x016ce6 (0xdb24 bytes) 000582 1785390121.865715 mapread 0x033ac9 thru 0x035228 (0x1760 bytes) 000583 1785390121.867430 write 0x027e34 thru 0x031a39 (0x9c06 bytes) 000586 1785390121.876217 MIRROR_EXTEND to 2 mirrors 000589 1785390122.405812 mapread 0x00bd12 thru 0x0197ab (0xda9a bytes) 000590 1785390122.440079 write 0x03a88c thru 0x03ffff (0x5774 bytes) 000591 1785390122.495795 read 0x022f95 thru 0x02bebb (0x8f27 bytes) 000592 1785390122.505108 mapwrite 0x01150f thru 0x01e8fd (0xd3ef bytes) 000593 1785390122.573646 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x155:0x0]: Invalid argument (22) 000594 1785390122.706574 trunc from 0x040000 to 0x036940 (0x96bf bytes) 000595 1785390122.728066 trunc from 0x036941 to 0x031343 (0x55fd bytes) 000598 1785390122.745455 trunc from 0x031344 to 0x02848e (0x8eb5 bytes) 000599 1785390122.763086 mapwrite 0x00b828 thru 0x00f1ca (0x39a3 bytes) 000600 1785390122.818635 trunc from 0x02848f to 0x00ab0c (0x1d982 bytes) 000606 1785390122.843437 mapwrite 0x01c192 thru 0x01f5af (0x341e bytes) 000607 1785390122.908767 write 0x0066e9 thru 0x015c46 (0xf55e bytes) 000608 1785390122.911543 mapwrite 0x0269a1 thru 0x02b946 (0x4fa6 bytes) 000609 1785390122.977443 mapwrite 0x007635 thru 0x016294 (0xec60 bytes) 000614 1785390123.021605 read 0x00d20b thru 0x016ff8 (0x9dee bytes) 000620 1785390123.028743 write 0x024610 thru 0x02af5d (0x694e bytes) 000623 1785390123.033656 read 0x027aba thru 0x02a3f2 (0x2939 bytes) 000625 1785390123.035350 trunc from 0x02b947 to 0x0341de (0x8898 bytes) 000628 1785390123.053460 mapread 0x012b2b thru 0x01c1f6 (0x96cc bytes) 000629 1785390123.054891 write 0x037e37 thru 0x03ffff (0x81c9 bytes) 000630 1785390123.057622 read 0x019663 thru 0x020624 (0x6fc2 bytes) 000631 1785390123.058700 trunc from 0x040000 to 0x0209b8 (0x1f647 bytes) 000634 1785390123.077400 trunc from 0x0209b9 to 0x03587b (0x14ec3 bytes) 000635 1785390123.092761 mapwrite 0x01f4d3 thru 0x023650 (0x417e bytes) 000637 1785390123.143742 mapread 0x024e29 thru 0x033fa3 (0xf17b bytes) 000641 1785390123.148696 write 0x0125f8 thru 0x01a84e (0x8257 bytes) 000644 1785390123.151770 write 0x0100d9 thru 0x01585a (0x5782 bytes) 000645 1785390123.153707 trunc from 0x03587c to 0x02a59d (0xb2de bytes) 000647 1785390123.179128 trunc from 0x02a59e to 0x01fffc (0xa5a1 bytes) 000649 1785390123.196582 trunc from 0x01fffd to 0x03776d (0x17771 bytes) 000651 1785390123.216585 mapread 0x0005d7 thru 0x0031d7 (0x2c01 bytes) 000653 1785390123.220086 write 0x011a4d thru 0x011e0f (0x3c3 bytes) 000654 1785390123.228927 MIRROR_EXTEND to 2 mirrors 000656 1785390124.279069 trunc from 0x03776e to 0x03996e (0x2201 bytes) 000659 1785390124.408726 mapread 0x014333 thru 0x022747 (0xe415 bytes) 000661 1785390124.468123 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x155:0x0]: Invalid argument (22) 000662 1785390124.755737 trunc from 0x03996f to 0x036179 (0x37f5 bytes) 000663 1785390124.774277 write 0x0135a5 thru 0x01ce91 (0x98ed bytes) 000669 1785390124.804663 write 0x039aa2 thru 0x03f210 (0x576f bytes) 000670 1785390124.805846 mapwrite 0x009456 thru 0x018a04 (0xf5af bytes) 000671 1785390124.951081 read 0x03c971 thru 0x03f210 (0x28a0 bytes) 000673 1785390124.953084 read 0x007a6e thru 0x017239 (0xf7cc bytes) 000675 1785390124.956284 mapread 0x01577c thru 0x024c6f (0xf4f4 bytes) 000676 1785390124.999603 write 0x02c5aa thru 0x03bda5 (0xf7fc bytes) 000677 1785390125.003209 mapwrite 0x01fd07 thru 0x0247d1 (0x4acb bytes) 000679 1785390125.129382 write 0x03f24a thru 0x03ffff (0xdb6 bytes) 000681 1785390125.130815 mapwrite 0x03c646 thru 0x03ffff (0x39ba bytes) 000682 1785390125.214426 mapread 0x0315b3 thru 0x03cc2e (0xb67c bytes) 000687 1785390125.224241 mapwrite 0x0048f2 thru 0x0099fc (0x510b bytes) 000688 1785390125.335966 write 0x015dc8 thru 0x0209e5 (0xac1e bytes) 000690 1785390125.337878 mapwrite 0x011bec thru 0x017b4f (0x5f64 bytes) 000698 1785390125.452383 mapwrite 0x00eeac thru 0x014fba (0x610f bytes) 000699 1785390125.653971 mapwrite 0x00ec7d thru 0x01969b (0xaa1f bytes) 000700 1785390125.743046 read 0x0084d3 thru 0x01272c (0xa25a bytes) 000701 1785390125.743827 trunc from 0x040000 to 0x02db76 (0x12489 bytes) 000704 1785390125.773366 mapwrite 0x035777 thru 0x0360a6 (0x930 bytes) 000706 1785390125.894764 mapwrite 0x005ab4 thru 0x0112f8 (0xb845 bytes) 000713 1785390125.980327 write 0x019433 thru 0x01dc27 (0x47f5 bytes) 000715 1785390125.983623 write 0x033f2c thru 0x034e2d (0xf02 bytes) 000716 1785390125.984754 mapread 0x01d964 thru 0x02b486 (0xdb23 bytes) 000717 1785390125.986920 trunc from 0x0360a7 to 0x0014df (0x34bc7 bytes) 000723 1785390126.010291 mapread 0x0013f5 thru 0x0014df (0xeb bytes) 000724 1785390126.032298 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x3ff14 000726 1785390127.180662 trunc from 0x0014e0 to 0x03ff13 (0x3ea34 bytes) 000727 1785390127.288326 mapwrite 0x0126be thru 0x015f85 (0x38c8 bytes) 000731 1785390127.397619 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ff14): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ff14): Mirror 2: 0xf4e74549 (stale) CRC-32 checksum value for chunk [0, 0x3ff14): Mirror 1: 0x77a28a79 Mirror 3: 0x77a28a79 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ff14: skipped 000732 1785390128.564634 MIRROR_RESYNC 3 mirrors 000733 1785390129.553716 trunc from 0x03ff14 to 0x00ebb8 (0x3135b bytes) 000735 1785390129.701712 mapwrite 0x03a6fc thru 0x03ffff (0x5904 bytes) 000737 1785390129.897096 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x155:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xda07d6a5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xda07d6a5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000740 1785390130.367307 trunc from 0x040000 to 0x017409 (0x28bf6 bytes) 000742 1785390130.389732 write 0x033612 thru 0x03c614 (0x9003 bytes) 000743 1785390130.392933 mapwrite 0x0078d4 thru 0x00e4ae (0x6bdb bytes) 000748 1785390130.562544 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3c615): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c615): Mirror 3: 0x9c0c8e79 (stale) CRC-32 checksum value for chunk [0, 0x3c615): Mirror 1: 0x249385b2 Mirror 4: 0x249385b2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c615: skipped 000749 1785390131.975897 mapwrite 0x02e94e thru 0x03bea9 (0xd55c bytes) 000750 1785390132.175665 trunc from 0x03c615 to 0x02239b (0x1a279 bytes) 000753 1785390132.241860 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2239c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2239c): Mirror 3: 0x8cc04a8 (stale) Mirror 4: 0x825a5273 (stale) CRC-32 checksum value for chunk [0, 0x2239c): Mirror 1: 0x825a5273 Mirror 5: 0x825a5273 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2239c: skipped 000754 1785390133.137968 trunc from 0x02239c to 0x03bb20 (0x19785 bytes) 000756 1785390133.230994 read 0x000bf5 thru 0x00e2aa (0xd6b6 bytes) 000757 1785390133.241920 read 0x017c14 thru 0x018f62 (0x134f bytes) 000758 1785390133.253856 write 0x03d908 thru 0x03ffff (0x26f8 bytes) 000760 1785390133.259678 mapwrite 0x01a692 thru 0x02679a (0xc109 bytes) 000761 1785390133.330673 trunc from 0x040000 to 0x0022cb (0x3dd34 bytes) 000762 1785390133.347451 trunc from 0x0022cc to 0x00a29a (0x7fcf bytes) 000764 1785390133.364679 trunc from 0x00a29b to 0x00a491 (0x1f7 bytes) 000766 1785390133.391780 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x155:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xa492): Mirror 3: 0xfe9741e2 (stale) Mirror 4: 0x4b6f55c6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa492): Mirror 3: 0xfe9741e2 (stale) Mirror 4: 0x4b6f55c6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa492: skipped 000770 1785390133.572758 read 0x001934 thru 0x00a491 (0x8b5e bytes) 000773 1785390133.580514 write 0x022622 thru 0x0255f2 (0x2fd1 bytes) 000775 1785390133.587579 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x255f3): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x255f3): Mirror 3: 0xfbd6dff9 (stale) Mirror 4: 0xf5d26980 (stale) CRC-32 checksum value for chunk [0, 0x255f3): Mirror 1: 0xb3be7766 Mirror 5: 0xb3be7766 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x255f3: skipped 000777 1785390134.194439 mapread 0x023dd4 thru 0x0255f2 (0x181f bytes) 000781 1785390134.241918 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x255f3): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x255f3): Mirror 3: 0xfbd6dff9 (stale) Mirror 4: 0xf5d26980 (stale) CRC-32 checksum value for chunk [0, 0x255f3): Mirror 1: 0xb3be7766 Mirror 5: 0xb3be7766 Mirror 6: 0xb3be7766 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x255f3: skipped 000784 1785390134.906491 write 0x02cae6 thru 0x037e23 (0xb33e bytes) 000786 1785390134.965748 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x37e24): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37e24): Mirror 3: 0x2e1ceee3 (stale) Mirror 4: 0x18d3b517 (stale) Mirror 5: 0x8e95bfc8 (stale) Mirror 6: 0x8e95bfc8 (stale) CRC-32 checksum value for chunk [0, 0x37e24): Mirror 1: 0x24e0050e Mirror 7: 0x24e0050e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37e24: skipped 000788 1785390135.822876 mapread 0x024e53 thru 0x031df7 (0xcfa5 bytes) 000791 1785390135.903224 read 0x02c5ed thru 0x032f4d (0x6961 bytes) 000795 1785390135.939346 trunc from 0x037e24 to 0x020aa4 (0x1737f bytes) 000796 1785390136.019498 read 0x016d27 thru 0x018fd4 (0x22ae bytes) 000797 1785390136.027532 read 0x017fd1 thru 0x01d27c (0x52ac bytes) 000799 1785390136.030088 mapwrite 0x03f34f thru 0x03ffff (0xcb1 bytes) 000802 1785390136.078746 trunc from 0x040000 to 0x0032aa (0x3cd55 bytes) 000803 1785390136.090753 trunc from 0x0032ab to 0x03fe71 (0x3cbc7 bytes) 000804 1785390136.102511 trunc from 0x03fe72 to 0x01421a (0x2bc57 bytes) 000805 1785390136.113502 mapwrite 0x003ee6 thru 0x0054aa (0x15c5 bytes) 000806 1785390136.146019 mapread 0x00f742 thru 0x01421a (0x4ad9 bytes) 000808 1785390136.169267 MIRROR_RESYNC 6 mirrors 000809 1785390136.957475 MIRROR_EXTEND to 7 mirrors 000812 1785390137.882613 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x155:0x0]: Invalid argument (22) 000814 1785390138.548684 write 0x01d400 thru 0x021c30 (0x4831 bytes) 000815 1785390138.691663 write 0x02926b thru 0x032a37 (0x97cd bytes) 000817 1785390138.702457 MIRROR_RESYNC 6 mirrors 000819 1785390140.764864 MIRROR_RESYNC 6 mirrors 000821 1785390141.736241 write 0x038abd thru 0x03b9f8 (0x2f3c bytes) 000822 1785390141.871252 mapwrite 0x0174c3 thru 0x027202 (0xfd40 bytes) 000824 1785390142.138037 trunc from 0x03b9f9 to 0x01084b (0x2b1ad bytes) 000825 1785390142.193586 read 0x0006a5 thru 0x005b45 (0x54a1 bytes) 000826 1785390142.210874 read 0x0053a8 thru 0x00ae98 (0x5af1 bytes) 000830 1785390142.232589 mapread 0x00cdb6 thru 0x01084b (0x3a96 bytes) 000831 1785390142.241113 mapread 0x00922d thru 0x00b6b2 (0x2486 bytes) 000833 1785390142.243679 read 0x006467 thru 0x00871e (0x22b8 bytes) 000834 1785390142.249851 mapread 0x00fb8e thru 0x01084b (0xcbe bytes) 000835 1785390142.259242 mapwrite 0x014666 thru 0x01d5d4 (0x8f6f bytes) 000836 1785390142.453643 MIRROR_RESYNC 6 mirrors 000837 1785390145.013021 read 0x017ec2 thru 0x01d5d4 (0x5713 bytes) 000838 1785390145.031103 mapread 0x018635 thru 0x01d5d4 (0x4fa0 bytes) 000840 1785390145.032250 mapwrite 0x02fb9a thru 0x03f6b3 (0xfb1a bytes) 000841 1785390145.326338 write 0x02a8c1 thru 0x02bee0 (0x1620 bytes) 000842 1785390145.332347 mapwrite 0x034803 thru 0x03f94c (0xb14a bytes) 000843 1785390145.435106 read 0x03ced5 thru 0x03f94c (0x2a78 bytes) 000844 1785390145.436195 mapwrite 0x023903 thru 0x02f09b (0xb799 bytes) 000845 1785390145.538216 mapread 0x028b74 thru 0x02d1ae (0x463b bytes) 000846 1785390145.541142 mapread 0x03ea0e thru 0x03f94c (0xf3f bytes) 000847 1785390145.542639 read 0x01f215 thru 0x021367 (0x2153 bytes) 000849 1785390145.545108 trunc from 0x03f94d to 0x00a9c5 (0x34f87 bytes) 000851 1785390145.560930 mapwrite 0x0361d1 thru 0x03b84e (0x567e bytes) 000852 1785390145.645033 mapread 0x03270b thru 0x03b84e (0x9144 bytes) 000853 1785390145.647370 mapread 0x02944a thru 0x0389fb (0xf5b2 bytes) 000854 1785390145.651495 write 0x027b2b thru 0x028cfd (0x11d3 bytes) 000856 1785390145.656511 read 0x00ae67 thru 0x00dfd1 (0x316b bytes) 000857 1785390145.663770 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x155:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b84f): Mirror 4: 0x3b04c2c5 (stale) Mirror 5: 0x3b04c2c5 (stale) Mirror 7: 0x3b04c2c5 (stale) Mirror 8: 0x3b04c2c5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b84f): Mirror 4: 0x3b04c2c5 (stale) Mirror 5: 0x3b04c2c5 (stale) Mirror 7: 0x3b04c2c5 (stale) Mirror 8: 0x3b04c2c5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b84f: skipped 000859 1785390146.268763 mapread 0x0129af thru 0x01e784 (0xbdd6 bytes) 000860 1785390146.337979 MIRROR_RESYNC 5 mirrors 000861 1785390148.997063 write 0x00f7a4 thru 0x018cba (0x9517 bytes) 000862 1785390149.199306 mapread 0x033ed5 thru 0x03b84e (0x797a bytes) 000863 1785390149.240904 write 0x02d374 thru 0x0308bd (0x354a bytes) 000865 1785390149.253095 trunc from 0x03b84f to 0x02e712 (0xd13c bytes) 000867 1785390149.331093 mapwrite 0x00feea thru 0x011e74 (0x1f8b bytes) 000868 1785390149.464292 trunc from 0x02e713 to 0x00549c (0x29276 bytes) 000870 1785390149.509065 mapwrite 0x012abc thru 0x01a18c (0x76d1 bytes) 000871 1785390149.719632 mapwrite 0x0250c3 thru 0x031f30 (0xce6e bytes) 000873 1785390149.871182 mapwrite 0x0152ae thru 0x01a75c (0x54af bytes) 000875 1785390149.990982 write 0x03bd5d thru 0x03ffff (0x42a3 bytes) 000876 1785390149.997208 mapread 0x013b53 thru 0x01da7c (0x9f2a bytes) 000877 1785390150.017864 mapread 0x011468 thru 0x01bcb7 (0xa850 bytes) 000878 1785390150.020602 mapwrite 0x01de20 thru 0x024a61 (0x6c42 bytes) 000884 1785390150.107238 mapread 0x03bea9 thru 0x03ffff (0x4157 bytes) 000885 1785390150.114534 trunc from 0x040000 to 0x032ddf (0xd220 bytes) 000886 1785390150.146964 write 0x02da7d thru 0x032168 (0x46ec bytes) 000888 1785390150.179228 MIRROR_RESYNC 5 mirrors 000889 1785390151.210844 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x155:0x0]: Invalid argument (22) 000890 1785390151.599938 MIRROR_EXTEND to 5 mirrors 000891 1785390152.405553 trunc from 0x032de0 to 0x034364 (0x1585 bytes) 000892 1785390152.518396 mapread 0x0186d9 thru 0x019ac1 (0x13e9 bytes) 000893 1785390152.558235 mapwrite 0x00bfc7 thru 0x014bd8 (0x8c12 bytes) 000894 1785390152.660471 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x34365): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34365): Mirror 4: 0xedcdde2c (stale) Mirror 5: 0xedcdde2c (stale) Mirror 7: 0xedcdde2c (stale) Mirror 8: 0xedcdde2c (stale) CRC-32 checksum value for chunk [0, 0x34365): Mirror 1: 0x2ee6d70a Mirror 9: 0x2ee6d70a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34365: skipped 000896 1785390153.681892 mapwrite 0x0133ec thru 0x01f9d7 (0xc5ec bytes) 000897 1785390153.833902 mapwrite 0x02aa45 thru 0x02cdaf (0x236b bytes) 000898 1785390153.867291 mapwrite 0x023950 thru 0x02be9c (0x854d bytes) 000899 1785390153.896327 trunc from 0x034365 to 0x0180b6 (0x1c2ae bytes) 000902 1785390153.912088 mapwrite 0x011f00 thru 0x01dd14 (0xbe15 bytes) 000903 1785390153.967588 read 0x006a66 thru 0x012069 (0xb604 bytes) 000904 1785390153.974973 mapread 0x00f7ef thru 0x01cecf (0xd6e1 bytes) 000908 1785390153.978697 read 0x014c73 thru 0x01dd14 (0x90a2 bytes) 000909 1785390153.982320 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1dd15): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dd15): Mirror 4: 0xb74b4a23 (stale) Mirror 5: 0xb74b4a23 (stale) Mirror 7: 0xb74b4a23 (stale) Mirror 8: 0xb74b4a23 (stale) Mirror 9: 0xde2d6c87 (stale) CRC-32 checksum value for chunk [0, 0x1dd15): Mirror 1: 0xccdb66b Mirror 10: 0xccdb66b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dd15: skipped 000910 1785390154.735124 trunc from 0x01dd15 to 0x0359cb (0x17cb7 bytes) 000911 1785390154.872875 MIRROR_RESYNC 7 mirrors 000912 1785390157.680452 mapwrite 0x039f6c thru 0x03cff2 (0x3087 bytes) 000914 1785390158.012259 read 0x037b88 thru 0x03cff2 (0x546b bytes) 000916 1785390158.037309 read 0x0154d1 thru 0x024215 (0xed45 bytes) 000917 1785390158.070028 write 0x0081f2 thru 0x01650c (0xe31b bytes) 000921 1785390158.086202 mapread 0x02bb1a thru 0x02e507 (0x29ee bytes) 000922 1785390158.087323 write 0x009ff1 thru 0x00e5a6 (0x45b6 bytes) 000923 1785390158.092699 write 0x023a67 thru 0x030b3b (0xd0d5 bytes) 000925 1785390158.107778 write 0x00a258 thru 0x00f880 (0x5629 bytes) 000928 1785390158.116894 read 0x005686 thru 0x01065e (0xafd9 bytes) 000931 1785390158.123115 read 0x033705 thru 0x03cff2 (0x98ee bytes) 000933 1785390158.129575 mapread 0x020b70 thru 0x02b54c (0xa9dd bytes) 000936 1785390158.137407 mapread 0x01f205 thru 0x02e349 (0xf145 bytes) 000943 1785390158.208095 MIRROR_RESYNC 7 mirrors 000945 1785390160.991397 trunc from 0x03cff3 to 0x03c9e0 (0x612 bytes) 000946 1785390161.191205 mapread 0x028d55 thru 0x032298 (0x9544 bytes) 000947 1785390161.318619 MIRROR_RESYNC 7 mirrors 000949 1785390164.518007 trunc from 0x03c9e1 to 0x021833 (0x1b1ad bytes) 000951 1785390164.792316 trunc from 0x021834 to 0x015a65 (0xbdce bytes) 000952 1785390164.825958 write 0x02c192 thru 0x02d2a9 (0x1118 bytes) 000953 1785390164.827098 read 0x01a7a4 thru 0x0253d1 (0xac2e bytes) 000954 1785390164.855622 MIRROR_RESYNC 7 mirrors 000957 1785390168.478400 mapwrite 0x025a6a thru 0x026ce0 (0x1277 bytes) 000962 1785390168.889251 MIRROR_RESYNC 7 mirrors 000963 1785390170.451110 write 0x00193c thru 0x001f88 (0x64d bytes) 000964 1785390170.485869 read 0x02a73a thru 0x02d2a9 (0x2b70 bytes) 000966 1785390170.492381 read 0x0020dd thru 0x006f5d (0x4e81 bytes) 000968 1785390170.500206 read 0x014b9a thru 0x01848b (0x38f2 bytes) 000971 1785390170.502103 write 0x02840d thru 0x036fa8 (0xeb9c bytes) 000973 1785390170.506778 mapwrite 0x0305de thru 0x0343e2 (0x3e05 bytes) 000977 1785390170.543874 trunc from 0x036fa9 to 0x0376d8 (0x730 bytes) 000978 1785390170.562914 mapread 0x01cddd thru 0x02918c (0xc3b0 bytes) 000979 1785390170.568300 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x155:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x376d9): Mirror 4: 0x45a6badd (stale) Mirror 5: 0x45a6badd (stale) Mirror 7: 0x45a6badd (stale) Mirror 8: 0x45a6badd (stale) Mirror 10: 0x45a6badd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x376d9): Mirror 4: 0x45a6badd (stale) Mirror 5: 0x45a6badd (stale) Mirror 7: 0x45a6badd (stale) Mirror 8: 0x45a6badd (stale) Mirror 10: 0x45a6badd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x376d9: skipped 000980 1785390170.990955 trunc from 0x0376d9 to 0x028e2f (0xe8a9 bytes) 000984 1785390171.008103 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x28e30): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28e30): Mirror 4: 0x84418857 (stale) Mirror 5: 0x84418857 (stale) Mirror 7: 0x84418857 (stale) Mirror 8: 0x84418857 (stale) Mirror 10: 0x84418857 (stale) CRC-32 checksum value for chunk [0, 0x28e30): Mirror 1: 0x7058cbf6 Mirror 11: 0x7058cbf6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28e30: skipped 000985 1785390171.511492 mapwrite 0x023663 thru 0x02c920 (0x92be bytes) 000986 1785390171.731307 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2c921): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c921): Mirror 4: 0x5c397f5d (stale) Mirror 5: 0x5c397f5d (stale) Mirror 7: 0x5c397f5d (stale) Mirror 8: 0x5c397f5d (stale) Mirror 10: 0x5c397f5d (stale) Mirror 11: 0x290f7891 (stale) CRC-32 checksum value for chunk [0, 0x2c921): Mirror 1: 0xce212daa Mirror 12: 0xce212daa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c921: skipped 000987 1785390172.792766 trunc from 0x02c921 to 0x012985 (0x19f9b bytes) 000988 1785390172.867173 mapwrite 0x022354 thru 0x02acb3 (0x8960 bytes) 000989 1785390172.936582 mapread 0x00de05 thru 0x010784 (0x2980 bytes) 000990 1785390172.945349 write 0x010756 thru 0x0112ab (0xb56 bytes) 000991 1785390172.946711 write 0x01af94 thru 0x0225e8 (0x7655 bytes) 000993 1785390172.951911 mapread 0x00d92a thru 0x013d04 (0x63db bytes) 000994 1785390172.958871 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x155:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2acb4): Mirror 5: 0xb380afa2 (stale) Mirror 7: 0xb380afa2 (stale) Mirror 8: 0xb380afa2 (stale) Mirror 10: 0xb380afa2 (stale) Mirror 11: 0xc76c78f3 (stale) Mirror 12: 0x131fbd04 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2acb4): Mirror 5: 0xb380afa2 (stale) Mirror 7: 0xb380afa2 (stale) Mirror 8: 0xb380afa2 (stale) Mirror 10: 0xb380afa2 (stale) Mirror 11: 0xc76c78f3 (stale) Mirror 12: 0x131fbd04 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2acb4: skipped 000996 1785390173.270603 read 0x018fc8 thru 0x02056b (0x75a4 bytes) 000997 1785390173.277827 mapread 0x008095 thru 0x00fad6 (0x7a42 bytes) 000999 1785390173.296562 MIRROR_RESYNC 7 mirrors 001000 1785390174.092209 trunc from 0x02acb4 to 0x01a6f8 (0x105bb bytes) All operations completed A-OK! PASS 70a (130s) == sanity-flr test 71: check mirror extend parent fid ==== 01:42:59 (1785390179) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (5s) == sanity-flr test 72: check append on FLR file ========== 01:43:04 (1785390184) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x283:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x267:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.439777 s, 4.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x283:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x284:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x267:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344883 s, 3.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x283:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x284:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x268:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x267:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x283:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x284:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x268:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x267:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x285:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (14s) == sanity-flr test 200a: stress test ===================== 01:43:18 (1785390198) Starting client: oleg247-client.virtnet: -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre2 Starting client: oleg247-client.virtnet: -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8634432 .. Extending file size to 2043936 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3143520 .. Extending file size to 6268992 .. Extending file size to 8432224 .. Extending file size to 8630400 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9201536 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 57234 57235 57236 57237 57240 192.168.202.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg247-client.virtnet /mnt/lustre2 (opts:) 192.168.202.147@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg247-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (91s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 01:44:49 (1785390289) Starting client: oleg247-client.virtnet: -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre2 Starting client: oleg247-client.virtnet: -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 2971232 .. Extending file size to 3319040 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7031776 .. Extending file size to 7527200 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8837472 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8261696 .. Extending file size to 8343008 .. Extending file size to 8586688 .. Extending file size to 8735968 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8777504 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58512 58514 58515 58516 58517 done 192.168.202.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg247-client.virtnet /mnt/lustre2 (opts:) 192.168.202.147@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg247-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (90s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 01:46:19 (1785390379) Starting client: oleg247-client.virtnet: -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0752781 s, 27.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.290606 s, 7.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xecb288c8 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg247-client.virtnet /mnt/lustre2 (opts:) PASS 200c (25s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 01:46:45 (1785390405) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0638825 s, 32.8 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 01:46:52 (1785390412) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x29a:0x0] } PASS 203 (11s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 01:47:03 (1785390423) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.274895 s, 7.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x280:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (12s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 01:47:15 (1785390435) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.251166 s, 8.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x281:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x282:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x281:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x282:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29e:0x0] } PASS 204b (13s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 01:47:28 (1785390448) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.606197 s, 3.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x283:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x283:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a0:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 01:48:04 (1785390484) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg247-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204764 s, 5.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x284:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x284:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x285:0x0] } PASS 204d (68s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 01:49:11 (1785390551) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg247-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.333533 s, 6.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x286:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x286:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (70s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 01:50:22 (1785390622) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg247-server mds-ost sync done. Creating new pool test_204f oleg247-server: Pool lustre.test_204f created Adding targets to pool oleg247-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg247-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.54187 s, 3.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x287:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x287:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg247-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg247-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg247-server: Pool lustre.test_204f destroyed PASS 204f (92s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 01:51:54 (1785390714) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 01:52:03 (1785390723) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 01:52:12 (1785390732) oleg247-server: Pool lustre.test_206 created Adding targets to pool oleg247-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg247-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg247-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg247-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg247-server: Pool lustre.test_206 destroyed PASS 206 (32s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 01:52:44 (1785390764) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113845 s, 9.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313107 s, 33.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xcc608216 Mirror 3: 0xcc608216 PASS 207 (15s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 01:52:59 (1785390779) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (6s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 01:53:05 (1785390785) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 01:53:10 (1785390790) Starting client: oleg247-client.virtnet: -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.160345 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x13e0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1770000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1770000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1770000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x25f0000 Read this (4 bytes): BBBB 192.168.202.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg247-client.virtnet /mnt/lustre2 (opts:) PASS 209a (47s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 01:53:57 (1785390837) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0262816 s, 312 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 01:54:05 (1785390845) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525628 s, 19.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1cb lmm_fid: [0x200000404:0x1cb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1cb lmm_fid: [0x200000404:0x1cb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x291:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834379 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 01:54:07.000000000 -0400 Modify: 2026-07-30 01:54:07.000000000 -0400 Change: 2026-07-30 01:54:09.000000000 -0400 Birth: 2026-07-30 01:54:07.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 01:54:15 (1785390855) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 01:54:18 (1785390858) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.746897 s, 54.8 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.127892 s, 32.0 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 30 01:54 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 30 01:54 /mnt/lustre/f211.sanity-flr PASS 211 (11s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 01:54:30 (1785390870) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (10s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 01:54:39 (1785390879) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (12s) == sanity-flr test complete, duration 2245 sec =========== 01:54:51 (1785390891) === sanity-flr: start cleanup 01:54:53 (1785390893) === === sanity-flr: finish cleanup 01:55:02 (1785390902) ===