-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 13:49:02 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg114-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 13:49:22 (1787852962) === oleg114-client.virtnet: executing check_config_client /mnt/lustre oleg114-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg114-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9c18d25d6000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c18d25d6000.idle_timeout=debug disable quota as required oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 13:49:56 (1787852996) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7623] == sanity-flr test 0a: lfs mirror create with -N option == 13:49:59 (1787852999) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 13:50:17 (1787853017) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 13:50:20 (1787853020) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 13:50:23 (1787853023) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 13:50:48 (1787853048) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 13:50:52 (1787853052) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 13:50:55 (1787853055) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9c18d25d6000.stats=clear osc.lustre-OST0001-osc-ffff9c18d25d6000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 13:51:03 (1787853063) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 13:51:16 (1787853076) PASS 0j (12s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 13:51:29 (1787853089) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 13:51:33 (1787853093) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 13:51:43 (1787853103) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 13:51:53 (1787853113) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 13:52:00 (1787853120) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 13:52:04 (1787853124) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 13:52:13 (1787853133) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 13:52:23 (1787853143) PASS 7 (9s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 13:52:32 (1787853152) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 2.84965 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104217 s, 10.1 MB/s PASS 21 (16s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 13:52:48 (1787853168) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 0.534986 s, 27.4 MB/s PASS 22 (12s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 13:53:01 (1787853181) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 13:53:13 (1787853193) Stopping /mnt/lustre-ost1 (opts:) on oleg114-server oleg114-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c18d25d6000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c18d25d6000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg114-server: oleg114-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg114-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg114-server oleg114-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c18d25d6000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c18d25d6000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg114-server: oleg114-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg114-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (81s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 13:54:34 (1787853274) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.228944 s, 13.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg114-server oleg114-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c18d25d6000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c18d25d6000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg114-server: oleg114-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg114-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 13:55:16 (1787853316) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg114-server oleg114-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c18d25d6000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c18d25d6000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg114-server: oleg114-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg114-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg114-server oleg114-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c18d25d6000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c18d25d6000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg114-server: oleg114-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg114-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (87s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 13:56:43 (1787853403) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.50551 s, 11.6 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 13:56 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.280s user 0m0.000s sys 0m0.170s Stopping /mnt/lustre-ost1 (opts:) on oleg114-server oleg114-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c18d25d6000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c18d25d6000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.493s user 0m0.001s sys 0m0.199s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg114-server: oleg114-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg114-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg114-server oleg114-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c18d25d6000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c18d25d6000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.411s user 0m0.000s sys 0m0.209s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg114-server: oleg114-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg114-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg114-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c18d25d6000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c18d25d6000.ost_server_uuid in FULL state after 0 sec PASS 33b (99s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 13:58:22 (1787853502) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (5s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 13:58:27 (1787853507) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 13:58:31 (1787853511) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 13:58:35 (1787853515) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 13:58:42 (1787853522) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.703309 s, 17.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 13:58:55 (1787853535) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396334 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211095 s, 49.7 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 13:59:01 (1787853541) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 13:59:11 (1787853551) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000776141 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00084091 s, 1.2 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000795008 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00337196 s, 304 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 13:59:23 (1787853563) create test files with size 15731155 .. 6cf83c906302d2b2d90e71ba5dd289ef - e7eb0701d5f64280446392a39bb70c1e - 609eaec016919b8be829b437bdaa2e18 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg114-client.virtnet /mnt/lustre (opts:) Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre PASS 37 (29s) == sanity-flr test 38: resync ============================ 13:59:52 (1787853592) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 14:00:05 (1787853605) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 14:00:12 (1787853612) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101014 s, 10.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719135 s, 14.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 14:00:20 (1787853620) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.142906 s, 14.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.256824 s, 16.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.257199 s, 8.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 14:00:35 (1787853635) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 14:00:38 (1787853638) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 14:00:41 (1787853641) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729603 s, 14.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 43c: read prefer uncompressed mirror == 14:00:50 (1787853650) llite.lustre-ffff9c18e1a9f000.enable_compression=1 fail_loc=0x1425 fail_val=-1 PASS 43c (7s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 14:00:57 (1787853657) llite.lustre-ffff9c18e1a9f000.enable_compression=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365139 s, 28.7 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180882 s, 58.0 MB/s **verify components PASS 43d (9s) == sanity-flr test 44a: lfs mirror split check =========== 14:01:05 (1787853665) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 14:01:09 (1787853669) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe2:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc8:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc8:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 14:01:19 (1787853679) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.251677 s, 41.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x1f:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg114-client.virtnet /mnt/lustre (opts:) Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 14:01:31 (1787853691) llite.lustre-ffff9c18c5b2b800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.257296 s, 40.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 Stopping client oleg114-client.virtnet /mnt/lustre (opts:) Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 llite.lustre-ffff9c18d0c64800.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 14:01:44 (1787853704) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (37s) == sanity-flr test 46: Verify setstripe --copy option ==== 14:02:21 (1787853741) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (34s) == sanity-flr test 47: Verify mirror obj alloc =========== 14:02:55 (1787853775) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 14:02:58 (1787853778) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.263963 s, 11.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.125206 s, 25.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xa53f699e (nosync,stale) Mirror 2: 0xa53f699e (nosync,stale) ** verify mirror 2 doesn't change original checksum: 2f5a8d35dd29b453b9136010489970ed - mirror 1 checksum: 1ddc14b9bb358204238b22ad231279b5 - mirror 2 checksum: 2f5a8d35dd29b453b9136010489970ed - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T14:03:01-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T14:03:01-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 2f5a8d35dd29b453b9136010489970ed - mirror 1 checksum: 1ddc14b9bb358204238b22ad231279b5 - mirror 2 checksum: 1ddc14b9bb358204238b22ad231279b5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T14:03:01-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T14:03:01-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 14:03:10 (1787853790) Failing mds1 on oleg114-server Stopping /mnt/lustre-mds1 (opts:) on oleg114-server 14:03:18 (1787853798) shut down facet: mds1 facet_host: oleg114-server facet_failover_host: oleg114-server Failover mds1 to oleg114-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-MDT0000 14:03:54 (1787853834) targets are mounted 14:03:54 (1787853834) facet_failover done oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024818 s, 42.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } PASS 50A (61s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 14:04:11 (1787853851) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 14:04:15 (1787853855) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 14:04:19 (1787853859) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00630186 s, 2.6 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 14:04:28 (1787853868) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 14:04:32 (1787853872) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00686204 s, 597 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0595499 s, 68.8 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 14:04:40 (1787853880) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 14:05:03 (1787853903) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 14:05:31 (1787853931) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 14:05:56 (1787853956) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 14:06:20 (1787853980) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 14:06:29 (1787853989) mirror create pid 55588 mirror split pid 55590 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 55588 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 55590 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 14:07:36 (1787854056) Seed set to 4058 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000004 1787854058.452427 mapwrite 0x03cf73 thru 0x03ffff (0x308d bytes) 000005 1787854058.603183 write 0x03624d thru 0x037a85 (0x1839 bytes) truncating to largest ever: 0x711d 000008 1787854058.623718 trunc from 0x040000 to 0x00711c (0x38ee3 bytes) truncating to largest ever: 0x2146e 000013 1787854058.685813 trunc from 0x00711d to 0x02146d (0x1a351 bytes) 000020 1787854058.722262 mapread 0x020d31 thru 0x02146d (0x73d bytes) 000021 1787854058.736953 trunc from 0x02146e to 0x00bdc7 (0x156a6 bytes) 000022 1787854058.771708 trunc from 0x00bdc8 to 0x00057e (0xb849 bytes) 000024 1787854058.810699 read 0x000507 thru 0x00057e (0x78 bytes) truncating to largest ever: 0x2e455 000026 1787854058.822188 trunc from 0x00057f to 0x02e454 (0x2ded6 bytes) 000027 1787854058.866597 mapwrite 0x00ea39 thru 0x01dfb3 (0xf57b bytes) 000028 1787854058.995346 MIRROR_EXTEND to 2 mirrors 000031 1787854059.938656 write 0x00acfc thru 0x014ef3 (0xa1f8 bytes) 000032 1787854060.035280 mapread 0x000cdd thru 0x009d62 (0x9086 bytes) 000034 1787854060.064696 mapread 0x012dd0 thru 0x021c34 (0xee65 bytes) 000035 1787854060.083132 MIRROR_RESYNC 2 mirrors 000036 1787854060.766379 mapread 0x0188fd thru 0x01b8b8 (0x2fbc bytes) 000037 1787854060.787094 write 0x0194eb thru 0x022632 (0x9148 bytes) 000038 1787854060.857583 read 0x0184d5 thru 0x026859 (0xe385 bytes) 000039 1787854060.867115 mapwrite 0x0379ea thru 0x03ffff (0x8616 bytes) 000040 1787854060.961588 MIRROR_RESYNC 2 mirrors 000041 1787854061.603286 mapread 0x02aea0 thru 0x02ccef (0x1e50 bytes) 000044 1787854061.625151 mapwrite 0x00f631 thru 0x0189e6 (0x93b6 bytes) 000046 1787854061.856167 mapwrite 0x00b7a9 thru 0x00de47 (0x269f bytes) 000047 1787854061.903289 write 0x004c9f thru 0x00897b (0x3cdd bytes) truncating to largest ever: 0x354eb 000048 1787854061.909056 trunc from 0x040000 to 0x0354ea (0xab15 bytes) 000051 1787854061.937945 trunc from 0x0354eb to 0x024676 (0x10e74 bytes) 000052 1787854061.958602 write 0x02705b thru 0x0369d1 (0xf977 bytes) 000054 1787854061.962520 trunc from 0x0369d2 to 0x020985 (0x1604c bytes) 000055 1787854061.978396 read 0x0188b5 thru 0x01e20c (0x5958 bytes) 000057 1787854061.979600 write 0x0085bd thru 0x00c20a (0x3c4e bytes) 000058 1787854061.989545 MIRROR_RESYNC 2 mirrors 000062 1787854062.956520 mapwrite 0x00ef51 thru 0x010956 (0x1a06 bytes) 000063 1787854063.158523 mapread 0x00c935 thru 0x01612b (0x97f7 bytes) 000064 1787854063.161659 mapwrite 0x0194e1 thru 0x01e4f5 (0x5015 bytes) 000065 1787854063.323221 mapread 0x0135fb thru 0x0140b5 (0xabb bytes) 000067 1787854063.324745 mapread 0x00cdbb thru 0x012028 (0x526e bytes) 000071 1787854063.327457 trunc from 0x020986 to 0x01ba77 (0x4f0e bytes) 000072 1787854063.359356 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) 000078 1787854063.543005 mapwrite 0x02f691 thru 0x039b17 (0xa487 bytes) truncating to largest ever: 0x3a554 000080 1787854063.695152 trunc from 0x039b18 to 0x03a553 (0xa3c bytes) 000081 1787854063.713559 mapwrite 0x0352e3 thru 0x03e3e4 (0x9102 bytes) 000083 1787854063.810863 read 0x039e07 thru 0x03e3e4 (0x45de bytes) 000085 1787854063.815563 trunc from 0x03e3e5 to 0x013d6b (0x2a679 bytes) 000086 1787854063.841735 MIRROR_EXTEND to 2 mirrors 000087 1787854064.541420 write 0x03329e thru 0x039990 (0x66f3 bytes) truncating to largest ever: 0x3bf15 000088 1787854064.599802 trunc from 0x039991 to 0x03bf14 (0x2584 bytes) 000090 1787854064.620374 trunc from 0x03bf15 to 0x007809 (0x3470b bytes) 000092 1787854064.651272 read 0x003239 thru 0x005595 (0x235d bytes) 000095 1787854064.660560 write 0x00d79c thru 0x00fff2 (0x2857 bytes) 000096 1787854064.662069 read 0x0032af thru 0x007405 (0x4157 bytes) 000097 1787854064.669819 read 0x00672c thru 0x00be2f (0x5704 bytes) 000098 1787854064.670963 read 0x00fe8b thru 0x00fff2 (0x168 bytes) 000100 1787854064.672090 trunc from 0x00fff3 to 0x003cad (0xc345 bytes) 000102 1787854064.693508 mapwrite 0x030dd3 thru 0x0340f1 (0x331f bytes) 000104 1787854064.788355 write 0x007e33 thru 0x017b0f (0xfcdd bytes) 000107 1787854064.792525 write 0x004155 thru 0x00f269 (0xb115 bytes) 000110 1787854064.798431 mapread 0x033f04 thru 0x0340f1 (0x1ee bytes) 000113 1787854064.822737 MIRROR_RESYNC 2 mirrors 000115 1787854065.572663 MIRROR_EXTEND to 3 mirrors 000116 1787854066.295886 mapread 0x00c00b thru 0x00d145 (0x113b bytes) 000117 1787854066.309397 write 0x038a8d thru 0x03aa86 (0x1ffa bytes) 000118 1787854066.404424 write 0x00276f thru 0x004960 (0x21f2 bytes) 000119 1787854066.417438 MIRROR_RESYNC 3 mirrors truncating to largest ever: 0x3da9a 000122 1787854067.456398 trunc from 0x03aa87 to 0x03da99 (0x3013 bytes) 000123 1787854067.540766 mapread 0x010080 thru 0x0150d8 (0x5059 bytes) 000124 1787854067.557720 mapread 0x022646 thru 0x026f58 (0x4913 bytes) truncating to largest ever: 0x3fb90 000126 1787854067.570308 trunc from 0x03da9a to 0x03fb8f (0x20f6 bytes) 000127 1787854067.594355 mapread 0x026922 thru 0x035c37 (0xf316 bytes) 000128 1787854067.609090 read 0x0221d4 thru 0x026531 (0x435e bytes) 000129 1787854067.609657 trunc from 0x03fb90 to 0x037aca (0x80c5 bytes) 000131 1787854067.627981 mapwrite 0x025d9f thru 0x02bc6d (0x5ecf bytes) 000133 1787854067.684302 mapwrite 0x017993 thru 0x01b4cd (0x3b3b bytes) 000134 1787854067.724523 write 0x00a762 thru 0x012fd9 (0x8878 bytes) 000139 1787854067.732819 mapwrite 0x03abb0 thru 0x03ffff (0x5450 bytes) 000140 1787854067.858496 read 0x017c17 thru 0x01874a (0xb34 bytes) 000144 1787854067.861792 mapread 0x038272 thru 0x03ffff (0x7d8e bytes) 000145 1787854067.862860 read 0x000623 thru 0x00dd9b (0xd779 bytes) 000146 1787854067.871428 mapwrite 0x032fff thru 0x03ea40 (0xba42 bytes) 000147 1787854067.915990 read 0x01914f thru 0x01f004 (0x5eb6 bytes) 000148 1787854067.916614 trunc from 0x040000 to 0x0128d2 (0x2d72d bytes) 000149 1787854067.938294 mapwrite 0x0025bf thru 0x00bfb0 (0x99f2 bytes) 000150 1787854067.986460 write 0x009b00 thru 0x00ff7d (0x647e bytes) 000151 1787854067.987634 read 0x011287 thru 0x0128d2 (0x164c bytes) 000153 1787854067.989729 write 0x00b46d thru 0x013eff (0x8a93 bytes) 000154 1787854067.991068 write 0x02f8a9 thru 0x03e9a1 (0xf0f9 bytes) 000155 1787854067.993851 read 0x00a432 thru 0x0193b0 (0xef7f bytes) 000156 1787854068.002203 mapwrite 0x038469 thru 0x03ffff (0x7b97 bytes) 000158 1787854068.083358 MIRROR_RESYNC 3 mirrors 000162 1787854069.095877 read 0x023047 thru 0x02a8c5 (0x787f bytes) 000163 1787854069.103358 mapread 0x024d62 thru 0x0270b0 (0x234f bytes) 000167 1787854069.105012 trunc from 0x040000 to 0x032fe0 (0xd01f bytes) 000169 1787854069.196689 trunc from 0x032fe1 to 0x02ca44 (0x659c bytes) 000170 1787854069.218511 trunc from 0x02ca45 to 0x030406 (0x39c2 bytes) 000173 1787854069.239438 mapwrite 0x01d1f7 thru 0x02c05c (0xee66 bytes) 000174 1787854069.314118 write 0x0085dc thru 0x01081c (0x8241 bytes) 000175 1787854069.323499 write 0x02ce0a thru 0x034b64 (0x7d5b bytes) 000177 1787854069.325791 read 0x00cce5 thru 0x013108 (0x6424 bytes) 000178 1787854069.332261 mapwrite 0x010ec7 thru 0x01da11 (0xcb4b bytes) 000180 1787854069.383262 mapwrite 0x02e564 thru 0x03b02f (0xcacc bytes) 000181 1787854069.458742 trunc from 0x03b030 to 0x013a3c (0x275f3 bytes) 000182 1787854069.481509 read 0x0122fa thru 0x013a3c (0x1743 bytes) 000183 1787854069.482521 mapread 0x0061b4 thru 0x00c118 (0x5f65 bytes) 000185 1787854069.488469 MIRROR_RESYNC 3 mirrors 000186 1787854070.121741 trunc from 0x013a3d to 0x016634 (0x2bf8 bytes) 000188 1787854070.224986 trunc from 0x016635 to 0x02aeb7 (0x14883 bytes) 000189 1787854070.269202 trunc from 0x02aeb8 to 0x001c41 (0x29276 bytes) 000190 1787854070.285438 mapwrite 0x031d62 thru 0x03afd8 (0x9277 bytes) 000191 1787854070.439664 read 0x00f2c7 thru 0x017a8d (0x87c7 bytes) 000192 1787854070.451181 mapread 0x00fb6d thru 0x01de7a (0xe30e bytes) 000194 1787854070.462232 trunc from 0x03afd9 to 0x038c77 (0x2361 bytes) 000195 1787854070.485174 read 0x0010b0 thru 0x01056f (0xf4c0 bytes) 000196 1787854070.497650 read 0x010559 thru 0x0105cf (0x77 bytes) 000197 1787854070.498262 mapread 0x01c12d thru 0x027595 (0xb469 bytes) 000199 1787854070.499584 read 0x02d72e thru 0x03098c (0x325f bytes) 000200 1787854070.501017 mapread 0x009b1d thru 0x017fb5 (0xe499 bytes) 000201 1787854070.502311 trunc from 0x038c78 to 0x004f92 (0x33ce5 bytes) 000203 1787854070.518003 mapwrite 0x01340d thru 0x01626c (0x2e60 bytes) 000206 1787854070.596539 write 0x025d2b thru 0x032cb6 (0xcf8c bytes) 000207 1787854070.598841 read 0x0173ec thru 0x01f200 (0x7e15 bytes) 000209 1787854070.607354 read 0x0186a6 thru 0x01f777 (0x70d2 bytes) 000210 1787854070.608538 trunc from 0x032cb7 to 0x00d8df (0x253d7 bytes) 000212 1787854070.630225 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd8e0): Mirror 3: 0x937056fb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd8e0): Mirror 3: 0x937056fb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd8e0: skipped 000214 1787854070.835638 MIRROR_RESYNC 2 mirrors 000215 1787854071.265246 read 0x0059ce thru 0x00d8df (0x7f12 bytes) 000216 1787854071.276977 mapwrite 0x014472 thru 0x015734 (0x12c3 bytes) 000219 1787854071.367522 mapread 0x003515 thru 0x00e1c9 (0xacb5 bytes) 000220 1787854071.377819 mapwrite 0x035640 thru 0x036130 (0xaf1 bytes) 000221 1787854071.440563 mapread 0x029c35 thru 0x036130 (0xc4fc bytes) 000222 1787854071.448757 write 0x01409c thru 0x01b7cb (0x7730 bytes) 000223 1787854071.458344 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36131): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36131): Mirror 3: 0x27655bd1 (stale) CRC-32 checksum value for chunk [0, 0x36131): Mirror 1: 0xa834fd25 Mirror 4: 0xa834fd25 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36131: skipped 000225 1787854072.288454 mapwrite 0x039ab5 thru 0x03ffff (0x654b bytes) 000226 1787854072.466646 mapread 0x0264d8 thru 0x02c58e (0x60b7 bytes) 000229 1787854072.481331 write 0x0316a4 thru 0x03dfde (0xc93b bytes) 000231 1787854072.488988 mapwrite 0x000753 thru 0x0032ac (0x2b5a bytes) 000233 1787854072.586855 mapread 0x0212d9 thru 0x02c0dd (0xae05 bytes) 000235 1787854072.590845 trunc from 0x040000 to 0x02db81 (0x1247e bytes) 000236 1787854072.628384 read 0x0091cc thru 0x00c90d (0x3742 bytes) 000237 1787854072.631235 write 0x03ec68 thru 0x03ffff (0x1398 bytes) 000238 1787854072.632442 mapread 0x02c779 thru 0x02ed1e (0x25a6 bytes) 000239 1787854072.650338 mapread 0x01fc4f thru 0x020552 (0x904 bytes) 000240 1787854072.651203 mapread 0x01194d thru 0x0156f5 (0x3da9 bytes) 000242 1787854072.667398 write 0x015414 thru 0x0185be (0x31ab bytes) 000243 1787854072.675452 write 0x00e0c6 thru 0x01cf47 (0xee82 bytes) 000244 1787854072.684087 write 0x00f747 thru 0x017fc0 (0x887a bytes) 000245 1787854072.689999 mapwrite 0x02880d thru 0x02ae37 (0x262b bytes) 000248 1787854072.783397 write 0x0278e4 thru 0x02f312 (0x7a2f bytes) 000249 1787854072.789236 MIRROR_RESYNC 3 mirrors 000251 1787854074.749554 mapread 0x02ee24 thru 0x0390f0 (0xa2cd bytes) 000252 1787854074.798433 mapwrite 0x03e3e6 thru 0x03fc24 (0x183f bytes) 000253 1787854075.074992 trunc from 0x040000 to 0x00378f (0x3c870 bytes) 000255 1787854075.121854 MIRROR_RESYNC 3 mirrors 000256 1787854076.027437 mapread 0x0036e0 thru 0x00378f (0xb0 bytes) 000257 1787854076.034485 trunc from 0x003790 to 0x02de36 (0x2a6a7 bytes) 000259 1787854076.122133 trunc from 0x02de37 to 0x01dc64 (0x101d2 bytes) 000260 1787854076.149157 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1dc65): Mirror 3: 0x362b7800 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dc65): Mirror 3: 0x362b7800 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dc65: skipped 000263 1787854076.420117 mapread 0x01bd9b thru 0x01dc64 (0x1eca bytes) 000266 1787854076.439128 write 0x002f5b thru 0x005a42 (0x2ae8 bytes) 000267 1787854076.447863 trunc from 0x01dc65 to 0x017c22 (0x6042 bytes) 000269 1787854076.462272 mapread 0x00b899 thru 0x0157cd (0x9f35 bytes) 000272 1787854076.472772 trunc from 0x017c23 to 0x03ab75 (0x22f53 bytes) 000273 1787854076.491370 mapwrite 0x00de26 thru 0x018b9c (0xad77 bytes) 000274 1787854076.556271 MIRROR_RESYNC 2 mirrors 000277 1787854077.276676 mapwrite 0x000157 thru 0x00492c (0x47d6 bytes) 000278 1787854077.523757 trunc from 0x03ab76 to 0x0041cb (0x369aa bytes) 000280 1787854077.595337 mapwrite 0x01dfd2 thru 0x025938 (0x7967 bytes) 000281 1787854077.719226 mapread 0x001ad8 thru 0x00724e (0x5777 bytes) 000283 1787854077.741783 mapread 0x01a9ac thru 0x024c95 (0xa2ea bytes) 000284 1787854077.742758 write 0x004161 thru 0x007c9f (0x3b3f bytes) 000285 1787854077.745895 read 0x0038b2 thru 0x00b58b (0x7cda bytes) 000286 1787854077.746774 mapwrite 0x02f1b7 thru 0x037638 (0x8482 bytes) 000287 1787854077.872073 trunc from 0x037639 to 0x03da07 (0x63cf bytes) 000288 1787854077.886107 mapread 0x017007 thru 0x024e54 (0xde4e bytes) 000289 1787854077.891335 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) 000290 1787854078.058766 MIRROR_EXTEND to 2 mirrors 000291 1787854079.112675 read 0x0163c0 thru 0x01db9f (0x77e0 bytes) 000293 1787854079.121272 write 0x03a37a thru 0x03ab5a (0x7e1 bytes) 000294 1787854079.216945 write 0x001ae7 thru 0x00377a (0x1c94 bytes) 000296 1787854079.228136 mapread 0x01e6ed thru 0x022f2a (0x483e bytes) 000299 1787854079.250233 write 0x018b82 thru 0x01f773 (0x6bf2 bytes) 000302 1787854079.258786 read 0x008218 thru 0x00dba5 (0x598e bytes) 000303 1787854079.285365 write 0x0019e6 thru 0x00825b (0x6876 bytes) 000304 1787854079.286491 mapwrite 0x039c6c thru 0x03e7f3 (0x4b88 bytes) 000306 1787854079.409647 mapwrite 0x033cf8 thru 0x03f91f (0xbc28 bytes) 000307 1787854079.487268 mapread 0x036d75 thru 0x036f14 (0x1a0 bytes) 000308 1787854079.488460 trunc from 0x03f920 to 0x00451c (0x3b403 bytes) 000309 1787854079.504176 read 0x00277d thru 0x00451c (0x1da0 bytes) 000310 1787854079.504762 read 0x0014d2 thru 0x00451c (0x304b bytes) 000311 1787854079.509203 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x451d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x451d): Mirror 2: 0xbfcce9b (stale) CRC-32 checksum value for chunk [0, 0x451d): Mirror 1: 0xdd79ae38 Mirror 3: 0xdd79ae38 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x451d: skipped 000312 1787854080.447859 mapwrite 0x02ca91 thru 0x0366fe (0x9c6e bytes) 000313 1787854080.613625 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x366ff): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x366ff): Mirror 2: 0xa8825f48 (stale) Mirror 3: 0xd06879a1 (stale) CRC-32 checksum value for chunk [0, 0x366ff): Mirror 1: 0x8b36f848 Mirror 4: 0x8b36f848 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x366ff: skipped 000314 1787854081.421169 write 0x0174a4 thru 0x01fe9b (0x89f8 bytes) 000315 1787854081.469387 read 0x01a7d9 thru 0x0202be (0x5ae6 bytes) 000316 1787854081.476403 mapread 0x0280d6 thru 0x02bb9b (0x3ac6 bytes) 000317 1787854081.489835 read 0x0137b5 thru 0x0191db (0x5a27 bytes) 000318 1787854081.496682 mapread 0x018196 thru 0x02597c (0xd7e7 bytes) 000319 1787854081.498051 trunc from 0x0366ff to 0x01f94e (0x16db0 bytes) 000320 1787854081.516708 trunc from 0x01f94f to 0x0100eb (0xf863 bytes) 000321 1787854081.535496 write 0x032725 thru 0x032c51 (0x52d bytes) 000322 1787854081.537476 mapread 0x01ff78 thru 0x02b1de (0xb267 bytes) 000323 1787854081.612563 read 0x031fe8 thru 0x032c51 (0xc6a bytes) 000326 1787854081.632777 mapwrite 0x013bb0 thru 0x0157f5 (0x1c46 bytes) 000328 1787854081.684117 mapread 0x0030fc thru 0x0057fb (0x2700 bytes) 000332 1787854081.687482 mapwrite 0x02e15a thru 0x035b7a (0x7a21 bytes) 000333 1787854081.738537 mapread 0x000c98 thru 0x00f7a7 (0xeb10 bytes) 000334 1787854081.739849 trunc from 0x035b7b to 0x029469 (0xc711 bytes) 000335 1787854081.754084 trunc from 0x02946a to 0x02948e (0x25 bytes) 000336 1787854081.768257 trunc from 0x02948f to 0x037827 (0xe399 bytes) 000337 1787854081.785531 mapread 0x0018db thru 0x005cef (0x4415 bytes) 000338 1787854081.786842 trunc from 0x037828 to 0x02e880 (0x8fa7 bytes) 000339 1787854081.804163 read 0x01c7bf thru 0x01f24a (0x2a8c bytes) 000340 1787854081.805029 trunc from 0x02e881 to 0x039eea (0xb66a bytes) 000341 1787854081.822137 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39eeb): Mirror 3: 0xa2e1ac4b (stale) Mirror 4: 0xc7ab29f0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39eeb): Mirror 3: 0xa2e1ac4b (stale) Mirror 4: 0xc7ab29f0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39eeb: skipped 000343 1787854082.136261 mapwrite 0x0014c5 thru 0x009b06 (0x8642 bytes) 000346 1787854082.222429 trunc from 0x039eeb to 0x015bb6 (0x24334 bytes) 000347 1787854082.249262 mapwrite 0x00d5c4 thru 0x01c4c0 (0xeefd bytes) 000350 1787854082.353686 trunc from 0x01c4c1 to 0x027135 (0xac75 bytes) 000351 1787854082.367437 read 0x004187 thru 0x0045d7 (0x451 bytes) 000354 1787854082.370196 mapread 0x001178 thru 0x0102d1 (0xf15a bytes) 000355 1787854082.372317 trunc from 0x027136 to 0x03efb5 (0x17e80 bytes) 000357 1787854082.393296 trunc from 0x03efb6 to 0x017acb (0x274ea bytes) 000359 1787854082.419668 MIRROR_RESYNC 3 mirrors 000361 1787854083.262389 MIRROR_EXTEND to 4 mirrors 000363 1787854083.948449 MIRROR_EXTEND to 5 mirrors 000364 1787854084.667003 write 0x03e1b0 thru 0x03ffff (0x1e50 bytes) 000365 1787854084.728753 read 0x022481 thru 0x02456d (0x20ed bytes) 000369 1787854084.739088 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xcf83c752 (stale) Mirror 5: 0xcf83c752 (stale) Mirror 6: 0xcf83c752 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xcf83c752 (stale) Mirror 5: 0xcf83c752 (stale) Mirror 6: 0xcf83c752 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000370 1787854085.144603 read 0x02724d thru 0x035537 (0xe2eb bytes) 000371 1787854085.154288 write 0x009316 thru 0x00cc95 (0x3980 bytes) 000373 1787854085.173279 trunc from 0x040000 to 0x00ac05 (0x353fa bytes) 000374 1787854085.210778 write 0x03a445 thru 0x03e681 (0x423d bytes) 000376 1787854085.219308 read 0x0236d0 thru 0x02f445 (0xbd76 bytes) 000377 1787854085.253769 trunc from 0x03e682 to 0x018f3e (0x25743 bytes) 000380 1787854085.284596 MIRROR_RESYNC 4 mirrors 000381 1787854086.697493 MIRROR_RESYNC 4 mirrors 000385 1787854087.095103 trunc from 0x018f3f to 0x0216e2 (0x87a4 bytes) 000387 1787854087.179250 mapread 0x018c46 thru 0x0216e2 (0x8a9d bytes) 000388 1787854087.198771 read 0x007a47 thru 0x011859 (0x9e13 bytes) 000389 1787854087.209251 write 0x0027bb thru 0x004c34 (0x247a bytes) 000390 1787854087.220254 mapread 0x003819 thru 0x00a07d (0x6865 bytes) 000394 1787854087.233357 MIRROR_RESYNC 4 mirrors 000395 1787854088.061980 mapwrite 0x036f1d thru 0x03ffff (0x90e3 bytes) 000396 1787854088.221850 write 0x035d87 thru 0x0376d5 (0x194f bytes) 000397 1787854088.224157 mapread 0x021ff3 thru 0x028ff5 (0x7003 bytes) 000398 1787854088.238324 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa03624a1 (stale) Mirror 5: 0xa03624a1 (stale) Mirror 6: 0xa03624a1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x82392276 Mirror 7: 0x82392276 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000399 1787854089.596604 write 0x019be0 thru 0x020001 (0x6422 bytes) 000400 1787854089.788829 MIRROR_RESYNC 5 mirrors 000402 1787854091.880143 write 0x016bbe thru 0x01e2fd (0x7740 bytes) 000405 1787854092.093094 mapwrite 0x015138 thru 0x0153e3 (0x2ac bytes) 000406 1787854092.242242 mapread 0x0219a3 thru 0x025be5 (0x4243 bytes) 000407 1787854092.270570 trunc from 0x040000 to 0x00598a (0x3a675 bytes) 000408 1787854092.299870 write 0x01ac70 thru 0x01f443 (0x47d4 bytes) 000410 1787854092.301445 trunc from 0x01f444 to 0x00139a (0x1e0a9 bytes) 000411 1787854092.342972 write 0x021d24 thru 0x026713 (0x49f0 bytes) 000412 1787854092.344178 mapwrite 0x0298e9 thru 0x034d46 (0xb45e bytes) 000414 1787854092.458841 read 0x006ab9 thru 0x008e52 (0x239a bytes) 000416 1787854092.466252 read 0x025f5a thru 0x02fbb2 (0x9c59 bytes) 000418 1787854092.471117 read 0x00b4d8 thru 0x0159cd (0xa4f6 bytes) 000419 1787854092.501594 mapwrite 0x037f03 thru 0x03860f (0x70d bytes) 000420 1787854092.600312 write 0x02d3c6 thru 0x038f6c (0xbba7 bytes) 000422 1787854092.603591 mapwrite 0x02d811 thru 0x031533 (0x3d23 bytes) 000424 1787854092.668365 MIRROR_RESYNC 5 mirrors 000426 1787854094.463614 mapwrite 0x00cd56 thru 0x00d892 (0xb3d bytes) 000428 1787854094.604245 mapwrite 0x02d97a thru 0x035233 (0x78ba bytes) 000429 1787854094.691111 write 0x03840b thru 0x03ffff (0x7bf5 bytes) 000430 1787854094.703480 mapread 0x01ad92 thru 0x01af2f (0x19e bytes) 000433 1787854094.720536 read 0x01c329 thru 0x01db98 (0x1870 bytes) 000434 1787854094.730041 trunc from 0x040000 to 0x036470 (0x9b8f bytes) 000435 1787854094.790386 trunc from 0x036471 to 0x007508 (0x2ef68 bytes) 000436 1787854094.819760 trunc from 0x007509 to 0x03c924 (0x3541c bytes) 000437 1787854094.842224 mapwrite 0x016e29 thru 0x01e2d9 (0x74b1 bytes) 000438 1787854094.951982 mapwrite 0x01b431 thru 0x01e461 (0x3031 bytes) 000440 1787854095.061741 mapread 0x014c93 thru 0x022311 (0xd67f bytes) 000441 1787854095.074316 mapread 0x02fdbc thru 0x035559 (0x579e bytes) 000445 1787854095.093629 trunc from 0x03c925 to 0x01b871 (0x210b3 bytes) 000450 1787854095.113637 read 0x000840 thru 0x00dd00 (0xd4c1 bytes) 000453 1787854095.123946 mapwrite 0x0333aa thru 0x03ffff (0xcc56 bytes) 000454 1787854095.196367 read 0x0348c3 thru 0x037f80 (0x36be bytes) 000455 1787854095.197548 mapread 0x00fdb1 thru 0x01e505 (0xe755 bytes) 000456 1787854095.203698 MIRROR_RESYNC 5 mirrors 000457 1787854096.718408 mapwrite 0x00f828 thru 0x010ec2 (0x169b bytes) 000458 1787854096.995830 mapwrite 0x039615 thru 0x03ffff (0x69eb bytes) 000459 1787854097.076863 trunc from 0x040000 to 0x000603 (0x3f9fc bytes) 000460 1787854097.109774 write 0x01fe2f thru 0x023425 (0x35f7 bytes) 000462 1787854097.116228 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23426): Mirror 5: 0x90b76733 (stale) Mirror 6: 0x90b76733 (stale) Mirror 7: 0x90b76733 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23426): Mirror 5: 0x90b76733 (stale) Mirror 6: 0x90b76733 (stale) Mirror 7: 0x90b76733 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23426: skipped 000463 1787854097.561953 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x23426): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23426): Mirror 5: 0x90b76733 (stale) Mirror 6: 0x90b76733 (stale) Mirror 7: 0x90b76733 (stale) CRC-32 checksum value for chunk [0, 0x23426): Mirror 1: 0x583bb959 Mirror 8: 0x583bb959 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23426: skipped 000465 1787854098.372242 read 0x0217cf thru 0x023425 (0x1c57 bytes) 000466 1787854098.379186 mapwrite 0x02a742 thru 0x02ffc3 (0x5882 bytes) 000468 1787854098.528461 trunc from 0x02ffc4 to 0x03e858 (0xe895 bytes) 000469 1787854098.546295 trunc from 0x03e859 to 0x016491 (0x283c7 bytes) 000470 1787854098.572287 trunc from 0x016492 to 0x03fb6d (0x296dc bytes) 000472 1787854098.602492 trunc from 0x03fb6e to 0x006939 (0x39234 bytes) 000473 1787854098.623681 mapread 0x0063c0 thru 0x006939 (0x57a bytes) 000477 1787854098.645262 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x693a): Mirror 6: 0xfb32fe59 (stale) Mirror 7: 0xfb32fe59 (stale) Mirror 8: 0xd3d3ca3d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x693a): Mirror 6: 0xfb32fe59 (stale) Mirror 7: 0xfb32fe59 (stale) Mirror 8: 0xd3d3ca3d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x693a: skipped 000478 1787854098.960818 mapread 0x001583 thru 0x006939 (0x53b7 bytes) 000479 1787854098.992574 read 0x0021f4 thru 0x006939 (0x4746 bytes) 000482 1787854098.997455 trunc from 0x00693a to 0x02f5b5 (0x28c7c bytes) 000483 1787854099.027438 read 0x00dcf0 thru 0x01069b (0x29ac bytes) 000484 1787854099.038817 read 0x020d91 thru 0x02c418 (0xb688 bytes) 000486 1787854099.044170 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2f5b6): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f5b6): Mirror 6: 0x25ce364e (stale) Mirror 7: 0x25ce364e (stale) Mirror 8: 0x2513dc79 (stale) CRC-32 checksum value for chunk [0, 0x2f5b6): Mirror 1: 0x7a7e23a9 Mirror 9: 0x7a7e23a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f5b6: skipped 000487 1787854100.007867 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2f5b6): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f5b6): Mirror 6: 0x25ce364e (stale) Mirror 7: 0x25ce364e (stale) Mirror 8: 0x2513dc79 (stale) CRC-32 checksum value for chunk [0, 0x2f5b6): Mirror 1: 0x7a7e23a9 Mirror 9: 0x7a7e23a9 Mirror 10: 0x7a7e23a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f5b6: skipped 000489 1787854100.935509 mapread 0x00d2dd thru 0x0116e5 (0x4409 bytes) 000491 1787854100.958347 mapread 0x01cd8d thru 0x028592 (0xb806 bytes) 000492 1787854100.978503 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2f5b6): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f5b6): Mirror 6: 0x25ce364e (stale) Mirror 7: 0x25ce364e (stale) Mirror 8: 0x2513dc79 (stale) CRC-32 checksum value for chunk [0, 0x2f5b6): Mirror 1: 0x7a7e23a9 Mirror 9: 0x7a7e23a9 Mirror 10: 0x7a7e23a9 Mirror 11: 0x7a7e23a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f5b6: skipped 000493 1787854102.181642 write 0x02d7c2 thru 0x03aeed (0xd72c bytes) 000494 1787854102.288827 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3aeee): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aeee): Mirror 6: 0xa3ab855b (stale) Mirror 7: 0xa3ab855b (stale) Mirror 8: 0xd6b8d67c (stale) Mirror 9: 0x6f550b1f (stale) Mirror 10: 0x6f550b1f (stale) Mirror 11: 0x6f550b1f (stale) CRC-32 checksum value for chunk [0, 0x3aeee): Mirror 1: 0x3e185ebb Mirror 12: 0x3e185ebb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aeee: skipped 000496 1787854103.334719 trunc from 0x03aeee to 0x032748 (0x87a5 bytes) 000497 1787854103.429634 read 0x00f961 thru 0x014782 (0x4e22 bytes) 000500 1787854103.438026 mapwrite 0x011da6 thru 0x01d2f2 (0xb54d bytes) 000503 1787854103.522624 write 0x026828 thru 0x031bc7 (0xb3a0 bytes) 000506 1787854103.535073 mapread 0x014db6 thru 0x024bfe (0xfe49 bytes) 000508 1787854103.545836 mapread 0x02a78f thru 0x032748 (0x7fba bytes) 000509 1787854103.552822 mapwrite 0x02417e thru 0x02c55e (0x83e1 bytes) 000510 1787854103.611160 mapwrite 0x033f4d thru 0x03cfa5 (0x9059 bytes) 000513 1787854103.715195 read 0x013f30 thru 0x01a572 (0x6643 bytes) 000514 1787854103.716374 write 0x016c9c thru 0x022b28 (0xbe8d bytes) 000516 1787854103.719582 write 0x0315de thru 0x03598d (0x43b0 bytes) 000518 1787854103.723270 mapwrite 0x02147b thru 0x025ed1 (0x4a57 bytes) 000519 1787854103.778139 mapread 0x008253 thru 0x012d84 (0xab32 bytes) 000520 1787854103.785885 trunc from 0x03cfa6 to 0x0012ed (0x3bcb8 bytes) 000521 1787854103.800942 read 0x0009b4 thru 0x0012ed (0x93a bytes) 000523 1787854103.802055 mapread 0x00128e thru 0x0012ed (0x60 bytes) 000525 1787854103.804434 trunc from 0x0012ee to 0x00ae29 (0x9b3c bytes) 000527 1787854103.824949 trunc from 0x00ae2a to 0x032ed3 (0x280aa bytes) 000529 1787854103.846818 write 0x03ef13 thru 0x03ffff (0x10ed bytes) 000530 1787854103.849441 read 0x023b87 thru 0x032c99 (0xf113 bytes) 000532 1787854103.865460 mapread 0x026dfa thru 0x034a31 (0xdc38 bytes) 000533 1787854103.866636 mapwrite 0x028b6b thru 0x028f76 (0x40c bytes) 000534 1787854103.908836 write 0x01d116 thru 0x01f8ba (0x27a5 bytes) 000535 1787854103.909861 read 0x029b9d thru 0x02e9b7 (0x4e1b bytes) 000536 1787854103.911054 trunc from 0x040000 to 0x03baab (0x4554 bytes) 000537 1787854103.926012 read 0x014a21 thru 0x014f8e (0x56e bytes) 000539 1787854103.932759 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3baac): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3baac): Mirror 6: 0x253970c6 (stale) Mirror 7: 0x253970c6 (stale) Mirror 8: 0x6ab65321 (stale) Mirror 9: 0x7e874254 (stale) Mirror 10: 0x7e874254 (stale) Mirror 11: 0x7e874254 (stale) Mirror 12: 0x7c9429d3 (stale) CRC-32 checksum value for chunk [0, 0x3baac): Mirror 1: 0x89793424 Mirror 13: 0x89793424 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3baac: skipped 000542 1787854105.008578 mapread 0x022a4a thru 0x0317db (0xed92 bytes) 000543 1787854105.038130 read 0x010580 thru 0x01628c (0x5d0d bytes) 000545 1787854105.045209 read 0x00590b thru 0x01322c (0xd922 bytes) 000546 1787854105.054977 write 0x013e20 thru 0x01762e (0x380f bytes) 000547 1787854105.132719 read 0x00c817 thru 0x013b90 (0x737a bytes) 000549 1787854105.143142 mapwrite 0x011baa thru 0x01abaa (0x9001 bytes) 000550 1787854105.187201 mapwrite 0x00c443 thru 0x0169d3 (0xa591 bytes) 000551 1787854105.232797 mapread 0x037649 thru 0x03b98f (0x4347 bytes) 000553 1787854105.235406 trunc from 0x03baac to 0x0053b0 (0x366fb bytes) 000554 1787854105.265242 MIRROR_RESYNC 9 mirrors 000555 1787854106.717851 mapread 0x0016a4 thru 0x002549 (0xea6 bytes) 000556 1787854106.727173 write 0x00a05f thru 0x00cd05 (0x2ca7 bytes) 000557 1787854106.797641 mapwrite 0x01f55f thru 0x02cc6e (0xd710 bytes) 000559 1787854106.912115 write 0x00a612 thru 0x010b9c (0x658b bytes) 000560 1787854106.918841 trunc from 0x02cc6f to 0x01f5c5 (0xd6a9 bytes) 000563 1787854106.938162 mapwrite 0x018632 thru 0x023d87 (0xb756 bytes) 000566 1787854107.024774 write 0x0134fb thru 0x018f90 (0x5a96 bytes) 000568 1787854107.032107 write 0x0002a9 thru 0x0017d5 (0x152d bytes) 000569 1787854107.044400 MIRROR_RESYNC 9 mirrors 000570 1787854109.232046 write 0x0093ee thru 0x010ee5 (0x7af8 bytes) 000572 1787854109.305083 read 0x005670 thru 0x0123f2 (0xcd83 bytes) 000574 1787854109.313349 trunc from 0x023d88 to 0x0153eb (0xe99c bytes) 000575 1787854109.334850 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x153ec): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x153ec): Mirror 6: 0x4c6bb4cf (stale) Mirror 7: 0x4c6bb4cf (stale) Mirror 8: 0x4c6bb4cf (stale) Mirror 9: 0x4c6bb4cf (stale) Mirror 10: 0x4c6bb4cf (stale) Mirror 11: 0x4c6bb4cf (stale) Mirror 12: 0x4c6bb4cf (stale) Mirror 13: 0x4c6bb4cf (stale) CRC-32 checksum value for chunk [0, 0x153ec): Mirror 1: 0xd490697f Mirror 14: 0xd490697f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x153ec: skipped 000576 1787854110.364110 read 0x006bc2 thru 0x00bf0e (0x534d bytes) 000577 1787854110.373573 read 0x00f814 thru 0x0153eb (0x5bd8 bytes) 000578 1787854110.382511 mapread 0x004c3b thru 0x00f003 (0xa3c9 bytes) 000581 1787854110.388104 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x153ec): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x153ec): Mirror 6: 0x4c6bb4cf (stale) Mirror 7: 0x4c6bb4cf (stale) Mirror 8: 0x4c6bb4cf (stale) Mirror 9: 0x4c6bb4cf (stale) Mirror 10: 0x4c6bb4cf (stale) Mirror 11: 0x4c6bb4cf (stale) Mirror 12: 0x4c6bb4cf (stale) Mirror 13: 0x4c6bb4cf (stale) CRC-32 checksum value for chunk [0, 0x153ec): Mirror 1: 0xd490697f Mirror 14: 0xd490697f Mirror 15: 0xd490697f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x153ec: skipped 000586 1787854111.229046 mapread 0x00df2a thru 0x0153eb (0x74c2 bytes) 000587 1787854111.247610 mapread 0x00a4ff thru 0x012832 (0x8334 bytes) 000588 1787854111.248366 write 0x025ec3 thru 0x027e09 (0x1f47 bytes) 000589 1787854111.330604 write 0x004521 thru 0x00fb7d (0xb65d bytes) 000590 1787854111.342208 trunc from 0x027e0a to 0x00fd76 (0x18093 bytes) 000591 1787854111.360443 mapwrite 0x01d5c3 thru 0x01e853 (0x1291 bytes) 000593 1787854111.433743 mapread 0x009036 thru 0x009350 (0x31b bytes) 000594 1787854111.435251 mapread 0x001c03 thru 0x00fc11 (0xe00f bytes) 000595 1787854111.441792 trunc from 0x01e854 to 0x03305b (0x14808 bytes) 000600 1787854111.460474 mapread 0x021aa0 thru 0x029c32 (0x8193 bytes) 000601 1787854111.468874 mapwrite 0x035575 thru 0x03ffff (0xaa8b bytes) 000602 1787854111.536843 mapwrite 0x030e1f thru 0x031f60 (0x1142 bytes) 000603 1787854111.572347 mapwrite 0x01bdf5 thru 0x029815 (0xda21 bytes) 000605 1787854111.620226 mapwrite 0x0028df thru 0x00e121 (0xb843 bytes) 000606 1787854111.665443 mapwrite 0x020b61 thru 0x021f4e (0x13ee bytes) 000608 1787854111.703666 mapwrite 0x02a629 thru 0x03715c (0xcb34 bytes) 000609 1787854111.765310 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xece734ea (stale) Mirror 7: 0xece734ea (stale) Mirror 8: 0xece734ea (stale) Mirror 9: 0xece734ea (stale) Mirror 10: 0xece734ea (stale) Mirror 11: 0xece734ea (stale) Mirror 12: 0xece734ea (stale) Mirror 13: 0xece734ea (stale) Mirror 15: 0x8a8ee5dc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xece734ea (stale) Mirror 7: 0xece734ea (stale) Mirror 8: 0xece734ea (stale) Mirror 9: 0xece734ea (stale) Mirror 10: 0xece734ea (stale) Mirror 11: 0xece734ea (stale) Mirror 12: 0xece734ea (stale) Mirror 13: 0xece734ea (stale) Mirror 15: 0x8a8ee5dc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000611 1787854112.234101 MIRROR_RESYNC 10 mirrors 000612 1787854114.292061 mapwrite 0x01af54 thru 0x026518 (0xb5c5 bytes) 000613 1787854114.479484 mapread 0x01cbe9 thru 0x027a4a (0xae62 bytes) 000614 1787854114.481403 mapread 0x00e9db thru 0x018a5b (0xa081 bytes) 000618 1787854114.493110 trunc from 0x040000 to 0x005de9 (0x3a216 bytes) 000619 1787854114.518097 read 0x001413 thru 0x005de9 (0x49d7 bytes) 000620 1787854114.532550 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x5dea): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5dea): Mirror 6: 0xd7bdee6c (stale) Mirror 7: 0xd7bdee6c (stale) Mirror 8: 0xd7bdee6c (stale) Mirror 9: 0xd7bdee6c (stale) Mirror 10: 0xd7bdee6c (stale) Mirror 11: 0xd7bdee6c (stale) Mirror 12: 0xd7bdee6c (stale) Mirror 13: 0xd7bdee6c (stale) Mirror 15: 0xd7bdee6c (stale) CRC-32 checksum value for chunk [0, 0x5dea): Mirror 1: 0xd7bdee6c Mirror 16: 0xd7bdee6c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5dea: skipped 000622 1787854115.216250 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x5dea): Mirror 6: 0xd7bdee6c (stale) Mirror 7: 0xd7bdee6c (stale) Mirror 8: 0xd7bdee6c (stale) Mirror 10: 0xd7bdee6c (stale) Mirror 11: 0xd7bdee6c (stale) Mirror 12: 0xd7bdee6c (stale) Mirror 13: 0xd7bdee6c (stale) Mirror 15: 0xd7bdee6c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5dea): Mirror 6: 0xd7bdee6c (stale) Mirror 7: 0xd7bdee6c (stale) Mirror 8: 0xd7bdee6c (stale) Mirror 10: 0xd7bdee6c (stale) Mirror 11: 0xd7bdee6c (stale) Mirror 12: 0xd7bdee6c (stale) Mirror 13: 0xd7bdee6c (stale) Mirror 15: 0xd7bdee6c (stale) CRC-32 checksum value for chunk [0, 0x5dea): Mirror 1: 0xd7bdee6c Mirror 16: 0xd7bdee6c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5dea: skipped 000623 1787854115.682447 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x5dea): Mirror 6: 0xd7bdee6c (stale) Mirror 8: 0xd7bdee6c (stale) Mirror 10: 0xd7bdee6c (stale) Mirror 11: 0xd7bdee6c (stale) Mirror 12: 0xd7bdee6c (stale) Mirror 13: 0xd7bdee6c (stale) Mirror 15: 0xd7bdee6c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5dea): Mirror 6: 0xd7bdee6c (stale) Mirror 8: 0xd7bdee6c (stale) Mirror 10: 0xd7bdee6c (stale) Mirror 11: 0xd7bdee6c (stale) Mirror 12: 0xd7bdee6c (stale) Mirror 13: 0xd7bdee6c (stale) Mirror 15: 0xd7bdee6c (stale) CRC-32 checksum value for chunk [0, 0x5dea): Mirror 1: 0xd7bdee6c Mirror 16: 0xd7bdee6c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5dea: skipped 000625 1787854116.155670 write 0x03595b thru 0x03ffff (0xa6a5 bytes) 000628 1787854116.267498 write 0x018282 thru 0x019c4c (0x19cb bytes) 000630 1787854116.278731 read 0x023310 thru 0x02c714 (0x9405 bytes) 000631 1787854116.288531 write 0x0248d5 thru 0x027b50 (0x327c bytes) 000632 1787854116.290229 read 0x02efe5 thru 0x03cd9c (0xddb8 bytes) 000633 1787854116.300973 trunc from 0x040000 to 0x0168fa (0x29705 bytes) 000636 1787854116.353962 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x168fb): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x168fb): Mirror 6: 0x570ddf99 (stale) Mirror 8: 0x570ddf99 (stale) Mirror 10: 0x570ddf99 (stale) Mirror 11: 0x570ddf99 (stale) Mirror 12: 0x570ddf99 (stale) Mirror 13: 0x570ddf99 (stale) Mirror 15: 0x570ddf99 (stale) Mirror 16: 0x5b3f6c29 (stale) CRC-32 checksum value for chunk [0, 0x168fb): Mirror 1: 0x5b3f6c29 Mirror 17: 0x5b3f6c29 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x168fb: skipped 000637 1787854117.147214 write 0x02bc54 thru 0x031697 (0x5a44 bytes) 000638 1787854117.222187 MIRROR_RESYNC 10 mirrors 000640 1787854119.079543 write 0x032789 thru 0x036966 (0x41de bytes) 000641 1787854119.160589 read 0x01202c thru 0x017b0a (0x5adf bytes) 000642 1787854119.167618 mapwrite 0x039734 thru 0x03ffff (0x68cc bytes) 000643 1787854119.266505 mapwrite 0x01bd29 thru 0x01d522 (0x17fa bytes) 000644 1787854119.335679 mapwrite 0x03777d thru 0x03faa4 (0x8328 bytes) 000645 1787854119.398090 mapread 0x002a05 thru 0x012144 (0xf740 bytes) 000646 1787854119.412366 read 0x02c7bb thru 0x030608 (0x3e4e bytes) 000647 1787854119.422742 mapread 0x0187f0 thru 0x02028a (0x7a9b bytes) 000651 1787854119.425565 read 0x02192f thru 0x028ec7 (0x7599 bytes) 000652 1787854119.426692 read 0x00a813 thru 0x011a15 (0x7203 bytes) 000653 1787854119.427511 read 0x028872 thru 0x028f37 (0x6c6 bytes) 000654 1787854119.428372 trunc from 0x040000 to 0x025921 (0x1a6de bytes) 000656 1787854119.450837 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x25922): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25922): Mirror 6: 0x6646f3fd (stale) Mirror 8: 0x6646f3fd (stale) Mirror 10: 0x6646f3fd (stale) Mirror 11: 0x6646f3fd (stale) Mirror 12: 0x6646f3fd (stale) Mirror 13: 0x6646f3fd (stale) Mirror 15: 0x6646f3fd (stale) Mirror 16: 0x6646f3fd (stale) Mirror 17: 0x6646f3fd (stale) CRC-32 checksum value for chunk [0, 0x25922): Mirror 1: 0x61904a90 Mirror 18: 0x61904a90 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25922: skipped 000657 1787854120.407495 trunc from 0x025922 to 0x025174 (0x7ad bytes) 000661 1787854120.572477 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x25175): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25175): Mirror 6: 0x1401e661 (stale) Mirror 8: 0x1401e661 (stale) Mirror 10: 0x1401e661 (stale) Mirror 11: 0x1401e661 (stale) Mirror 12: 0x1401e661 (stale) Mirror 13: 0x1401e661 (stale) Mirror 15: 0x1401e661 (stale) Mirror 16: 0x1401e661 (stale) Mirror 17: 0x1401e661 (stale) Mirror 18: 0x2a06a676 (stale) CRC-32 checksum value for chunk [0, 0x25175): Mirror 1: 0x2a06a676 Mirror 19: 0x2a06a676 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25175: skipped 000663 1787854122.018812 write 0x03bc9d thru 0x03ffff (0x4363 bytes) 000664 1787854122.206167 MIRROR_SPLIT mirror 2 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xfa1d5e58 (stale) Mirror 10: 0xfa1d5e58 (stale) Mirror 11: 0xfa1d5e58 (stale) Mirror 12: 0xfa1d5e58 (stale) Mirror 13: 0xfa1d5e58 (stale) Mirror 15: 0xfa1d5e58 (stale) Mirror 16: 0xfa1d5e58 (stale) Mirror 17: 0xfa1d5e58 (stale) Mirror 18: 0xaa6ead9f (stale) Mirror 19: 0xaa6ead9f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xfa1d5e58 (stale) Mirror 10: 0xfa1d5e58 (stale) Mirror 11: 0xfa1d5e58 (stale) Mirror 12: 0xfa1d5e58 (stale) Mirror 13: 0xfa1d5e58 (stale) Mirror 15: 0xfa1d5e58 (stale) Mirror 16: 0xfa1d5e58 (stale) Mirror 17: 0xfa1d5e58 (stale) Mirror 18: 0xaa6ead9f (stale) Mirror 19: 0xaa6ead9f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000666 1787854122.972616 read 0x01d558 thru 0x01e298 (0xd41 bytes) 000668 1787854122.980337 mapread 0x01a634 thru 0x01d50b (0x2ed8 bytes) 000669 1787854123.004096 mapwrite 0x01fc1e thru 0x020dc2 (0x11a5 bytes) 000670 1787854123.070724 read 0x031c80 thru 0x03ad41 (0x90c2 bytes) 000671 1787854123.093449 mapwrite 0x0210f3 thru 0x028cf6 (0x7c04 bytes) 000672 1787854123.195441 mapwrite 0x0302fc thru 0x03deff (0xdc04 bytes) 000673 1787854123.230865 mapread 0x00cbd2 thru 0x0183d0 (0xb7ff bytes) 000674 1787854123.231577 write 0x02a538 thru 0x02f305 (0x4dce bytes) 000675 1787854123.232770 mapwrite 0x038545 thru 0x03ffff (0x7abb bytes) 000676 1787854123.269542 mapread 0x0281fe thru 0x031cab (0x9aae bytes) 000678 1787854123.271536 mapwrite 0x035fd6 thru 0x03c59a (0x65c5 bytes) 000680 1787854123.318342 write 0x017e1d thru 0x0185b1 (0x795 bytes) 000684 1787854123.326220 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xfa1d5e58 (stale) Mirror 10: 0xfa1d5e58 (stale) Mirror 11: 0xfa1d5e58 (stale) Mirror 12: 0xfa1d5e58 (stale) Mirror 13: 0xfa1d5e58 (stale) Mirror 15: 0xfa1d5e58 (stale) Mirror 16: 0xfa1d5e58 (stale) Mirror 17: 0xfa1d5e58 (stale) Mirror 18: 0xaa6ead9f (stale) Mirror 19: 0xaa6ead9f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x16e7ef31 Mirror 20: 0x16e7ef31 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000685 1787854125.024630 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xfa1d5e58 (stale) Mirror 10: 0xfa1d5e58 (stale) Mirror 11: 0xfa1d5e58 (stale) Mirror 12: 0xfa1d5e58 (stale) Mirror 13: 0xfa1d5e58 (stale) Mirror 15: 0xfa1d5e58 (stale) Mirror 16: 0xfa1d5e58 (stale) Mirror 17: 0xfa1d5e58 (stale) Mirror 18: 0xaa6ead9f (stale) Mirror 19: 0xaa6ead9f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xfa1d5e58 (stale) Mirror 10: 0xfa1d5e58 (stale) Mirror 11: 0xfa1d5e58 (stale) Mirror 12: 0xfa1d5e58 (stale) Mirror 13: 0xfa1d5e58 (stale) Mirror 15: 0xfa1d5e58 (stale) Mirror 16: 0xfa1d5e58 (stale) Mirror 17: 0xfa1d5e58 (stale) Mirror 18: 0xaa6ead9f (stale) Mirror 19: 0xaa6ead9f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000687 1787854125.965430 write 0x018176 thru 0x026b50 (0xe9db bytes) 000689 1787854125.988079 mapwrite 0x0002d5 thru 0x00bea3 (0xbbcf bytes) 000691 1787854126.072983 write 0x008020 thru 0x00bb9e (0x3b7f bytes) 000692 1787854126.074043 trunc from 0x040000 to 0x01f0e8 (0x20f17 bytes) 000695 1787854126.098527 read 0x017bd4 thru 0x01b1fa (0x3627 bytes) 000696 1787854126.104809 mapwrite 0x01037d thru 0x015b5d (0x57e1 bytes) 000697 1787854126.171565 trunc from 0x01f0e9 to 0x03eef0 (0x1fe08 bytes) 000700 1787854126.212670 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3eef1): Mirror 6: 0x6f413ac8 (stale) Mirror 10: 0x6f413ac8 (stale) Mirror 11: 0x6f413ac8 (stale) Mirror 12: 0x6f413ac8 (stale) Mirror 13: 0x6f413ac8 (stale) Mirror 15: 0x6f413ac8 (stale) Mirror 16: 0x6f413ac8 (stale) Mirror 17: 0x6f413ac8 (stale) Mirror 19: 0xd08e096 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eef1): Mirror 6: 0x6f413ac8 (stale) Mirror 10: 0x6f413ac8 (stale) Mirror 11: 0x6f413ac8 (stale) Mirror 12: 0x6f413ac8 (stale) Mirror 13: 0x6f413ac8 (stale) Mirror 15: 0x6f413ac8 (stale) Mirror 16: 0x6f413ac8 (stale) Mirror 17: 0x6f413ac8 (stale) Mirror 19: 0xd08e096 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eef1: skipped 000701 1787854127.079420 mapwrite 0x03c1ac thru 0x03eb86 (0x29db bytes) 000705 1787854127.163468 trunc from 0x03eef1 to 0x0201ed (0x1ed03 bytes) 000708 1787854127.186658 mapwrite 0x021072 thru 0x02aa23 (0x99b2 bytes) 000711 1787854127.307317 read 0x0214b0 thru 0x02aa23 (0x9574 bytes) 000712 1787854127.307979 trunc from 0x02aa24 to 0x011ad1 (0x18f52 bytes) 000713 1787854127.332793 trunc from 0x011ad2 to 0x00b34c (0x6785 bytes) 000714 1787854127.383205 write 0x031f3d thru 0x037c5a (0x5d1e bytes) 000717 1787854127.395967 write 0x018a28 thru 0x025ad3 (0xd0ac bytes) 000720 1787854127.410032 mapwrite 0x01fa6e thru 0x027e73 (0x8406 bytes) 000721 1787854127.458524 read 0x00e43d thru 0x014d92 (0x6956 bytes) 000722 1787854127.465766 mapwrite 0x002648 thru 0x00aa34 (0x83ed bytes) 000723 1787854127.516242 read 0x00ea10 thru 0x01704a (0x863b bytes) 000724 1787854127.516813 mapread 0x021a3f thru 0x0268dc (0x4e9e bytes) 000725 1787854127.517714 write 0x036740 thru 0x03e3e0 (0x7ca1 bytes) 000726 1787854127.519585 mapwrite 0x01ea98 thru 0x021d8a (0x32f3 bytes) 000730 1787854127.594434 read 0x03974f thru 0x03e3e0 (0x4c92 bytes) 000732 1787854127.595505 mapwrite 0x014c31 thru 0x021a4e (0xce1e bytes) 000734 1787854127.706704 trunc from 0x03e3e1 to 0x02037b (0x1e065 bytes) 000736 1787854127.754109 write 0x00721d thru 0x012152 (0xaf36 bytes) 000737 1787854127.760924 mapwrite 0x02d377 thru 0x035084 (0x7d0e bytes) 000738 1787854127.841451 MIRROR_RESYNC 10 mirrors 000741 1787854131.394646 read 0x0201c9 thru 0x02ef18 (0xed50 bytes) 000746 1787854131.410130 write 0x0057c2 thru 0x008b19 (0x3358 bytes) 000747 1787854131.494700 read 0x02ad7f thru 0x033785 (0x8a07 bytes) 000749 1787854131.505792 mapread 0x033964 thru 0x0343d6 (0xa73 bytes) 000750 1787854131.506636 write 0x0092cd thru 0x01916d (0xfea1 bytes) 000751 1787854131.507601 read 0x00cc7e thru 0x00d378 (0x6fb bytes) 000752 1787854131.508143 write 0x00e728 thru 0x01b788 (0xd061 bytes) 000754 1787854131.509566 trunc from 0x035085 to 0x024816 (0x1086e bytes) 000755 1787854131.531450 write 0x016e4c thru 0x020c29 (0x9dde bytes) 000758 1787854131.535454 mapwrite 0x0212b7 thru 0x029e96 (0x8be0 bytes) 000761 1787854131.614165 mapread 0x01b79f thru 0x0256dd (0x9f3f bytes) 000763 1787854131.616331 write 0x009a72 thru 0x012ca6 (0x9235 bytes) 000764 1787854131.617876 mapread 0x00232b thru 0x00cfdf (0xacb5 bytes) 000765 1787854131.618898 mapread 0x00fa68 thru 0x0139dd (0x3f76 bytes) 000766 1787854131.623636 MIRROR_RESYNC 10 mirrors 000768 1787854133.109196 mapwrite 0x008466 thru 0x010a40 (0x85db bytes) 000770 1787854133.286202 trunc from 0x029e97 to 0x03e166 (0x142d0 bytes) 000774 1787854133.313136 write 0x02cb99 thru 0x037d7f (0xb1e7 bytes) 000776 1787854133.331470 MIRROR_RESYNC 10 mirrors 000781 1787854136.363886 MIRROR_RESYNC 10 mirrors 000783 1787854136.895584 trunc from 0x03e167 to 0x004439 (0x39d2d bytes) 000785 1787854137.008393 write 0x03315a thru 0x03f671 (0xc518 bytes) 000787 1787854137.019230 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f672): Mirror 6: 0x563949d6 (stale) Mirror 10: 0x563949d6 (stale) Mirror 11: 0x563949d6 (stale) Mirror 12: 0x563949d6 (stale) Mirror 15: 0x563949d6 (stale) Mirror 16: 0x563949d6 (stale) Mirror 17: 0x563949d6 (stale) Mirror 19: 0x563949d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f672): Mirror 6: 0x563949d6 (stale) Mirror 10: 0x563949d6 (stale) Mirror 11: 0x563949d6 (stale) Mirror 12: 0x563949d6 (stale) Mirror 15: 0x563949d6 (stale) Mirror 16: 0x563949d6 (stale) Mirror 17: 0x563949d6 (stale) Mirror 19: 0x563949d6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f672: skipped 000788 1787854137.539050 read 0x01c27f thru 0x01ca09 (0x78b bytes) 000791 1787854137.548874 MIRROR_RESYNC 9 mirrors 000793 1787854139.259246 mapread 0x00a819 thru 0x014138 (0x9920 bytes) 000794 1787854139.279611 trunc from 0x03f672 to 0x00e9e5 (0x30c8c bytes) 000797 1787854139.377721 write 0x02e6f5 thru 0x038719 (0xa025 bytes) 000798 1787854139.380180 write 0x0357ba thru 0x03833a (0x2b81 bytes) 000804 1787854139.385168 mapwrite 0x010941 thru 0x01f462 (0xeb22 bytes) 000805 1787854139.474838 read 0x02c2cf thru 0x02fb88 (0x38ba bytes) 000807 1787854139.481228 mapread 0x03466e thru 0x038719 (0x40ac bytes) 000808 1787854139.482136 read 0x0382a4 thru 0x038719 (0x476 bytes) 000810 1787854139.483192 mapwrite 0x0256a6 thru 0x02aeb9 (0x5814 bytes) 000811 1787854139.548575 MIRROR_RESYNC 9 mirrors 000814 1787854142.210062 MIRROR_EXTEND to 10 mirrors 000815 1787854143.076279 mapread 0x019b66 thru 0x01f37c (0x5817 bytes) 000816 1787854143.094202 trunc from 0x03871a to 0x00dcb4 (0x2aa65 bytes) 000817 1787854143.178675 mapwrite 0x0038e4 thru 0x0134b4 (0xfbd1 bytes) 000819 1787854143.266755 trunc from 0x0134b5 to 0x018afb (0x5647 bytes) 000820 1787854143.281211 trunc from 0x018afc to 0x0144bc (0x463f bytes) 000822 1787854143.297725 mapread 0x008188 thru 0x010a91 (0x890a bytes) 000824 1787854143.300751 mapwrite 0x03910e thru 0x03ffff (0x6ef2 bytes) 000826 1787854143.366064 mapwrite 0x00dff3 thru 0x01cfd3 (0xefe1 bytes) 000827 1787854143.420562 write 0x002c9e thru 0x00f86f (0xcbd2 bytes) 000828 1787854143.421897 mapread 0x021390 thru 0x024db5 (0x3a26 bytes) 000831 1787854143.435880 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x19d9ea97 (stale) Mirror 11: 0x19d9ea97 (stale) Mirror 12: 0x19d9ea97 (stale) Mirror 15: 0x19d9ea97 (stale) Mirror 16: 0x19d9ea97 (stale) Mirror 17: 0x19d9ea97 (stale) Mirror 19: 0x19d9ea97 (stale) Mirror 20: 0x19d9ea97 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x19d9ea97 (stale) Mirror 11: 0x19d9ea97 (stale) Mirror 12: 0x19d9ea97 (stale) Mirror 15: 0x19d9ea97 (stale) Mirror 16: 0x19d9ea97 (stale) Mirror 17: 0x19d9ea97 (stale) Mirror 19: 0x19d9ea97 (stale) Mirror 20: 0x19d9ea97 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000832 1787854143.927089 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x19d9ea97 (stale) Mirror 11: 0x19d9ea97 (stale) Mirror 12: 0x19d9ea97 (stale) Mirror 15: 0x19d9ea97 (stale) Mirror 16: 0x19d9ea97 (stale) Mirror 17: 0x19d9ea97 (stale) Mirror 19: 0x19d9ea97 (stale) Mirror 20: 0x19d9ea97 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd1955cd9 Mirror 21: 0xd1955cd9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000838 1787854144.976146 mapwrite 0x00426b thru 0x00d12e (0x8ec4 bytes) 000841 1787854145.133195 mapwrite 0x006e5c thru 0x0107fd (0x99a2 bytes) 000842 1787854145.180122 mapread 0x015903 thru 0x01a63d (0x4d3b bytes) 000845 1787854145.189219 mapwrite 0x02f224 thru 0x03ee8b (0xfc68 bytes) 000846 1787854145.256842 mapread 0x03a639 thru 0x03f4e6 (0x4eae bytes) 000847 1787854145.257794 write 0x03f171 thru 0x03ffff (0xe8f bytes) 000849 1787854145.263959 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x19d9ea97 (stale) Mirror 11: 0x19d9ea97 (stale) Mirror 12: 0x19d9ea97 (stale) Mirror 15: 0x19d9ea97 (stale) Mirror 16: 0x19d9ea97 (stale) Mirror 17: 0x19d9ea97 (stale) Mirror 19: 0x19d9ea97 (stale) Mirror 20: 0x19d9ea97 (stale) Mirror 21: 0xd1955cd9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa6a8c5f7 Mirror 22: 0xa6a8c5f7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000851 1787854146.563011 write 0x008b1a thru 0x015a55 (0xcf3c bytes) 000855 1787854146.721661 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x19d9ea97 (stale) Mirror 11: 0x19d9ea97 (stale) Mirror 12: 0x19d9ea97 (stale) Mirror 15: 0x19d9ea97 (stale) Mirror 16: 0x19d9ea97 (stale) Mirror 17: 0x19d9ea97 (stale) Mirror 19: 0x19d9ea97 (stale) Mirror 20: 0x19d9ea97 (stale) Mirror 21: 0xd1955cd9 (stale) Mirror 22: 0xa6a8c5f7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x79f20533 Mirror 23: 0x79f20533 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000857 1787854148.383504 trunc from 0x040000 to 0x00349a (0x3cb65 bytes) 000860 1787854148.474394 read 0x000eb2 thru 0x00349a (0x25e9 bytes) 000862 1787854148.481244 read 0x002836 thru 0x00349a (0xc65 bytes) 000864 1787854148.482292 trunc from 0x00349b to 0x0269d7 (0x2353d bytes) 000866 1787854148.506500 trunc from 0x0269d8 to 0x0060a9 (0x2092e bytes) 000869 1787854148.527695 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x60aa): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x60aa): Mirror 10: 0xccd3b340 (stale) Mirror 11: 0xccd3b340 (stale) Mirror 12: 0xccd3b340 (stale) Mirror 15: 0xccd3b340 (stale) Mirror 16: 0xccd3b340 (stale) Mirror 17: 0xccd3b340 (stale) Mirror 19: 0xccd3b340 (stale) Mirror 20: 0xccd3b340 (stale) Mirror 21: 0xca463535 (stale) Mirror 22: 0xd7789f80 (stale) Mirror 23: 0xd7789f80 (stale) CRC-32 checksum value for chunk [0, 0x60aa): Mirror 1: 0x812fad4 Mirror 24: 0x812fad4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x60aa: skipped 000870 1787854149.534516 read 0x001f21 thru 0x0060a9 (0x4189 bytes) 000871 1787854149.545541 write 0x0050b3 thru 0x011c53 (0xcba1 bytes) 000872 1787854149.651888 MIRROR_RESYNC 13 mirrors 000873 1787854152.530275 MIRROR_RESYNC 13 mirrors 000876 1787854153.011259 read 0x0087ee thru 0x011963 (0x9176 bytes) 000877 1787854153.022108 MIRROR_SPLIT mirror 10 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) 000878 1787854153.326720 trunc from 0x011c54 to 0x039a8a (0x27e37 bytes) 000879 1787854153.469539 mapread 0x032aaa thru 0x037eac (0x5403 bytes) 000882 1787854153.487272 write 0x014c81 thru 0x020690 (0xba10 bytes) 000883 1787854153.515559 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x39a8b): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a8b): Mirror 10: 0x4b2527a8 (stale) Mirror 11: 0x4b2527a8 (stale) Mirror 12: 0x4b2527a8 (stale) Mirror 15: 0x4b2527a8 (stale) Mirror 16: 0x4b2527a8 (stale) Mirror 17: 0x4b2527a8 (stale) Mirror 19: 0x4b2527a8 (stale) Mirror 20: 0x4b2527a8 (stale) Mirror 21: 0x4b2527a8 (stale) Mirror 23: 0x4b2527a8 (stale) Mirror 24: 0x4b2527a8 (stale) CRC-32 checksum value for chunk [0, 0x39a8b): Mirror 1: 0x6126792b Mirror 25: 0x6126792b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a8b: skipped 000884 1787854154.500563 read 0x032c4a thru 0x039a8a (0x6e41 bytes) 000885 1787854154.510032 trunc from 0x039a8b to 0x030140 (0x994a bytes) 000886 1787854154.663666 MIRROR_RESYNC 13 mirrors 000887 1787854159.169238 mapread 0x0246a2 thru 0x02a790 (0x60ef bytes) 000888 1787854159.196499 MIRROR_RESYNC 13 mirrors 000889 1787854159.913098 trunc from 0x030141 to 0x02d773 (0x29cd bytes) 000891 1787854160.014802 MIRROR_RESYNC 13 mirrors 000892 1787854163.794984 MIRROR_SPLIT mirror 8 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) 000894 1787854164.147345 read 0x000c59 thru 0x003502 (0x28aa bytes) 000895 1787854164.161801 read 0x00b732 thru 0x013b0d (0x83dc bytes) 000898 1787854164.200676 trunc from 0x02d774 to 0x01d0d4 (0x1069f bytes) 000901 1787854164.315752 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x1d0d5): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d0d5): Mirror 10: 0x4925a350 (stale) Mirror 11: 0x4925a350 (stale) Mirror 12: 0x4925a350 (stale) Mirror 15: 0x4925a350 (stale) Mirror 16: 0x4925a350 (stale) Mirror 17: 0x4925a350 (stale) Mirror 19: 0x4925a350 (stale) Mirror 21: 0x4925a350 (stale) Mirror 23: 0x4925a350 (stale) Mirror 24: 0x4925a350 (stale) Mirror 25: 0x4925a350 (stale) CRC-32 checksum value for chunk [0, 0x1d0d5): Mirror 1: 0x4925a350 Mirror 26: 0x4925a350 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d0d5: skipped 000902 1787854165.689780 read 0x013d37 thru 0x01869a (0x4964 bytes) 000903 1787854165.697239 read 0x0041c2 thru 0x008fa0 (0x4ddf bytes) 000905 1787854165.706402 write 0x0116f0 thru 0x01f96a (0xe27b bytes) 000906 1787854165.827372 read 0x017fb0 thru 0x01f96a (0x79bb bytes) 000907 1787854165.828090 mapread 0x015734 thru 0x01f96a (0xa237 bytes) 000908 1787854165.828816 trunc from 0x01f96b to 0x02210e (0x27a4 bytes) 000909 1787854165.876362 mapread 0x011649 thru 0x020aac (0xf464 bytes) 000911 1787854165.903458 trunc from 0x02210f to 0x017f5f (0xa1af bytes) 000913 1787854165.924829 trunc from 0x017f60 to 0x02fdf8 (0x17e99 bytes) 000915 1787854165.951401 mapread 0x01f374 thru 0x020a2e (0x16bb bytes) 000916 1787854165.966803 trunc from 0x02fdf9 to 0x0261f1 (0x9c07 bytes) 000917 1787854166.004282 trunc from 0x0261f2 to 0x031ba8 (0xb9b7 bytes) 000918 1787854166.020481 mapwrite 0x02eb4a thru 0x034484 (0x593b bytes) 000920 1787854166.087233 trunc from 0x034485 to 0x028d56 (0xb72e bytes) 000921 1787854166.107566 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x28d57): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28d57): Mirror 10: 0x8fdc5d88 (stale) Mirror 11: 0x8fdc5d88 (stale) Mirror 12: 0x8fdc5d88 (stale) Mirror 15: 0x8fdc5d88 (stale) Mirror 16: 0x8fdc5d88 (stale) Mirror 17: 0x8fdc5d88 (stale) Mirror 19: 0x8fdc5d88 (stale) Mirror 21: 0x8fdc5d88 (stale) Mirror 23: 0x8fdc5d88 (stale) Mirror 24: 0x8fdc5d88 (stale) Mirror 25: 0x8fdc5d88 (stale) Mirror 26: 0x2dc2b1 (stale) CRC-32 checksum value for chunk [0, 0x28d57): Mirror 1: 0x7255a9 Mirror 27: 0x7255a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28d57: skipped 000922 1787854167.346289 mapread 0x01314a thru 0x019615 (0x64cc bytes) 000923 1787854167.386184 MIRROR_SPLIT mirror 12 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28d57): Mirror 10: 0x8fdc5d88 (stale) Mirror 11: 0x8fdc5d88 (stale) Mirror 12: 0x8fdc5d88 (stale) Mirror 15: 0x8fdc5d88 (stale) Mirror 16: 0x8fdc5d88 (stale) Mirror 17: 0x8fdc5d88 (stale) Mirror 19: 0x8fdc5d88 (stale) Mirror 21: 0x8fdc5d88 (stale) Mirror 23: 0x8fdc5d88 (stale) Mirror 24: 0x8fdc5d88 (stale) Mirror 25: 0x8fdc5d88 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28d57): Mirror 10: 0x8fdc5d88 (stale) Mirror 11: 0x8fdc5d88 (stale) Mirror 12: 0x8fdc5d88 (stale) Mirror 15: 0x8fdc5d88 (stale) Mirror 16: 0x8fdc5d88 (stale) Mirror 17: 0x8fdc5d88 (stale) Mirror 19: 0x8fdc5d88 (stale) Mirror 21: 0x8fdc5d88 (stale) Mirror 23: 0x8fdc5d88 (stale) Mirror 24: 0x8fdc5d88 (stale) Mirror 25: 0x8fdc5d88 (stale) CRC-32 checksum value for chunk [0, 0x28d57): Mirror 1: 0x7255a9 Mirror 27: 0x7255a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28d57: skipped 000924 1787854168.242994 read 0x00c8e6 thru 0x013beb (0x7306 bytes) 000925 1787854168.253720 trunc from 0x028d57 to 0x014577 (0x147df bytes) 000926 1787854168.348754 mapread 0x012d95 thru 0x014577 (0x17e3 bytes) 000927 1787854168.365746 trunc from 0x014578 to 0x036def (0x22878 bytes) 000928 1787854168.383086 read 0x010a63 thru 0x013112 (0x26b0 bytes) 000929 1787854168.384189 mapwrite 0x0095de thru 0x017ad6 (0xe4f9 bytes) 000931 1787854168.486143 MIRROR_SPLIT mirror 2 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36df0): Mirror 10: 0x33ee522b (stale) Mirror 12: 0x33ee522b (stale) Mirror 15: 0x33ee522b (stale) Mirror 16: 0x33ee522b (stale) Mirror 17: 0x33ee522b (stale) Mirror 19: 0x33ee522b (stale) Mirror 21: 0x33ee522b (stale) Mirror 23: 0x33ee522b (stale) Mirror 24: 0x33ee522b (stale) Mirror 25: 0x33ee522b (stale) Mirror 27: 0xc4c0c115 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36df0): Mirror 10: 0x33ee522b (stale) Mirror 12: 0x33ee522b (stale) Mirror 15: 0x33ee522b (stale) Mirror 16: 0x33ee522b (stale) Mirror 17: 0x33ee522b (stale) Mirror 19: 0x33ee522b (stale) Mirror 21: 0x33ee522b (stale) Mirror 23: 0x33ee522b (stale) Mirror 24: 0x33ee522b (stale) Mirror 25: 0x33ee522b (stale) Mirror 27: 0xc4c0c115 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36df0: skipped 000933 1787854169.157570 mapwrite 0x02c01a thru 0x038d97 (0xcd7e bytes) 000934 1787854169.288547 read 0x0211ae thru 0x02bcf1 (0xab44 bytes) 000935 1787854169.298753 mapread 0x019eb8 thru 0x022d7e (0x8ec7 bytes) 000936 1787854169.307104 mapwrite 0x03f939 thru 0x03ffff (0x6c7 bytes) 000939 1787854169.367873 mapread 0x02a49f thru 0x0350f5 (0xac57 bytes) 000942 1787854169.374584 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x8bbe0283 (stale) Mirror 12: 0x8bbe0283 (stale) Mirror 16: 0x8bbe0283 (stale) Mirror 17: 0x8bbe0283 (stale) Mirror 19: 0x8bbe0283 (stale) Mirror 21: 0x8bbe0283 (stale) Mirror 23: 0x8bbe0283 (stale) Mirror 24: 0x8bbe0283 (stale) Mirror 25: 0x8bbe0283 (stale) Mirror 27: 0x30b6da7d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x8bbe0283 (stale) Mirror 12: 0x8bbe0283 (stale) Mirror 16: 0x8bbe0283 (stale) Mirror 17: 0x8bbe0283 (stale) Mirror 19: 0x8bbe0283 (stale) Mirror 21: 0x8bbe0283 (stale) Mirror 23: 0x8bbe0283 (stale) Mirror 24: 0x8bbe0283 (stale) Mirror 25: 0x8bbe0283 (stale) Mirror 27: 0x30b6da7d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000946 1787854169.839484 read 0x005162 thru 0x006c03 (0x1aa2 bytes) 000947 1787854169.845513 read 0x03b62a thru 0x03ffff (0x49d6 bytes) 000948 1787854169.860028 MIRROR_RESYNC 11 mirrors 000949 1787854173.631357 MIRROR_RESYNC 11 mirrors 000950 1787854174.321386 mapwrite 0x03b55f thru 0x03ffff (0x4aa1 bytes) 000951 1787854174.758261 mapwrite 0x0320c8 thru 0x03400b (0x1f44 bytes) 000955 1787854174.897391 read 0x02c712 thru 0x0306be (0x3fad bytes) 000956 1787854174.901198 mapwrite 0x0281e4 thru 0x0334d9 (0xb2f6 bytes) 000957 1787854175.034735 trunc from 0x040000 to 0x018c45 (0x273ba bytes) 000959 1787854175.069199 trunc from 0x018c46 to 0x03239f (0x1975a bytes) 000960 1787854175.096292 trunc from 0x0323a0 to 0x022cf7 (0xf6a8 bytes) 000961 1787854175.128146 trunc from 0x022cf8 to 0x007eb7 (0x1ae40 bytes) 000962 1787854175.165545 mapwrite 0x00accf thru 0x016579 (0xb8ab bytes) 000963 1787854175.254571 read 0x011775 thru 0x012c19 (0x14a5 bytes) 000965 1787854175.261247 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x1657a): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1657a): Mirror 10: 0x8986fd02 (stale) Mirror 12: 0x8986fd02 (stale) Mirror 16: 0x8986fd02 (stale) Mirror 17: 0x8986fd02 (stale) Mirror 19: 0x8986fd02 (stale) Mirror 21: 0x8986fd02 (stale) Mirror 23: 0x8986fd02 (stale) Mirror 24: 0x8986fd02 (stale) Mirror 25: 0x8986fd02 (stale) Mirror 27: 0x8986fd02 (stale) CRC-32 checksum value for chunk [0, 0x1657a): Mirror 1: 0xadc2fc61 Mirror 28: 0xadc2fc61 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1657a: skipped 000967 1787854176.437163 read 0x012ecf thru 0x016579 (0x36ab bytes) 000970 1787854176.452862 mapread 0x005f16 thru 0x01174b (0xb836 bytes) 000971 1787854176.491330 mapwrite 0x017974 thru 0x01f265 (0x78f2 bytes) 000972 1787854176.893206 mapread 0x009c71 thru 0x010cc2 (0x7052 bytes) 000973 1787854176.908578 mapread 0x01b511 thru 0x01f265 (0x3d55 bytes) 000975 1787854176.911677 mapread 0x006ea5 thru 0x015b9f (0xecfb bytes) 000976 1787854176.913121 mapwrite 0x0075fd thru 0x012347 (0xad4b bytes) 000979 1787854177.056367 read 0x00427e thru 0x00d362 (0x90e5 bytes) 000980 1787854177.057662 trunc from 0x01f266 to 0x02258d (0x3328 bytes) 000981 1787854177.090067 mapwrite 0x02dc91 thru 0x03bc6c (0xdfdc bytes) 000982 1787854177.266397 trunc from 0x03bc6d to 0x030fcb (0xaca1 bytes) 000983 1787854177.284510 mapread 0x0087da thru 0x0134ca (0xacf1 bytes) 000984 1787854177.285620 mapread 0x026b1e thru 0x030b84 (0xa067 bytes) 000985 1787854177.289849 mapwrite 0x037e1f thru 0x03bcb1 (0x3e93 bytes) 000988 1787854177.442392 mapwrite 0x018ec6 thru 0x01b4f1 (0x262c bytes) 000989 1787854177.507086 trunc from 0x03bcb2 to 0x02e4bd (0xd7f4 bytes) 000990 1787854177.546834 MIRROR_RESYNC 12 mirrors 000991 1787854182.037040 mapread 0x02c90f thru 0x02e4bd (0x1baf bytes) 000994 1787854182.059578 write 0x0332bd thru 0x034685 (0x13c9 bytes) 000996 1787854182.191007 MIRROR_RESYNC 12 mirrors 000998 1787854189.343244 write 0x03974f thru 0x03bda9 (0x265b bytes) All operations completed A-OK! PASS 70a (141s) == sanity-flr test 71: check mirror extend parent fid ==== 14:09:58 (1787854198) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (5s) == sanity-flr test 72: check append on FLR file ========== 14:10:02 (1787854202) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ec:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.25875 s, 8.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ec:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15173 s, 6.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ec:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ec:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 14:10:10 (1787854210) Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre2 Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 2152416 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5043424 .. done Extending file size to 5032416 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5598368 .. Extending file size to 7422784 .. Extending file size to 7778400 .. Extending file size to 8367008 .. Extending file size to 8542080 .. Extending file size to 7977856 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8950144 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8710528 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8979424 .. done Waiting 58748 58749 58751 58754 58755 192.168.201.114@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg114-client.virtnet /mnt/lustre2 (opts:) 192.168.201.114@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg114-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 14:11:32 (1787854292) Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre2 Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3171392 .. Extending file size to 7429344 .. Extending file size to 7835424 .. Extending file size to 7861984 .. Extending file size to 7723072 .. Extending file size to 7962400 .. Extending file size to 7992736 .. failed Extending file size to 8342112 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8345120 .. Extending file size to 8901376 .. Extending file size to 8481056 .. Extending file size to 8741728 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9015232 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8905792 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 60348 60350 60351 60353 60354 failed 192.168.201.114@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg114-client.virtnet /mnt/lustre2 (opts:) 192.168.201.114@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg114-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 14:12:52 (1787854372) Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.119718 s, 17.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.46155 s, 4.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0xaca4beff (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.201.114@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg114-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 14:13:11 (1787854391) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.108265 s, 19.4 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 14:13:18 (1787854398) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e8:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2ed:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 14:13:26 (1787854406) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.178492 s, 11.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ee:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ef:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 14:13:34 (1787854414) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.292718 s, 7.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 14:13:41 (1787854421) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.205818 s, 10.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f3:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 14:14:15 (1787854455) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg114-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105542 s, 9.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 14:15:11 (1787854511) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg114-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.145376 s, 14.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (60s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 14:16:11 (1787854571) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg114-server mds-ost sync done. Creating new pool test_204f oleg114-server: Pool lustre.test_204f created Adding targets to pool oleg114-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg114-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.251973 s, 8.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg114-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg114-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg114-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 14:17:31 (1787854651) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 14:17:40 (1787854660) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 14:17:47 (1787854667) oleg114-server: Pool lustre.test_206 created Adding targets to pool oleg114-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg114-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg114-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg114-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg114-server: Pool lustre.test_206 destroyed PASS 206 (23s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 14:18:11 (1787854691) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108696 s, 9.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303576 s, 34.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xe3770cc8 Mirror 3: 0xe3770cc8 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 14:18:21 (1787854701) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 14:18:24 (1787854704) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 14:18:28 (1787854708) Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.155164 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x18b0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1660000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1660000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1660000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1990000 Read this (4 bytes): BBBB 192.168.201.114@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg114-client.virtnet /mnt/lustre2 (opts:) PASS 209a (35s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 14:19:03 (1787854743) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0423339 s, 194 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 14:19:10 (1787854750) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603086 s, 17.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20d lmm_fid: [0x200000404:0x20d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20d lmm_fid: [0x200000404:0x20d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x301:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834445 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 14:19:12.000000000 -0400 Modify: 2026-08-27 14:19:12.000000000 -0400 Change: 2026-08-27 14:19:13.000000000 -0400 Birth: 2026-08-27 14:19:12.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 14:19:18 (1787854758) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 14:19:21 (1787854761) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.498153 s, 82.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0468967 s, 87.3 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 14:19 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20f:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 14:19 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 14:19:28 (1787854768) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (6s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 14:19:34 (1787854774) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1838 sec =========== 14:19:42 (1787854782) === sanity-flr: start cleanup 14:19:43 (1787854783) === === sanity-flr: finish cleanup 14:19:49 (1787854789) ===