-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 09:36:28 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg105-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 09:36:49 (1787837809) === oleg105-client.virtnet: executing check_config_client /mnt/lustre oleg105-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg105-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff964a85646000.idle_timeout=debug osc.lustre-OST0001-osc-ffff964a85646000.idle_timeout=debug disable quota as required oleg105-server: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 09:37:21 (1787837841) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7636] == sanity-flr test 0a: lfs mirror create with -N option == 09:37:24 (1787837844) PASS 0a (20s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:37:44 (1787837864) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:37:49 (1787837869) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:37:53 (1787837873) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:38:21 (1787837901) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:38:24 (1787837904) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 09:38:28 (1787837908) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff964a85646000.stats=clear osc.lustre-OST0001-osc-ffff964a85646000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:38:35 (1787837915) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:38:47 (1787837927) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:38:55 (1787837935) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 09:38:58 (1787837938) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:39:06 (1787837946) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 09:39:15 (1787837955) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:39:23 (1787837963) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:39:26 (1787837966) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:39:34 (1787837974) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:39:45 (1787837985) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:39:53 (1787837993) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 1.60226 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15409 s, 6.8 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:40:05 (1787838005) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.604707 s, 27.7 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 09:40:15 (1787838015) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:40:26 (1787838026) Stopping /mnt/lustre-ost1 (opts:) on oleg105-server oleg105-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff964a85646000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff964a85646000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg105-server: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg105-server: oleg105-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg105-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg105-server oleg105-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff964a85646000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff964a85646000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg105-server: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg105-server: oleg105-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg105-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (78s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:41:45 (1787838105) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.515245 s, 16.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg105-server oleg105-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff964a85646000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff964a85646000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg105-server: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg105-server: oleg105-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg105-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (45s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:42:31 (1787838151) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg105-server oleg105-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff964a85646000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff964a85646000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg105-server: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg105-server: oleg105-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg105-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg105-server oleg105-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff964a85646000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff964a85646000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg105-server: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg105-server: oleg105-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg105-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (83s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:43:54 (1787838234) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.96554 s, 13.2 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 09:44 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.088s user 0m0.000s sys 0m0.178s Stopping /mnt/lustre-ost1 (opts:) on oleg105-server oleg105-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff964a85646000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff964a85646000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.200s user 0m0.000s sys 0m0.177s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg105-server: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg105-server: oleg105-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg105-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg105-server oleg105-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff964a85646000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff964a85646000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.058s user 0m0.000s sys 0m0.179s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg105-server: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg105-server: oleg105-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg105-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg105-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff964a85646000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff964a85646000.ost_server_uuid in FULL state after 0 sec PASS 33b (94s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:45:29 (1787838329) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:45:32 (1787838332) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:45:35 (1787838335) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 09:45:39 (1787838339) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 09:45:47 (1787838347) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.781157 s, 16.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:46:01 (1787838361) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505964 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287816 s, 36.4 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:46:09 (1787838369) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:46:18 (1787838378) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000861937 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00186557 s, 549 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000782155 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00132444 s, 773 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 09:46:30 (1787838390) create test files with size 15739391 .. 61a61af693cf2bef6613d67a2ae20485 - dca2556c01c089012a046f1afbb3b0ef - b49ce51a111fe1f3d259352a147e9c4d - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg105-client.virtnet /mnt/lustre (opts:) Starting client: oleg105-client.virtnet: -o user_xattr,flock 192.168.201.105@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 09:46:53 (1787838413) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:47:06 (1787838426) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:47:13 (1787838433) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141648 s, 7.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321919 s, 32.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 09:47:23 (1787838443) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.234051 s, 9.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.473679 s, 8.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.298541 s, 7.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 09:47:40 (1787838460) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 09:47:44 (1787838464) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:47:48 (1787838468) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377368 s, 27.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 43c: read prefer uncompressed mirror == 09:47:57 (1787838477) llite.lustre-ffff964a82aaf800.enable_compression=1 fail_loc=0x1425 fail_val=-1 PASS 43c (7s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 09:48:04 (1787838484) llite.lustre-ffff964a82aaf800.enable_compression=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275383 s, 38.1 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451522 s, 23.2 MB/s **verify components PASS 43d (9s) == sanity-flr test 44a: lfs mirror split check =========== 09:48:14 (1787838494) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:48:17 (1787838497) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe2:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc8:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc8:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:48:28 (1787838508) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.251373 s, 41.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x1f:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg105-client.virtnet /mnt/lustre (opts:) Starting client: oleg105-client.virtnet: -o user_xattr,flock 192.168.201.105@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (15s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:48:43 (1787838523) llite.lustre-ffff964aa10a7000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.265448 s, 39.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg105-client.virtnet /mnt/lustre (opts:) Starting client: oleg105-client.virtnet: -o user_xattr,flock 192.168.201.105@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff964a85c3c000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:48:58 (1787838538) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (41s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:49:39 (1787838579) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (42s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:50:21 (1787838621) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 09:50:25 (1787838625) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.252354 s, 12.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.172923 s, 18.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xb13c460 (nosync,stale) Mirror 2: 0xb13c460 (nosync,stale) ** verify mirror 2 doesn't change original checksum: b67b745f469ee1cc2f0ac8fed7095cbf - mirror 1 checksum: a259a95fec0a994dccc3b2f0c1cf00af - mirror 2 checksum: b67b745f469ee1cc2f0ac8fed7095cbf - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T09:50:28-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T09:50:28-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: b67b745f469ee1cc2f0ac8fed7095cbf - mirror 1 checksum: a259a95fec0a994dccc3b2f0c1cf00af - mirror 2 checksum: a259a95fec0a994dccc3b2f0c1cf00af - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T09:50:28-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T09:50:28-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:50:37 (1787838637) Failing mds1 on oleg105-server Stopping /mnt/lustre-mds1 (opts:) on oleg105-server 09:50:44 (1787838644) shut down facet: mds1 facet_host: oleg105-server facet_failover_host: oleg105-server Failover mds1 to oleg105-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg105-server: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Started lustre-MDT0000 09:51:18 (1787838678) targets are mounted 09:51:18 (1787838678) facet_failover done oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309833 s, 33.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } PASS 50A (57s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:51:34 (1787838694) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:51:37 (1787838697) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:51:41 (1787838701) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.012001 s, 1.4 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:51:48 (1787838708) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:51:51 (1787838711) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0076366 s, 536 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0849498 s, 48.2 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:51:59 (1787838719) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:52:20 (1787838740) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:52:47 (1787838767) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 09:53:10 (1787838790) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 09:53:33 (1787838813) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 09:53:41 (1787838821) mirror create pid 55577 mirror split pid 55580 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 55577 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 55580 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 09:54:48 (1787838888) Seed set to 8890 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x249b7 000001 1787838890.778223 trunc from 00000000 to 0x0249b6 (0x249b7 bytes) truncating to largest ever: 0x2d440 000002 1787838890.794887 trunc from 0x0249b7 to 0x02d43f (0x8a89 bytes) 000005 1787838890.810559 read 0x01c5da thru 0x029eae (0xd8d5 bytes) 000006 1787838890.840541 write 0x0206dc thru 0x023b2e (0x3453 bytes) 000009 1787838890.847906 trunc from 0x02d440 to 0x02a1cb (0x3274 bytes) 000010 1787838890.880955 write 0x0144f8 thru 0x01fff6 (0xbaff bytes) 000012 1787838890.904458 trunc from 0x02a1cc to 0x00589b (0x24930 bytes) truncating to largest ever: 0x31ccb 000016 1787838890.922925 trunc from 0x00589c to 0x031cca (0x2c42f bytes) 000017 1787838890.945769 read 0x022457 thru 0x031cca (0xf874 bytes) 000019 1787838890.970377 MIRROR_EXTEND to 2 mirrors 000021 1787838891.644814 MIRROR_EXTEND to 3 mirrors 000022 1787838892.502986 mapread 0x022035 thru 0x0235c2 (0x158e bytes) truncating to largest ever: 0x3bb4d 000024 1787838892.515063 trunc from 0x031ccb to 0x03bb4c (0x9e82 bytes) 000025 1787838892.585416 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3bb4d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb4d): Mirror 2: 0xbaed6b15 (stale) Mirror 3: 0xbaed6b15 (stale) CRC-32 checksum value for chunk [0, 0x3bb4d): Mirror 1: 0xbaed6b15 Mirror 4: 0xbaed6b15 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb4d: skipped 000026 1787838893.491679 trunc from 0x03bb4d to 0x00511a (0x36a32 bytes) 000030 1787838893.565952 mapwrite 0x028969 thru 0x029e5a (0x14f2 bytes) 000031 1787838893.647060 mapwrite 0x00d8ff thru 0x00e6d3 (0xdd5 bytes) 000032 1787838893.699402 MIRROR_RESYNC 4 mirrors 000033 1787838895.320022 mapread 0x000597 thru 0x00a6f8 (0xa162 bytes) 000034 1787838895.350523 MIRROR_EXTEND to 5 mirrors 000035 1787838896.158846 mapwrite 0x0219fe thru 0x026c6a (0x526d bytes) 000036 1787838896.371614 write 0x00b170 thru 0x00cbf3 (0x1a84 bytes) 000039 1787838896.384403 trunc from 0x029e5b to 0x004051 (0x25e09 bytes) 000041 1787838896.417175 trunc from 0x004052 to 0x029379 (0x25328 bytes) 000042 1787838896.435141 mapwrite 0x018454 thru 0x021b4e (0x96fb bytes) 000043 1787838896.516395 mapwrite 0x03ffcf thru 0x03ffff (0x31 bytes) 000044 1787838896.607491 mapread 0x028e99 thru 0x038953 (0xfabb bytes) 000045 1787838896.632365 read 0x033ac7 thru 0x036648 (0x2b82 bytes) 000046 1787838896.632979 mapread 0x011acd thru 0x01fa96 (0xdfca bytes) 000047 1787838896.635038 mapread 0x00e036 thru 0x01dece (0xfe99 bytes) 000049 1787838896.663721 trunc from 0x040000 to 0x0256bb (0x1a944 bytes) 000050 1787838896.705091 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x256bc): Mirror 2: 0xe8f2f8a (stale) Mirror 3: 0xe8f2f8a (stale) Mirror 4: 0xe8f2f8a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x256bc): Mirror 2: 0xe8f2f8a (stale) Mirror 3: 0xe8f2f8a (stale) Mirror 4: 0xe8f2f8a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x256bc: skipped 000051 1787838897.081309 trunc from 0x0256bc to 0x022c9c (0x2a1f bytes) 000052 1787838897.107312 write 0x006ef0 thru 0x00bf76 (0x5087 bytes) 000053 1787838897.139702 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x22c9d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22c9d): Mirror 2: 0xc0891ab8 (stale) Mirror 3: 0xc0891ab8 (stale) Mirror 4: 0xc0891ab8 (stale) CRC-32 checksum value for chunk [0, 0x22c9d): Mirror 1: 0x71211665 Mirror 5: 0x71211665 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22c9d: skipped 000057 1787838898.237234 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22c9d): Mirror 2: 0xc0891ab8 (stale) Mirror 4: 0xc0891ab8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22c9d): Mirror 2: 0xc0891ab8 (stale) Mirror 4: 0xc0891ab8 (stale) CRC-32 checksum value for chunk [0, 0x22c9d): Mirror 1: 0x71211665 Mirror 5: 0x71211665 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22c9d: skipped 000058 1787838898.714333 mapwrite 0x01d24b thru 0x01faa2 (0x2858 bytes) 000061 1787838898.947261 mapread 0x009335 thru 0x0093e7 (0xb3 bytes) 000063 1787838898.962870 read 0x00baf5 thru 0x00f431 (0x393d bytes) 000064 1787838898.964034 mapwrite 0x032711 thru 0x03e61c (0xbf0c bytes) 000065 1787838899.100777 mapread 0x02a892 thru 0x03a81f (0xff8e bytes) 000067 1787838899.112887 mapwrite 0x0320e4 thru 0x03feb4 (0xddd1 bytes) 000069 1787838899.207330 mapwrite 0x00f35e thru 0x01ca1d (0xd6c0 bytes) 000070 1787838899.319802 trunc from 0x03feb5 to 0x02b685 (0x1482f bytes) 000073 1787838899.351944 write 0x02e442 thru 0x0300ca (0x1c89 bytes) 000076 1787838899.355094 write 0x02b675 thru 0x03a717 (0xf0a3 bytes) 000077 1787838899.357300 mapwrite 0x0138ef thru 0x0188e6 (0x4ff8 bytes) 000078 1787838899.438910 read 0x0185c5 thru 0x023840 (0xb27c bytes) 000079 1787838899.440890 write 0x032eb4 thru 0x03ffff (0xd14c bytes) 000080 1787838899.449507 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xc50a4204 (stale) Mirror 5: 0xb4541012 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xc50a4204 (stale) Mirror 5: 0xb4541012 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000081 1787838899.812803 mapread 0x01af96 thru 0x026c9a (0xbd05 bytes) 000082 1787838899.852839 trunc from 0x040000 to 0x012a88 (0x2d577 bytes) 000085 1787838899.868934 read 0x00bc4a thru 0x012748 (0x6aff bytes) 000086 1787838899.880311 write 0x00f64d thru 0x017dc2 (0x8776 bytes) 000087 1787838899.881514 write 0x00cbac thru 0x0119a0 (0x4df5 bytes) 000088 1787838899.882466 trunc from 0x017dc3 to 0x0364a7 (0x1e6e5 bytes) 000090 1787838899.917990 read 0x0259b8 thru 0x02bd3d (0x6386 bytes) 000091 1787838899.934835 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x364a8): Mirror 5: 0x6124cd47 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x364a8): Mirror 5: 0x6124cd47 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x364a8: skipped 000093 1787838900.225169 write 0x00670b thru 0x00c52d (0x5e23 bytes) 000094 1787838900.241568 read 0x02d566 thru 0x0364a7 (0x8f42 bytes) 000095 1787838900.272072 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x364a8): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x364a8): Mirror 5: 0x6124cd47 (stale) CRC-32 checksum value for chunk [0, 0x364a8): Mirror 1: 0x2c64c882 Mirror 6: 0x2c64c882 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x364a8: skipped 000096 1787838901.845529 read 0x0051ef thru 0x010d6b (0xbb7d bytes) 000098 1787838901.854619 read 0x02712e thru 0x0364a7 (0xf37a bytes) 000099 1787838901.867070 read 0x004afe thru 0x006a17 (0x1f1a bytes) 000100 1787838901.867565 read 0x0323ba thru 0x0364a7 (0x40ee bytes) 000101 1787838901.868470 mapwrite 0x03b36d thru 0x03ffff (0x4c93 bytes) 000102 1787838902.083746 write 0x03f950 thru 0x03ffff (0x6b0 bytes) 000103 1787838902.086150 mapwrite 0x01b4cd thru 0x028504 (0xd038 bytes) 000105 1787838902.138817 mapread 0x009b5c thru 0x00cf97 (0x343c bytes) 000106 1787838902.140455 mapwrite 0x0013a6 thru 0x00d1de (0xbe39 bytes) 000113 1787838902.193321 mapwrite 0x03810f thru 0x039dd3 (0x1cc5 bytes) 000119 1787838902.240109 write 0x00cebd thru 0x00e39a (0x14de bytes) 000120 1787838902.245622 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xb4541012 (stale) Mirror 6: 0x3a13680d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf96a9c8b Mirror 7: 0xf96a9c8b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000121 1787838903.122717 MIRROR_RESYNC 4 mirrors truncating to largest ever: 0x3bc8f 000123 1787838904.174344 trunc from 0x040000 to 0x03bc8e (0x4371 bytes) 000124 1787838904.325624 mapwrite 0x0001fb thru 0x005e7a (0x5c80 bytes) 000127 1787838904.452306 mapwrite 0x01cbf7 thru 0x02203e (0x5448 bytes) 000132 1787838904.526958 write 0x00d715 thru 0x018f94 (0xb880 bytes) 000135 1787838904.531015 mapwrite 0x00a3d3 thru 0x01a186 (0xfdb4 bytes) 000136 1787838904.626531 mapread 0x0175d6 thru 0x019826 (0x2251 bytes) 000137 1787838904.631629 trunc from 0x03bc8f to 0x02eb75 (0xd119 bytes) 000138 1787838904.701702 write 0x038fa2 thru 0x03ffff (0x705e bytes) 000141 1787838904.716773 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xf96a9c8b (stale) Mirror 6: 0xf96a9c8b (stale) Mirror 7: 0xf96a9c8b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x520adf1c Mirror 8: 0x520adf1c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000143 1787838905.924856 write 0x0329ed thru 0x03f64f (0xcc63 bytes) 000144 1787838906.035263 mapwrite 0x018328 thru 0x018b2d (0x806 bytes) 000145 1787838906.110315 trunc from 0x040000 to 0x00ea82 (0x3157d bytes) 000148 1787838906.143596 write 0x026061 thru 0x02caa9 (0x6a49 bytes) 000149 1787838906.150095 mapread 0x01eba0 thru 0x02caa9 (0xdf0a bytes) 000150 1787838906.170653 write 0x03d45a thru 0x03ffff (0x2ba6 bytes) 000154 1787838906.173007 write 0x001114 thru 0x0101dc (0xf0c9 bytes) 000159 1787838906.198345 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xf96a9c8b (stale) Mirror 6: 0xf96a9c8b (stale) Mirror 7: 0xf96a9c8b (stale) Mirror 8: 0x520adf1c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x533fae0a Mirror 9: 0x533fae0a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000160 1787838907.281021 trunc from 0x040000 to 0x006e6f (0x39190 bytes) 000162 1787838907.418520 read 0x001d23 thru 0x006e6f (0x514d bytes) 000165 1787838907.442468 mapread 0x002f7b thru 0x006e6f (0x3ef5 bytes) 000166 1787838907.445999 mapwrite 0x01d6db thru 0x02712c (0x9a52 bytes) 000168 1787838907.615369 mapread 0x00e270 thru 0x0125af (0x4340 bytes) 000170 1787838907.625877 write 0x02005e thru 0x02d170 (0xd113 bytes) 000171 1787838907.629475 mapwrite 0x00edb5 thru 0x0111c7 (0x2413 bytes) 000172 1787838907.677895 write 0x033266 thru 0x03ffff (0xcd9a bytes) 000173 1787838907.679360 write 0x00ee1d thru 0x0105ed (0x17d1 bytes) 000177 1787838907.687556 MIRROR_RESYNC 6 mirrors 000178 1787838909.693856 trunc from 0x040000 to 0x034ddb (0xb224 bytes) 000179 1787838909.798042 read 0x024c70 thru 0x02f5af (0xa940 bytes) 000180 1787838909.804323 write 0x011cee thru 0x018db4 (0x70c7 bytes) 000181 1787838909.812163 read 0x00029e thru 0x00fe5b (0xfbbe bytes) 000182 1787838909.828435 MIRROR_RESYNC 6 mirrors 000183 1787838911.734181 read 0x0282a6 thru 0x029751 (0x14ac bytes) 000184 1787838911.740579 mapread 0x0117db thru 0x015f9d (0x47c3 bytes) 000185 1787838911.759402 trunc from 0x034ddc to 0x02f13a (0x5ca1 bytes) 000187 1787838911.887923 mapwrite 0x026626 thru 0x02b509 (0x4ee4 bytes) 000188 1787838912.050588 read 0x01e50e thru 0x0230ad (0x4ba0 bytes) 000193 1787838912.082111 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2f13b): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f13b): Mirror 5: 0xe1b46cf8 (stale) Mirror 6: 0xe1b46cf8 (stale) Mirror 7: 0xe1b46cf8 (stale) Mirror 8: 0xe1b46cf8 (stale) Mirror 9: 0xe1b46cf8 (stale) CRC-32 checksum value for chunk [0, 0x2f13b): Mirror 1: 0x3ba48923 Mirror 10: 0x3ba48923 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f13b: skipped 000194 1787838913.223636 read 0x018c02 thru 0x020f2e (0x832d bytes) 000195 1787838913.234015 trunc from 0x02f13b to 0x025866 (0x98d4 bytes) 000196 1787838913.318541 write 0x020d4c thru 0x027f73 (0x7228 bytes) 000197 1787838913.324070 write 0x010e47 thru 0x01e367 (0xd521 bytes) 000198 1787838913.334683 trunc from 0x027f74 to 0x00e7c5 (0x197ae bytes) 000200 1787838913.358055 write 0x00a7f1 thru 0x00e0e3 (0x38f3 bytes) 000201 1787838913.367129 mapwrite 0x002e56 thru 0x00eaed (0xbc98 bytes) 000202 1787838913.476643 trunc from 0x00eaee to 0x00d408 (0x16e5 bytes) 000203 1787838913.503325 trunc from 0x00d409 to 0x010c42 (0x383a bytes) 000204 1787838913.543343 MIRROR_RESYNC 7 mirrors 000206 1787838914.728237 write 0x021d7f thru 0x0240ef (0x2371 bytes) 000207 1787838914.788337 write 0x030b90 thru 0x037391 (0x6802 bytes) 000208 1787838914.789616 trunc from 0x037392 to 0x00d6d8 (0x29cb9 bytes) 000209 1787838914.809578 mapread 0x008843 thru 0x00d6d8 (0x4e96 bytes) 000210 1787838914.824641 MIRROR_RESYNC 7 mirrors 000211 1787838916.104571 trunc from 0x00d6d9 to 0x01fa7b (0x123a3 bytes) 000212 1787838916.242861 MIRROR_RESYNC 7 mirrors 000214 1787838918.006657 write 0x033be8 thru 0x03ffff (0xc418 bytes) 000217 1787838918.153992 mapread 0x02b126 thru 0x032e56 (0x7d31 bytes) 000220 1787838918.177403 write 0x031607 thru 0x03ef14 (0xd90e bytes) 000221 1787838918.180548 trunc from 0x040000 to 0x02af3d (0x150c2 bytes) 000223 1787838918.209457 mapread 0x01354b thru 0x013b73 (0x629 bytes) 000224 1787838918.222513 trunc from 0x02af3e to 0x027f21 (0x301c bytes) 000225 1787838918.248870 mapwrite 0x0053fa thru 0x00d814 (0x841b bytes) 000229 1787838918.307427 trunc from 0x027f22 to 0x012275 (0x15cac bytes) 000231 1787838918.325865 mapwrite 0x026fc4 thru 0x0338fa (0xc937 bytes) 000232 1787838918.431675 read 0x02609a thru 0x03343c (0xd3a3 bytes) 000233 1787838918.434003 trunc from 0x0338fb to 0x02fb29 (0x3dd1 bytes) 000235 1787838918.476883 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2fb2a): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fb2a): Mirror 5: 0x40faf7dc (stale) Mirror 6: 0x40faf7dc (stale) Mirror 7: 0x40faf7dc (stale) Mirror 8: 0x40faf7dc (stale) Mirror 9: 0x40faf7dc (stale) Mirror 10: 0x40faf7dc (stale) CRC-32 checksum value for chunk [0, 0x2fb2a): Mirror 1: 0x727709db Mirror 11: 0x727709db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fb2a: skipped 000236 1787838919.447765 mapread 0x00ac98 thru 0x016b59 (0xbec2 bytes) 000238 1787838919.477055 write 0x029dd9 thru 0x03203e (0x8266 bytes) 000239 1787838919.584802 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3203f): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3203f): Mirror 5: 0xd25643ae (stale) Mirror 6: 0xd25643ae (stale) Mirror 7: 0xd25643ae (stale) Mirror 8: 0xd25643ae (stale) Mirror 9: 0xd25643ae (stale) Mirror 10: 0xd25643ae (stale) Mirror 11: 0xb1e045bc (stale) CRC-32 checksum value for chunk [0, 0x3203f): Mirror 1: 0x69ff3e52 Mirror 12: 0x69ff3e52 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3203f: skipped 000240 1787838920.863894 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3203f): Mirror 5: 0xd25643ae (stale) Mirror 6: 0xd25643ae (stale) Mirror 7: 0xd25643ae (stale) Mirror 8: 0xd25643ae (stale) Mirror 9: 0xd25643ae (stale) Mirror 11: 0xb1e045bc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3203f): Mirror 5: 0xd25643ae (stale) Mirror 6: 0xd25643ae (stale) Mirror 7: 0xd25643ae (stale) Mirror 8: 0xd25643ae (stale) Mirror 9: 0xd25643ae (stale) Mirror 11: 0xb1e045bc (stale) CRC-32 checksum value for chunk [0, 0x3203f): Mirror 1: 0x69ff3e52 Mirror 12: 0x69ff3e52 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3203f: skipped 000241 1787838921.746008 trunc from 0x03203f to 0x0005ae (0x31a90 bytes) 000242 1787838921.858632 read 0x0004d7 thru 0x0005ae (0xd8 bytes) 000243 1787838921.866459 write 0x00fde7 thru 0x01ef0e (0xf128 bytes) 000245 1787838921.869775 read 0x005d7e thru 0x00a106 (0x4389 bytes) 000247 1787838921.879873 write 0x039953 thru 0x03f7aa (0x5e58 bytes) 000248 1787838921.881541 read 0x03aad4 thru 0x03f7aa (0x4cd7 bytes) 000249 1787838921.882543 write 0x03c0dc thru 0x03ffff (0x3f24 bytes) 000250 1787838921.883713 read 0x019420 thru 0x025ae8 (0xc6c9 bytes) 000252 1787838921.892766 write 0x03dd3a thru 0x03ffff (0x22c6 bytes) 000253 1787838921.893848 read 0x03fb41 thru 0x03ffff (0x4bf bytes) 000254 1787838921.895414 read 0x030044 thru 0x031e76 (0x1e33 bytes) 000256 1787838921.896375 trunc from 0x040000 to 0x036527 (0x9ad8 bytes) 000257 1787838921.913434 write 0x01c722 thru 0x021049 (0x4928 bytes) 000258 1787838921.918541 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x36528): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36528): Mirror 5: 0xb324657c (stale) Mirror 6: 0xb324657c (stale) Mirror 7: 0xb324657c (stale) Mirror 8: 0xb324657c (stale) Mirror 9: 0xb324657c (stale) Mirror 11: 0x5ea3e0 (stale) Mirror 12: 0x2142a1a7 (stale) CRC-32 checksum value for chunk [0, 0x36528): Mirror 1: 0xbd53803 Mirror 13: 0xbd53803 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36528: skipped 000259 1787838923.139401 mapwrite 0x02026c thru 0x02de56 (0xdbeb bytes) 000261 1787838923.328323 trunc from 0x036528 to 0x003d02 (0x32825 bytes) 000263 1787838923.358308 mapwrite 0x01c229 thru 0x026133 (0x9f0b bytes) 000264 1787838923.437133 trunc from 0x026134 to 0x00e648 (0x17aeb bytes) 000266 1787838923.458692 read 0x00cd95 thru 0x00e648 (0x18b4 bytes) 000267 1787838923.464856 trunc from 0x00e649 to 0x0113a7 (0x2d5f bytes) 000271 1787838923.482496 mapread 0x00dc00 thru 0x0113a7 (0x37a8 bytes) 000272 1787838923.490962 trunc from 0x0113a8 to 0x01b32c (0x9f85 bytes) 000273 1787838923.508652 read 0x01ac4a thru 0x01b32c (0x6e3 bytes) 000274 1787838923.517733 write 0x015d35 thru 0x0242d9 (0xe5a5 bytes) 000277 1787838923.525767 MIRROR_RESYNC 9 mirrors 000278 1787838925.920823 trunc from 0x0242da to 0x008d77 (0x1b562 bytes) 000280 1787838926.012168 mapwrite 0x02ef8b thru 0x03002a (0x10a0 bytes) 000282 1787838926.105122 trunc from 0x03002b to 0x02b391 (0x4c99 bytes) 000286 1787838926.123651 mapwrite 0x012f91 thru 0x01896e (0x59de bytes) 000287 1787838926.169227 write 0x03f2e3 thru 0x03ffff (0xd1d bytes) 000288 1787838926.170345 mapwrite 0x02d041 thru 0x02fa42 (0x2a02 bytes) 000289 1787838926.255295 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x94a375af (stale) Mirror 6: 0x94a375af (stale) Mirror 7: 0x94a375af (stale) Mirror 8: 0x94a375af (stale) Mirror 9: 0x94a375af (stale) Mirror 11: 0x94a375af (stale) Mirror 12: 0x94a375af (stale) Mirror 13: 0x94a375af (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4b6f3198 Mirror 14: 0x4b6f3198 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000290 1787838927.527899 mapwrite 0x007648 thru 0x007929 (0x2e2 bytes) 000291 1787838927.878123 read 0x006fa8 thru 0x00a38d (0x33e6 bytes) 000292 1787838927.896886 write 0x0296df thru 0x02a296 (0xbb8 bytes) 000293 1787838927.926814 mapwrite 0x033b8b thru 0x039ef9 (0x636f bytes) 000297 1787838928.065819 write 0x039497 thru 0x03ffff (0x6b69 bytes) 000299 1787838928.067401 write 0x00622e thru 0x01325f (0xd032 bytes) 000301 1787838928.078759 write 0x000147 thru 0x00e0a3 (0xdf5d bytes) 000303 1787838928.106159 read 0x0283fc thru 0x02deb8 (0x5abd bytes) 000304 1787838928.124330 read 0x03b57b thru 0x03e474 (0x2efa bytes) 000307 1787838928.126912 read 0x03e864 thru 0x03ffff (0x179c bytes) 000308 1787838928.133065 mapwrite 0x01865d thru 0x019c65 (0x1609 bytes) 000310 1787838928.199171 mapwrite 0x00a8b7 thru 0x015208 (0xa952 bytes) 000311 1787838928.294720 read 0x036df2 thru 0x03ffff (0x920e bytes) 000313 1787838928.305540 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x94a375af (stale) Mirror 7: 0x94a375af (stale) Mirror 8: 0x94a375af (stale) Mirror 9: 0x94a375af (stale) Mirror 11: 0x94a375af (stale) Mirror 12: 0x94a375af (stale) Mirror 13: 0x94a375af (stale) Mirror 14: 0x4b6f3198 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x94a375af (stale) Mirror 7: 0x94a375af (stale) Mirror 8: 0x94a375af (stale) Mirror 9: 0x94a375af (stale) Mirror 11: 0x94a375af (stale) Mirror 12: 0x94a375af (stale) Mirror 13: 0x94a375af (stale) Mirror 14: 0x4b6f3198 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000314 1787838929.009077 write 0x039a63 thru 0x03b46c (0x1a0a bytes) 000315 1787838929.019602 read 0x0087c6 thru 0x009d73 (0x15ae bytes) 000316 1787838929.025614 write 0x029548 thru 0x02ee17 (0x58d0 bytes) 000317 1787838929.037076 write 0x028b8a thru 0x032fce (0xa445 bytes) 000318 1787838929.051818 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x94a375af (stale) Mirror 7: 0x94a375af (stale) Mirror 8: 0x94a375af (stale) Mirror 9: 0x94a375af (stale) Mirror 11: 0x94a375af (stale) Mirror 12: 0x94a375af (stale) Mirror 13: 0x94a375af (stale) Mirror 14: 0x4b6f3198 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3daa5d0e Mirror 15: 0x3daa5d0e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000321 1787838930.135653 read 0x021f89 thru 0x02be6d (0x9ee5 bytes) 000322 1787838930.146939 read 0x010ac5 thru 0x011026 (0x562 bytes) 000323 1787838930.160171 write 0x017290 thru 0x024d77 (0xdae8 bytes) 000324 1787838930.309570 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x94a375af (stale) Mirror 7: 0x94a375af (stale) Mirror 8: 0x94a375af (stale) Mirror 9: 0x94a375af (stale) Mirror 11: 0x94a375af (stale) Mirror 12: 0x94a375af (stale) Mirror 13: 0x94a375af (stale) Mirror 14: 0x4b6f3198 (stale) Mirror 15: 0x3daa5d0e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc317f37e Mirror 16: 0xc317f37e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000325 1787838931.496501 mapwrite 0x02e9ba thru 0x038748 (0x9d8f bytes) 000327 1787838931.807367 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x94a375af (stale) Mirror 7: 0x94a375af (stale) Mirror 8: 0x94a375af (stale) Mirror 11: 0x94a375af (stale) Mirror 12: 0x94a375af (stale) Mirror 13: 0x94a375af (stale) Mirror 14: 0x4b6f3198 (stale) Mirror 15: 0x3daa5d0e (stale) Mirror 16: 0xc317f37e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x94a375af (stale) Mirror 7: 0x94a375af (stale) Mirror 8: 0x94a375af (stale) Mirror 11: 0x94a375af (stale) Mirror 12: 0x94a375af (stale) Mirror 13: 0x94a375af (stale) Mirror 14: 0x4b6f3198 (stale) Mirror 15: 0x3daa5d0e (stale) Mirror 16: 0xc317f37e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000328 1787838932.466458 write 0x039910 thru 0x03d132 (0x3823 bytes) 000333 1787838932.483792 mapwrite 0x009fcd thru 0x018803 (0xe837 bytes) 000334 1787838932.585718 write 0x0259a1 thru 0x027cd1 (0x2331 bytes) 000337 1787838932.645210 mapwrite 0x011f3d thru 0x020d7f (0xee43 bytes) 000340 1787838932.776720 read 0x02b4d6 thru 0x037cf5 (0xc820 bytes) 000341 1787838932.798769 write 0x00737e thru 0x00c2bb (0x4f3e bytes) 000342 1787838932.810953 MIRROR_RESYNC 10 mirrors 000343 1787838934.190929 trunc from 0x040000 to 0x00aa35 (0x355ca bytes) 000344 1787838934.279853 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0xaa36): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xaa36): Mirror 5: 0xbaf8dad5 (stale) Mirror 7: 0xbaf8dad5 (stale) Mirror 8: 0xbaf8dad5 (stale) Mirror 11: 0xbaf8dad5 (stale) Mirror 12: 0xbaf8dad5 (stale) Mirror 13: 0xbaf8dad5 (stale) Mirror 14: 0xbaf8dad5 (stale) Mirror 15: 0xbaf8dad5 (stale) Mirror 16: 0xbaf8dad5 (stale) CRC-32 checksum value for chunk [0, 0xaa36): Mirror 1: 0xbaf8dad5 Mirror 17: 0xbaf8dad5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xaa36: skipped 000346 1787838935.186332 mapread 0x00460d thru 0x00aa35 (0x6429 bytes) 000348 1787838935.202803 mapwrite 0x03ce40 thru 0x03f072 (0x2233 bytes) 000351 1787838935.395784 write 0x00abb3 thru 0x018cce (0xe11c bytes) 000352 1787838935.414964 write 0x01bc51 thru 0x01ed7f (0x312f bytes) 000360 1787838935.437587 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3f073): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f073): Mirror 5: 0xb1afb75 (stale) Mirror 7: 0xb1afb75 (stale) Mirror 8: 0xb1afb75 (stale) Mirror 11: 0xb1afb75 (stale) Mirror 12: 0xb1afb75 (stale) Mirror 13: 0xb1afb75 (stale) Mirror 14: 0xb1afb75 (stale) Mirror 15: 0xb1afb75 (stale) Mirror 16: 0xb1afb75 (stale) Mirror 17: 0x87c1e353 (stale) CRC-32 checksum value for chunk [0, 0x3f073): Mirror 1: 0x6b56530f Mirror 18: 0x6b56530f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f073: skipped 000362 1787838936.432077 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3f073): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f073): Mirror 5: 0xb1afb75 (stale) Mirror 7: 0xb1afb75 (stale) Mirror 8: 0xb1afb75 (stale) Mirror 11: 0xb1afb75 (stale) Mirror 12: 0xb1afb75 (stale) Mirror 13: 0xb1afb75 (stale) Mirror 14: 0xb1afb75 (stale) Mirror 15: 0xb1afb75 (stale) Mirror 16: 0xb1afb75 (stale) Mirror 17: 0x87c1e353 (stale) CRC-32 checksum value for chunk [0, 0x3f073): Mirror 1: 0x6b56530f Mirror 18: 0x6b56530f Mirror 19: 0x6b56530f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f073: skipped 000363 1787838937.850523 read 0x01bc37 thru 0x02317d (0x7547 bytes) 000364 1787838937.859077 mapread 0x02e283 thru 0x03cc45 (0xe9c3 bytes) 000365 1787838937.883993 mapwrite 0x000f97 thru 0x007a6d (0x6ad7 bytes) 000366 1787838938.055219 read 0x01c771 thru 0x0222a4 (0x5b34 bytes) 000368 1787838938.072934 MIRROR_SPLIT mirror 12 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f073): Mirror 5: 0xb1afb75 (stale) Mirror 7: 0xb1afb75 (stale) Mirror 8: 0xb1afb75 (stale) Mirror 11: 0xb1afb75 (stale) Mirror 12: 0xb1afb75 (stale) Mirror 13: 0xb1afb75 (stale) Mirror 14: 0xb1afb75 (stale) Mirror 15: 0xb1afb75 (stale) Mirror 16: 0xb1afb75 (stale) Mirror 17: 0x87c1e353 (stale) Mirror 18: 0x6b56530f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f073): Mirror 5: 0xb1afb75 (stale) Mirror 7: 0xb1afb75 (stale) Mirror 8: 0xb1afb75 (stale) Mirror 11: 0xb1afb75 (stale) Mirror 12: 0xb1afb75 (stale) Mirror 13: 0xb1afb75 (stale) Mirror 14: 0xb1afb75 (stale) Mirror 15: 0xb1afb75 (stale) Mirror 16: 0xb1afb75 (stale) Mirror 17: 0x87c1e353 (stale) Mirror 18: 0x6b56530f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f073: skipped 000369 1787838939.098840 read 0x02f7d1 thru 0x0375ec (0x7e1c bytes) 000370 1787838939.113025 mapread 0x01e70f thru 0x02b257 (0xcb49 bytes) 000371 1787838939.185695 trunc from 0x03f073 to 0x01aaa0 (0x245d2 bytes) 000372 1787838939.216144 mapwrite 0x002e4b thru 0x00b373 (0x8529 bytes) 000374 1787838939.383272 write 0x031b38 thru 0x03f32b (0xd7f4 bytes) 000375 1787838939.385110 write 0x029bb0 thru 0x02db9f (0x3ff0 bytes) 000377 1787838939.423147 write 0x01a0a8 thru 0x0268d8 (0xc831 bytes) 000378 1787838939.433216 write 0x02e066 thru 0x02effe (0xf99 bytes) 000382 1787838939.448369 MIRROR_SPLIT mirror 5 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f32c): Mirror 5: 0x4db329fb (stale) Mirror 7: 0x4db329fb (stale) Mirror 8: 0x4db329fb (stale) Mirror 11: 0x4db329fb (stale) Mirror 13: 0x4db329fb (stale) Mirror 14: 0x4db329fb (stale) Mirror 15: 0x4db329fb (stale) Mirror 16: 0x4db329fb (stale) Mirror 17: 0x5be41b9a (stale) Mirror 18: 0x295711d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f32c): Mirror 5: 0x4db329fb (stale) Mirror 7: 0x4db329fb (stale) Mirror 8: 0x4db329fb (stale) Mirror 11: 0x4db329fb (stale) Mirror 13: 0x4db329fb (stale) Mirror 14: 0x4db329fb (stale) Mirror 15: 0x4db329fb (stale) Mirror 16: 0x4db329fb (stale) Mirror 17: 0x5be41b9a (stale) Mirror 18: 0x295711d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f32c: skipped 000383 1787838940.294889 trunc from 0x03f32c to 0x002769 (0x3cbc2 bytes) 000390 1787838940.342394 MIRROR_RESYNC 11 mirrors 000394 1787838942.021097 write 0x003589 thru 0x012e22 (0xf89a bytes) 000395 1787838942.097939 mapwrite 0x013ea4 thru 0x01c734 (0x8891 bytes) 000397 1787838942.195601 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c735): Mirror 5: 0xb11b2720 (stale) Mirror 7: 0xb11b2720 (stale) Mirror 8: 0xb11b2720 (stale) Mirror 11: 0xb11b2720 (stale) Mirror 13: 0xb11b2720 (stale) Mirror 14: 0xb11b2720 (stale) Mirror 15: 0xb11b2720 (stale) Mirror 16: 0xb11b2720 (stale) Mirror 18: 0xb11b2720 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c735): Mirror 5: 0xb11b2720 (stale) Mirror 7: 0xb11b2720 (stale) Mirror 8: 0xb11b2720 (stale) Mirror 11: 0xb11b2720 (stale) Mirror 13: 0xb11b2720 (stale) Mirror 14: 0xb11b2720 (stale) Mirror 15: 0xb11b2720 (stale) Mirror 16: 0xb11b2720 (stale) Mirror 18: 0xb11b2720 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c735: skipped 000399 1787838942.733349 mapwrite 0x027c18 thru 0x033c88 (0xc071 bytes) 000400 1787838942.959052 trunc from 0x033c89 to 0x02c8f7 (0x7391 bytes) 000401 1787838942.997976 read 0x025b8c thru 0x02c8f7 (0x6d6c bytes) 000402 1787838943.004103 read 0x00b032 thru 0x010aa5 (0x5a74 bytes) 000403 1787838943.023452 mapwrite 0x02dd4f thru 0x03da31 (0xfce3 bytes) 000406 1787838943.179056 write 0x02e439 thru 0x02ec77 (0x83f bytes) 000408 1787838943.181345 write 0x00b2c6 thru 0x00f92e (0x4669 bytes) 000409 1787838943.186278 mapread 0x0115b1 thru 0x01d37b (0xbdcb bytes) 000410 1787838943.191165 trunc from 0x03da32 to 0x00cc66 (0x30dcb bytes) 000412 1787838943.218178 mapwrite 0x01e4da thru 0x027107 (0x8c2e bytes) 000415 1787838943.404039 trunc from 0x027108 to 0x026896 (0x871 bytes) 000416 1787838943.440365 read 0x01dfc6 thru 0x026896 (0x88d1 bytes) 000419 1787838943.464755 write 0x00f4b2 thru 0x014b83 (0x56d2 bytes) 000423 1787838943.478495 mapread 0x005caf thru 0x0144b5 (0xe807 bytes) 000424 1787838943.488222 mapread 0x005bf6 thru 0x0107a8 (0xabb3 bytes) 000426 1787838943.492147 trunc from 0x026897 to 0x031199 (0xa903 bytes) 000427 1787838943.535058 read 0x02202b thru 0x02b1d5 (0x91ab bytes) 000428 1787838943.573479 mapread 0x008cdc thru 0x017b09 (0xee2e bytes) 000429 1787838943.585885 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3119a): Mirror 5: 0xf7388c43 (stale) Mirror 7: 0xf7388c43 (stale) Mirror 8: 0xf7388c43 (stale) Mirror 11: 0xf7388c43 (stale) Mirror 13: 0xf7388c43 (stale) Mirror 14: 0xf7388c43 (stale) Mirror 15: 0xf7388c43 (stale) Mirror 16: 0xf7388c43 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3119a): Mirror 5: 0xf7388c43 (stale) Mirror 7: 0xf7388c43 (stale) Mirror 8: 0xf7388c43 (stale) Mirror 11: 0xf7388c43 (stale) Mirror 13: 0xf7388c43 (stale) Mirror 14: 0xf7388c43 (stale) Mirror 15: 0xf7388c43 (stale) Mirror 16: 0xf7388c43 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3119a: skipped 000431 1787838944.065800 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3119a): Mirror 5: 0xf7388c43 (stale) Mirror 8: 0xf7388c43 (stale) Mirror 11: 0xf7388c43 (stale) Mirror 13: 0xf7388c43 (stale) Mirror 14: 0xf7388c43 (stale) Mirror 15: 0xf7388c43 (stale) Mirror 16: 0xf7388c43 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3119a): Mirror 5: 0xf7388c43 (stale) Mirror 8: 0xf7388c43 (stale) Mirror 11: 0xf7388c43 (stale) Mirror 13: 0xf7388c43 (stale) Mirror 14: 0xf7388c43 (stale) Mirror 15: 0xf7388c43 (stale) Mirror 16: 0xf7388c43 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3119a: skipped 000433 1787838944.500074 trunc from 0x03119a to 0x02a8c6 (0x68d3 bytes) 000434 1787838944.519486 read 0x003842 thru 0x005bec (0x23ab bytes) 000437 1787838944.528007 mapread 0x004e31 thru 0x00f321 (0xa4f1 bytes) 000438 1787838944.545463 write 0x012cd1 thru 0x02223a (0xf56a bytes) 000439 1787838944.559983 write 0x00e2e1 thru 0x00f032 (0xd52 bytes) 000440 1787838944.560997 mapwrite 0x0004c9 thru 0x004035 (0x3b6d bytes) 000441 1787838944.614764 trunc from 0x02a8c7 to 0x00b33b (0x1f58b bytes) 000442 1787838944.634479 trunc from 0x00b33c to 0x018d3b (0xda00 bytes) 000447 1787838944.651321 mapread 0x0124bf thru 0x018d3b (0x687d bytes) 000448 1787838944.660538 write 0x022844 thru 0x02924d (0x6a0a bytes) 000453 1787838944.666795 read 0x01fd92 thru 0x02924d (0x94bc bytes) 000454 1787838944.674544 trunc from 0x02924e to 0x0361e0 (0xcf93 bytes) 000455 1787838944.689410 mapwrite 0x01048e thru 0x01a744 (0xa2b7 bytes) 000456 1787838944.746362 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x361e1): Mirror 8: 0x5ee7bfc0 (stale) Mirror 11: 0x5ee7bfc0 (stale) Mirror 13: 0x5ee7bfc0 (stale) Mirror 14: 0x5ee7bfc0 (stale) Mirror 15: 0x5ee7bfc0 (stale) Mirror 16: 0x5ee7bfc0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x361e1): Mirror 8: 0x5ee7bfc0 (stale) Mirror 11: 0x5ee7bfc0 (stale) Mirror 13: 0x5ee7bfc0 (stale) Mirror 14: 0x5ee7bfc0 (stale) Mirror 15: 0x5ee7bfc0 (stale) Mirror 16: 0x5ee7bfc0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x361e1: skipped 000460 1787838945.090989 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x361e1): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x361e1): Mirror 8: 0x5ee7bfc0 (stale) Mirror 11: 0x5ee7bfc0 (stale) Mirror 13: 0x5ee7bfc0 (stale) Mirror 14: 0x5ee7bfc0 (stale) Mirror 15: 0x5ee7bfc0 (stale) Mirror 16: 0x5ee7bfc0 (stale) CRC-32 checksum value for chunk [0, 0x361e1): Mirror 1: 0x7bf4416e Mirror 17: 0x7bf4416e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x361e1: skipped 000461 1787838946.094099 write 0x03625b thru 0x03ffff (0x9da5 bytes) 000464 1787838946.180919 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x237f34fa (stale) Mirror 11: 0x237f34fa (stale) Mirror 14: 0x237f34fa (stale) Mirror 15: 0x237f34fa (stale) Mirror 16: 0x237f34fa (stale) Mirror 17: 0xdea21547 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x237f34fa (stale) Mirror 11: 0x237f34fa (stale) Mirror 14: 0x237f34fa (stale) Mirror 15: 0x237f34fa (stale) Mirror 16: 0x237f34fa (stale) Mirror 17: 0xdea21547 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped truncating to largest ever: 0x3eb6b 000466 1787838946.617490 trunc from 0x040000 to 0x03eb6a (0x1495 bytes) 000468 1787838946.640977 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3eb6b): Mirror 8: 0xf7a6b2b0 (stale) Mirror 14: 0xf7a6b2b0 (stale) Mirror 15: 0xf7a6b2b0 (stale) Mirror 16: 0xf7a6b2b0 (stale) Mirror 17: 0x3f47e133 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eb6b): Mirror 8: 0xf7a6b2b0 (stale) Mirror 14: 0xf7a6b2b0 (stale) Mirror 15: 0xf7a6b2b0 (stale) Mirror 16: 0xf7a6b2b0 (stale) Mirror 17: 0x3f47e133 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eb6b: skipped 000469 1787838947.064311 trunc from 0x03eb6b to 0x009f8a (0x34be0 bytes) 000472 1787838947.090391 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x9f8b): Mirror 8: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9f8b): Mirror 8: 0x8de05e25 (stale) Mirror 14: 0x8de05e25 (stale) Mirror 15: 0x8de05e25 (stale) Mirror 16: 0x8de05e25 (stale) Mirror 17: 0x7ddba008 (stale) CRC-32 checksum value for chunk [0, 0x9f8b): Mirror 1: 0x7ddba008 Mirror 18: 0x7ddba008 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9f8b: skipped 000473 1787838948.120439 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x9f8b): Mirror 8: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9f8b): Mirror 8: 0x8de05e25 (stale) Mirror 14: 0x8de05e25 (stale) Mirror 15: 0x8de05e25 (stale) Mirror 16: 0x8de05e25 (stale) Mirror 17: 0x7ddba008 (stale) CRC-32 checksum value for chunk [0, 0x9f8b): Mirror 1: 0x7ddba008 Mirror 18: 0x7ddba008 Mirror 19: 0x7ddba008 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9f8b: skipped 000474 1787838949.430978 trunc from 0x009f8b to 0x025cb1 (0x1bd27 bytes) 000475 1787838949.522985 mapread 0x025860 thru 0x025cb1 (0x452 bytes) 000477 1787838949.534319 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25cb2): Mirror 14: 0x4162969d (stale) Mirror 15: 0x4162969d (stale) Mirror 16: 0x4162969d (stale) Mirror 17: 0xdf325c8b (stale) Mirror 18: 0x57481b21 (stale) Mirror 19: 0x57481b21 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25cb2): Mirror 14: 0x4162969d (stale) Mirror 15: 0x4162969d (stale) Mirror 16: 0x4162969d (stale) Mirror 17: 0xdf325c8b (stale) Mirror 18: 0x57481b21 (stale) Mirror 19: 0x57481b21 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25cb2: skipped 000479 1787838949.967550 read 0x007206 thru 0x0162f7 (0xf0f2 bytes) 000480 1787838949.978954 write 0x01cce5 thru 0x022fe1 (0x62fd bytes) 000481 1787838949.997642 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x25cb2): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25cb2): Mirror 14: 0x4162969d (stale) Mirror 15: 0x4162969d (stale) Mirror 16: 0x4162969d (stale) Mirror 17: 0xdf325c8b (stale) Mirror 18: 0x57481b21 (stale) Mirror 19: 0x57481b21 (stale) CRC-32 checksum value for chunk [0, 0x25cb2): Mirror 1: 0xfdb32f42 Mirror 20: 0xfdb32f42 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25cb2: skipped 000482 1787838951.372877 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x25cb2): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25cb2): Mirror 14: 0x4162969d (stale) Mirror 15: 0x4162969d (stale) Mirror 16: 0x4162969d (stale) Mirror 17: 0xdf325c8b (stale) Mirror 18: 0x57481b21 (stale) Mirror 19: 0x57481b21 (stale) CRC-32 checksum value for chunk [0, 0x25cb2): Mirror 1: 0xfdb32f42 Mirror 20: 0xfdb32f42 Mirror 21: 0xfdb32f42 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25cb2: skipped 000484 1787838952.909123 trunc from 0x025cb2 to 0x00cf1b (0x18d96 bytes) 000485 1787838953.039102 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0xcf1c): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xcf1c): Mirror 14: 0x98297f05 (stale) Mirror 15: 0x98297f05 (stale) Mirror 16: 0x98297f05 (stale) Mirror 17: 0xbb4ae3c8 (stale) Mirror 18: 0xd5fdfd2d (stale) Mirror 19: 0xd5fdfd2d (stale) Mirror 20: 0xd5fdfd2d (stale) Mirror 21: 0xd5fdfd2d (stale) CRC-32 checksum value for chunk [0, 0xcf1c): Mirror 1: 0xd5fdfd2d Mirror 22: 0xd5fdfd2d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xcf1c: skipped 000486 1787838954.461924 mapread 0x002c2d thru 0x00cf1b (0xa2ef bytes) 000488 1787838954.496996 trunc from 0x00cf1c to 0x004911 (0x860a bytes) 000490 1787838954.637992 write 0x033394 thru 0x03ef05 (0xbb72 bytes) 000495 1787838954.645336 mapwrite 0x030e0d thru 0x0366d3 (0x58c7 bytes) 000498 1787838954.716980 trunc from 0x03ef06 to 0x00664a (0x388bb bytes) 000499 1787838954.748708 read 0x005be6 thru 0x00664a (0xa65 bytes) 000500 1787838954.755257 mapwrite 0x00f411 thru 0x01c591 (0xd181 bytes) 000501 1787838954.923176 write 0x00f2b3 thru 0x01a79c (0xb4ea bytes) 000502 1787838954.934676 trunc from 0x01c592 to 0x038d62 (0x1c7d1 bytes) 000504 1787838954.982622 MIRROR_RESYNC 10 mirrors 000505 1787838958.774921 MIRROR_RESYNC 10 mirrors 000507 1787838959.311655 trunc from 0x038d63 to 0x011088 (0x27cda bytes) 000509 1787838959.400585 read 0x00420a thru 0x00777f (0x3576 bytes) 000510 1787838959.406163 read 0x00e223 thru 0x011088 (0x2e66 bytes) truncating to largest ever: 0x3f6b5 000512 1787838959.416806 trunc from 0x011089 to 0x03f6b4 (0x2e62c bytes) 000513 1787838959.438662 mapwrite 0x0272fe thru 0x02a38c (0x308f bytes) 000514 1787838959.495387 read 0x021d48 thru 0x02ff46 (0xe1ff bytes) 000515 1787838959.496690 mapwrite 0x02aa2e thru 0x03a758 (0xfd2b bytes) 000516 1787838959.550243 trunc from 0x03f6b5 to 0x00ce40 (0x32874 bytes) 000517 1787838959.573997 read 0x00ab8d thru 0x00ce40 (0x22b4 bytes) 000519 1787838959.576587 trunc from 0x00ce41 to 0x00fc31 (0x2df1 bytes) 000520 1787838959.599708 write 0x02e41b thru 0x038468 (0xa04e bytes) 000523 1787838959.608743 read 0x001d5d thru 0x011985 (0xfc29 bytes) 000525 1787838959.627087 trunc from 0x038469 to 0x033d3d (0x472b bytes) 000526 1787838959.674880 write 0x036193 thru 0x03ffff (0x9e6d bytes) 000529 1787838959.680247 mapread 0x02c163 thru 0x02fbf6 (0x3a94 bytes) 000531 1787838959.684919 mapwrite 0x02942e thru 0x035c8e (0xc861 bytes) 000532 1787838959.804919 read 0x034b69 thru 0x036343 (0x17db bytes) 000533 1787838959.805785 mapwrite 0x02d865 thru 0x03cf4f (0xf6eb bytes) 000534 1787838959.952655 read 0x027a35 thru 0x02db71 (0x613d bytes) 000535 1787838959.953453 mapwrite 0x01ade1 thru 0x02639f (0xb5bf bytes) 000536 1787838960.092842 mapwrite 0x009638 thru 0x014839 (0xb202 bytes) 000538 1787838960.161850 write 0x0374b4 thru 0x03ffff (0x8b4c bytes) 000539 1787838960.164371 trunc from 0x040000 to 0x036265 (0x9d9a bytes) 000540 1787838960.186913 trunc from 0x036266 to 0x00d62c (0x28c39 bytes) 000541 1787838960.210578 mapwrite 0x02ad69 thru 0x037d86 (0xd01e bytes) 000542 1787838960.317099 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37d87): Mirror 14: 0xc7c16a80 (stale) Mirror 16: 0xc7c16a80 (stale) Mirror 17: 0xc7c16a80 (stale) Mirror 18: 0xc7c16a80 (stale) Mirror 19: 0xc7c16a80 (stale) Mirror 20: 0xc7c16a80 (stale) Mirror 21: 0xc7c16a80 (stale) Mirror 22: 0xc7c16a80 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37d87): Mirror 14: 0xc7c16a80 (stale) Mirror 16: 0xc7c16a80 (stale) Mirror 17: 0xc7c16a80 (stale) Mirror 18: 0xc7c16a80 (stale) Mirror 19: 0xc7c16a80 (stale) Mirror 20: 0xc7c16a80 (stale) Mirror 21: 0xc7c16a80 (stale) Mirror 22: 0xc7c16a80 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37d87: skipped 000543 1787838960.929674 read 0x002c0f thru 0x004b25 (0x1f17 bytes) 000544 1787838960.942324 MIRROR_RESYNC 9 mirrors 000546 1787838964.036578 trunc from 0x037d87 to 0x01fd15 (0x18071 bytes) 000550 1787838964.163672 mapwrite 0x03314c thru 0x0355d8 (0x248d bytes) 000552 1787838964.295668 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x355d9): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x355d9): Mirror 14: 0x6bc5583a (stale) Mirror 16: 0x6bc5583a (stale) Mirror 17: 0x6bc5583a (stale) Mirror 18: 0x6bc5583a (stale) Mirror 19: 0x6bc5583a (stale) Mirror 20: 0x6bc5583a (stale) Mirror 21: 0x6bc5583a (stale) Mirror 22: 0x6bc5583a (stale) CRC-32 checksum value for chunk [0, 0x355d9): Mirror 1: 0xf9c243b7 Mirror 23: 0xf9c243b7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x355d9: skipped 000553 1787838965.550578 write 0x02bd9d thru 0x033c3b (0x7e9f bytes) 000557 1787838965.674939 trunc from 0x0355d9 to 0x02c14a (0x948e bytes) 000558 1787838965.729357 mapwrite 0x03b799 thru 0x03ffff (0x4867 bytes) 000559 1787838965.807712 read 0x039546 thru 0x03a1f0 (0xcab bytes) 000564 1787838965.819179 write 0x01da5b thru 0x01eb14 (0x10ba bytes) 000568 1787838965.839112 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xc87c8842 (stale) Mirror 16: 0xc87c8842 (stale) Mirror 17: 0xc87c8842 (stale) Mirror 18: 0xc87c8842 (stale) Mirror 19: 0xc87c8842 (stale) Mirror 20: 0xc87c8842 (stale) Mirror 21: 0xc87c8842 (stale) Mirror 22: 0xc87c8842 (stale) Mirror 23: 0xbd9e2305 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x929e925e Mirror 24: 0x929e925e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000569 1787838967.433093 mapread 0x02fac3 thru 0x03925a (0x9798 bytes) 000574 1787838967.458989 read 0x023550 thru 0x02d84e (0xa2ff bytes) 000575 1787838967.465612 trunc from 0x040000 to 0x01c46f (0x23b90 bytes) 000577 1787838967.600093 mapwrite 0x022efd thru 0x03156b (0xe66f bytes) 000578 1787838967.750169 trunc from 0x03156c to 0x0125bc (0x1efaf bytes) 000579 1787838967.788713 write 0x031eb1 thru 0x0383cd (0x651d bytes) 000580 1787838967.792623 trunc from 0x0383ce to 0x005910 (0x32abd bytes) 000581 1787838967.824601 write 0x03319f thru 0x035a60 (0x28c2 bytes) 000582 1787838967.829305 write 0x002a92 thru 0x00f0a4 (0xc613 bytes) 000584 1787838967.862161 mapread 0x01362f thru 0x0151c5 (0x1b97 bytes) 000586 1787838967.884490 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35a61): Mirror 14: 0xfc83846a (stale) Mirror 16: 0xfc83846a (stale) Mirror 17: 0xfc83846a (stale) Mirror 19: 0xfc83846a (stale) Mirror 20: 0xfc83846a (stale) Mirror 21: 0xfc83846a (stale) Mirror 22: 0xfc83846a (stale) Mirror 23: 0x97aa2845 (stale) Mirror 24: 0xcd2d5496 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35a61): Mirror 14: 0xfc83846a (stale) Mirror 16: 0xfc83846a (stale) Mirror 17: 0xfc83846a (stale) Mirror 19: 0xfc83846a (stale) Mirror 20: 0xfc83846a (stale) Mirror 21: 0xfc83846a (stale) Mirror 22: 0xfc83846a (stale) Mirror 23: 0x97aa2845 (stale) Mirror 24: 0xcd2d5496 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35a61: skipped 000587 1787838968.501526 write 0x00d18a thru 0x0168fc (0x9773 bytes) 000589 1787838968.521560 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x35a61): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35a61): Mirror 14: 0xfc83846a (stale) Mirror 16: 0xfc83846a (stale) Mirror 17: 0xfc83846a (stale) Mirror 19: 0xfc83846a (stale) Mirror 20: 0xfc83846a (stale) Mirror 21: 0xfc83846a (stale) Mirror 22: 0xfc83846a (stale) Mirror 23: 0x97aa2845 (stale) Mirror 24: 0xcd2d5496 (stale) CRC-32 checksum value for chunk [0, 0x35a61): Mirror 1: 0x1293e89d Mirror 25: 0x1293e89d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35a61: skipped 000590 1787838969.935753 write 0x0351f8 thru 0x0384f8 (0x3301 bytes) 000593 1787838970.207130 mapread 0x00cf13 thru 0x0189d3 (0xbac1 bytes) 000594 1787838970.257029 read 0x01fc17 thru 0x024a5d (0x4e47 bytes) 000596 1787838970.269597 mapwrite 0x03d025 thru 0x03ffff (0x2fdb bytes) 000597 1787838970.385240 mapread 0x014356 thru 0x016a0c (0x26b7 bytes) 000598 1787838970.387290 mapwrite 0x00d092 thru 0x00d9d2 (0x941 bytes) 000599 1787838970.455420 mapread 0x0302d2 thru 0x03e14f (0xde7e bytes) 000604 1787838970.483225 trunc from 0x040000 to 0x01906e (0x26f91 bytes) 000606 1787838970.521596 mapread 0x010489 thru 0x013069 (0x2be1 bytes) 000607 1787838970.524254 write 0x01a85f thru 0x01ec25 (0x43c7 bytes) 000608 1787838970.531721 mapread 0x01d3e4 thru 0x01ec25 (0x1842 bytes) 000616 1787838970.570001 MIRROR_RESYNC 11 mirrors 000617 1787838973.275492 mapread 0x006cc6 thru 0x00733d (0x678 bytes) 000618 1787838973.295875 mapread 0x01729e thru 0x0187f3 (0x1556 bytes) 000619 1787838973.338727 read 0x0077bc thru 0x013ed6 (0xc71b bytes) 000620 1787838973.401518 mapread 0x00789d thru 0x0107f2 (0x8f56 bytes) 000621 1787838973.403590 mapread 0x00e770 thru 0x01973f (0xafd0 bytes) 000622 1787838973.404676 write 0x0196d1 thru 0x023d37 (0xa667 bytes) 000624 1787838973.624064 MIRROR_RESYNC 11 mirrors 000625 1787838975.775283 write 0x00a55d thru 0x00dfd1 (0x3a75 bytes) 000626 1787838975.998752 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23d38): Mirror 14: 0x3e07bdd5 (stale) Mirror 16: 0x3e07bdd5 (stale) Mirror 17: 0x3e07bdd5 (stale) Mirror 19: 0x3e07bdd5 (stale) Mirror 20: 0x3e07bdd5 (stale) Mirror 22: 0x3e07bdd5 (stale) Mirror 23: 0x3e07bdd5 (stale) Mirror 24: 0x3e07bdd5 (stale) Mirror 25: 0x3e07bdd5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23d38): Mirror 14: 0x3e07bdd5 (stale) Mirror 16: 0x3e07bdd5 (stale) Mirror 17: 0x3e07bdd5 (stale) Mirror 19: 0x3e07bdd5 (stale) Mirror 20: 0x3e07bdd5 (stale) Mirror 22: 0x3e07bdd5 (stale) Mirror 23: 0x3e07bdd5 (stale) Mirror 24: 0x3e07bdd5 (stale) Mirror 25: 0x3e07bdd5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23d38: skipped 000628 1787838976.894380 read 0x01d29f thru 0x0209ba (0x371c bytes) 000629 1787838976.905125 mapread 0x0044cf thru 0x0092fb (0x4e2d bytes) 000631 1787838976.970773 mapwrite 0x025ff5 thru 0x026fdd (0xfe9 bytes) 000632 1787838977.163556 read 0x007ce0 thru 0x00da88 (0x5da9 bytes) 000634 1787838977.164704 mapwrite 0x0245dd thru 0x02ad50 (0x6774 bytes) 000635 1787838977.295424 mapread 0x008fb8 thru 0x00a066 (0x10af bytes) 000636 1787838977.296261 trunc from 0x02ad51 to 0x020d52 (0x9ffe bytes) 000637 1787838977.319808 mapwrite 0x037b32 thru 0x03a5cf (0x2a9e bytes) 000638 1787838977.409596 write 0x017d1d thru 0x026450 (0xe734 bytes) 000639 1787838977.418978 read 0x03795b thru 0x03a5cf (0x2c75 bytes) 000640 1787838977.420968 trunc from 0x03a5d0 to 0x02b13e (0xf491 bytes) 000641 1787838977.443029 trunc from 0x02b13f to 0x035be3 (0xaaa5 bytes) 000642 1787838977.479418 write 0x029850 thru 0x035b97 (0xc348 bytes) 000643 1787838977.499815 write 0x00d8a7 thru 0x01d844 (0xff9e bytes) 000645 1787838977.501283 write 0x02f3da thru 0x03b689 (0xc2b0 bytes) 000647 1787838977.504042 mapread 0x01c5c3 thru 0x028844 (0xc282 bytes) 000648 1787838977.505167 mapread 0x01a589 thru 0x01d537 (0x2faf bytes) 000650 1787838977.512877 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b68a): Mirror 16: 0x97062b41 (stale) Mirror 17: 0x97062b41 (stale) Mirror 19: 0x97062b41 (stale) Mirror 20: 0x97062b41 (stale) Mirror 22: 0x97062b41 (stale) Mirror 23: 0x97062b41 (stale) Mirror 24: 0x97062b41 (stale) Mirror 25: 0x97062b41 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b68a): Mirror 16: 0x97062b41 (stale) Mirror 17: 0x97062b41 (stale) Mirror 19: 0x97062b41 (stale) Mirror 20: 0x97062b41 (stale) Mirror 22: 0x97062b41 (stale) Mirror 23: 0x97062b41 (stale) Mirror 24: 0x97062b41 (stale) Mirror 25: 0x97062b41 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b68a: skipped 000651 1787838978.248876 write 0x011406 thru 0x018b24 (0x771f bytes) 000653 1787838978.259012 mapwrite 0x00c069 thru 0x0163b8 (0xa350 bytes) 000654 1787838978.363925 write 0x01b9af thru 0x01cc16 (0x1268 bytes) 000658 1787838978.378952 mapread 0x0191fd thru 0x01b081 (0x1e85 bytes) 000659 1787838978.386585 trunc from 0x03b68a to 0x01289d (0x28dec bytes) 000663 1787838978.404358 write 0x00a418 thru 0x015768 (0xb351 bytes) 000664 1787838978.406282 mapwrite 0x0091a3 thru 0x015fdd (0xce3b bytes) 000665 1787838978.469472 trunc from 0x015fde to 0x0323d0 (0x1c3f3 bytes) 000666 1787838978.490020 mapwrite 0x01d178 thru 0x020393 (0x321c bytes) 000667 1787838978.580753 mapread 0x01b3ed thru 0x024f2f (0x9b43 bytes) 000669 1787838978.583891 mapread 0x000259 thru 0x005a75 (0x581d bytes) 000672 1787838978.597590 mapwrite 0x02381a thru 0x02d24e (0x9a35 bytes) 000674 1787838978.696552 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x323d1): Mirror 16: 0x117bc559 (stale) Mirror 17: 0x117bc559 (stale) Mirror 19: 0x117bc559 (stale) Mirror 20: 0x117bc559 (stale) Mirror 23: 0x117bc559 (stale) Mirror 24: 0x117bc559 (stale) Mirror 25: 0x117bc559 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x323d1): Mirror 16: 0x117bc559 (stale) Mirror 17: 0x117bc559 (stale) Mirror 19: 0x117bc559 (stale) Mirror 20: 0x117bc559 (stale) Mirror 23: 0x117bc559 (stale) Mirror 24: 0x117bc559 (stale) Mirror 25: 0x117bc559 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x323d1: skipped 000675 1787838979.176121 read 0x01c3b8 thru 0x023de3 (0x7a2c bytes) 000679 1787838979.186867 trunc from 0x0323d1 to 0x00552e (0x2cea2 bytes) 000684 1787838979.205933 trunc from 0x00552f to 0x0085d6 (0x30a8 bytes) 000686 1787838979.225828 mapwrite 0x0331dd thru 0x034fd3 (0x1df7 bytes) 000687 1787838979.307267 mapwrite 0x03f820 thru 0x03ffff (0x7e0 bytes) 000692 1787838979.405680 mapread 0x034645 thru 0x03ffff (0xb9bb bytes) 000694 1787838979.419755 mapwrite 0x02cba1 thru 0x0356fc (0x8b5c bytes) 000698 1787838979.542564 write 0x008eda thru 0x00c121 (0x3248 bytes) 000700 1787838979.565725 read 0x037935 thru 0x03ffff (0x86cb bytes) 000701 1787838979.566795 trunc from 0x040000 to 0x006bad (0x39452 bytes) 000702 1787838979.600113 MIRROR_RESYNC 8 mirrors 000703 1787838982.055812 read 0x00572e thru 0x006bad (0x1480 bytes) 000704 1787838982.062577 write 0x022a71 thru 0x0311f9 (0xe789 bytes) 000706 1787838982.265947 mapread 0x01b729 thru 0x0253ed (0x9cc5 bytes) 000708 1787838982.322046 write 0x01c120 thru 0x02b557 (0xf438 bytes) 000709 1787838982.324913 mapread 0x01b1d1 thru 0x02701e (0xbe4e bytes) 000710 1787838982.326275 trunc from 0x0311fa to 0x03a368 (0x916f bytes) 000711 1787838982.399475 MIRROR_RESYNC 8 mirrors 000712 1787838985.469567 read 0x038635 thru 0x03a368 (0x1d34 bytes) 000714 1787838985.475158 mapread 0x036bb7 thru 0x03a368 (0x37b2 bytes) 000715 1787838985.486479 mapwrite 0x009c10 thru 0x00cac7 (0x2eb8 bytes) 000717 1787838985.678364 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a369): Mirror 16: 0x80115f15 (stale) Mirror 17: 0x80115f15 (stale) Mirror 19: 0x80115f15 (stale) Mirror 20: 0x80115f15 (stale) Mirror 24: 0x80115f15 (stale) Mirror 25: 0x80115f15 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a369): Mirror 16: 0x80115f15 (stale) Mirror 17: 0x80115f15 (stale) Mirror 19: 0x80115f15 (stale) Mirror 20: 0x80115f15 (stale) Mirror 24: 0x80115f15 (stale) Mirror 25: 0x80115f15 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a369: skipped 000718 1787838986.594049 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a369): Mirror 16: 0x80115f15 (stale) Mirror 17: 0x80115f15 (stale) Mirror 19: 0x80115f15 (stale) Mirror 24: 0x80115f15 (stale) Mirror 25: 0x80115f15 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a369): Mirror 16: 0x80115f15 (stale) Mirror 17: 0x80115f15 (stale) Mirror 19: 0x80115f15 (stale) Mirror 24: 0x80115f15 (stale) Mirror 25: 0x80115f15 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a369: skipped 000719 1787838987.154498 mapwrite 0x03865b thru 0x03d269 (0x4c0f bytes) 000720 1787838987.294903 mapread 0x016e43 thru 0x017278 (0x436 bytes) 000721 1787838987.305885 read 0x00c9bd thru 0x0153de (0x8a22 bytes) 000722 1787838987.311774 mapread 0x01202e thru 0x021973 (0xf946 bytes) 000724 1787838987.320506 mapread 0x026901 thru 0x0341db (0xd8db bytes) 000725 1787838987.339904 MIRROR_RESYNC 6 mirrors 000727 1787838989.843865 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) 000728 1787838990.239800 write 0x01618a thru 0x01fc17 (0x9a8e bytes) 000729 1787838990.379302 mapread 0x02d5c5 thru 0x02df21 (0x95d bytes) 000730 1787838990.394291 read 0x03bcb9 thru 0x03d269 (0x15b1 bytes) 000731 1787838990.407710 read 0x035119 thru 0x036ac5 (0x19ad bytes) 000732 1787838990.435773 read 0x002d19 thru 0x00f59d (0xc885 bytes) 000733 1787838990.436730 mapwrite 0x01aed8 thru 0x029059 (0xe182 bytes) 000734 1787838990.564996 MIRROR_RESYNC 5 mirrors 000735 1787838993.301085 trunc from 0x03d26a to 0x0158dd (0x2798c bytes) 000736 1787838993.421010 MIRROR_RESYNC 5 mirrors 000737 1787838994.990950 write 0x00cfc9 thru 0x00d2b6 (0x2ee bytes) 000740 1787838995.143946 mapwrite 0x020db2 thru 0x030c04 (0xfe53 bytes) 000741 1787838995.303109 write 0x02ac54 thru 0x033172 (0x851f bytes) 000742 1787838995.304687 mapread 0x022b84 thru 0x030055 (0xd4d2 bytes) 000743 1787838995.307092 write 0x0151d4 thru 0x01548c (0x2b9 bytes) 000744 1787838995.319884 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x33173): Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33173): Mirror 17: 0xf1524a9d (stale) Mirror 19: 0xf1524a9d (stale) Mirror 24: 0xf1524a9d (stale) Mirror 25: 0xf1524a9d (stale) CRC-32 checksum value for chunk [0, 0x33173): Mirror 1: 0x467eeb46 Mirror 26: 0x467eeb46 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33173: skipped 000745 1787838996.725717 mapread 0x0018cd thru 0x00baf5 (0xa229 bytes) 000749 1787838996.764077 mapwrite 0x03412d thru 0x03ffff (0xbed3 bytes) 000750 1787838997.037807 write 0x01a6d3 thru 0x01c789 (0x20b7 bytes) 000753 1787838997.048980 trunc from 0x040000 to 0x001431 (0x3ebce bytes) 000754 1787838997.081947 mapwrite 0x0355e8 thru 0x03ffff (0xaa18 bytes) 000757 1787838997.197984 mapwrite 0x02a9de thru 0x02d00a (0x262d bytes) 000758 1787838997.296719 mapwrite 0x0391b1 thru 0x03ffff (0x6e4f bytes) 000760 1787838997.368377 mapread 0x01f02a thru 0x02c4ce (0xd4a5 bytes) 000761 1787838997.378489 mapwrite 0x039b8d thru 0x03ffff (0x6473 bytes) 000762 1787838997.424664 mapread 0x012673 thru 0x01e481 (0xbe0f bytes) 000766 1787838997.437832 read 0x010f8a thru 0x018cf7 (0x7d6e bytes) 000772 1787838997.443955 trunc from 0x040000 to 0x01620b (0x29df4 bytes) 000773 1787838997.471810 mapwrite 0x009c35 thru 0x00afbc (0x1388 bytes) 000774 1787838997.521115 mapwrite 0x012f2a thru 0x017cee (0x4dc5 bytes) 000776 1787838997.627224 trunc from 0x017cef to 0x02ef0e (0x17220 bytes) 000777 1787838997.651523 write 0x02cada thru 0x03234c (0x5873 bytes) 000778 1787838997.658696 mapread 0x01f975 thru 0x02e5a8 (0xec34 bytes) 000779 1787838997.676793 mapwrite 0x02a651 thru 0x037afe (0xd4ae bytes) 000782 1787838997.824529 mapread 0x036391 thru 0x037afe (0x176e bytes) 000783 1787838997.826226 mapwrite 0x0028fc thru 0x00b408 (0x8b0d bytes) 000784 1787838997.888243 trunc from 0x037aff to 0x0115d9 (0x26525 bytes) 000786 1787838997.933572 write 0x001d7b thru 0x00a51f (0x87a5 bytes) 000787 1787838997.945906 mapwrite 0x02bec1 thru 0x02c2a1 (0x3e1 bytes) 000790 1787838998.024822 trunc from 0x02c2a2 to 0x0396b9 (0xd418 bytes) 000791 1787838998.041902 mapwrite 0x0395bc thru 0x03ffff (0x6a44 bytes) 000792 1787838998.110564 read 0x0120e7 thru 0x01367c (0x1596 bytes) 000795 1787838998.123588 MIRROR_RESYNC 6 mirrors 000796 1787839001.836040 read 0x006013 thru 0x009f4e (0x3f3c bytes) 000798 1787839001.852533 MIRROR_RESYNC 6 mirrors 000799 1787839002.816118 write 0x010d68 thru 0x01ea04 (0xdc9d bytes) 000801 1787839003.006439 trunc from 0x040000 to 0x03d3b2 (0x2c4d bytes) 000802 1787839003.071631 MIRROR_RESYNC 6 mirrors 000804 1787839006.490171 mapread 0x025abd thru 0x02a4f1 (0x4a35 bytes) 000805 1787839006.516248 write 0x035bb5 thru 0x036b05 (0xf51 bytes) 000806 1787839006.629263 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3d3b3): Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d3b3): Mirror 17: 0xc1d9fea9 (stale) Mirror 19: 0xc1d9fea9 (stale) Mirror 24: 0xc1d9fea9 (stale) Mirror 25: 0xc1d9fea9 (stale) Mirror 26: 0xc1d9fea9 (stale) CRC-32 checksum value for chunk [0, 0x3d3b3): Mirror 1: 0x2e97800 Mirror 27: 0x2e97800 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d3b3: skipped 000807 1787839008.381631 write 0x019629 thru 0x01fd67 (0x673f bytes) 000808 1787839008.551317 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d3b3): Mirror 17: 0xc1d9fea9 (stale) Mirror 24: 0xc1d9fea9 (stale) Mirror 25: 0xc1d9fea9 (stale) Mirror 26: 0xc1d9fea9 (stale) Mirror 27: 0x2e97800 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d3b3): Mirror 17: 0xc1d9fea9 (stale) Mirror 24: 0xc1d9fea9 (stale) Mirror 25: 0xc1d9fea9 (stale) Mirror 26: 0xc1d9fea9 (stale) Mirror 27: 0x2e97800 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d3b3: skipped 000809 1787839009.066840 mapread 0x035eb0 thru 0x03d3b2 (0x7503 bytes) 000810 1787839009.110713 trunc from 0x03d3b3 to 0x012ccc (0x2a6e6 bytes) 000812 1787839009.145035 mapread 0x00f1ab thru 0x012ccc (0x3b22 bytes) 000814 1787839009.180418 read 0x006903 thru 0x010462 (0x9b60 bytes) 000815 1787839009.204854 mapread 0x011430 thru 0x012ccc (0x189d bytes) 000817 1787839009.213354 write 0x035a4c thru 0x03ffff (0xa5b4 bytes) 000818 1787839009.228414 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 24: 0x72525cd9 (stale) Mirror 25: 0x72525cd9 (stale) Mirror 26: 0x72525cd9 (stale) Mirror 27: 0xd1a8e9c5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 24: 0x72525cd9 (stale) Mirror 25: 0x72525cd9 (stale) Mirror 26: 0x72525cd9 (stale) Mirror 27: 0xd1a8e9c5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000820 1787839009.973699 mapwrite 0x028ea9 thru 0x029494 (0x5ec bytes) 000821 1787839010.064134 mapwrite 0x02843e thru 0x0293cb (0xf8e bytes) 000822 1787839010.111066 write 0x015023 thru 0x01fd4a (0xad28 bytes) 000823 1787839010.121724 read 0x0393ea thru 0x03ffff (0x6c16 bytes) 000826 1787839010.131136 write 0x00844e thru 0x016a3b (0xe5ee bytes) 000828 1787839010.141298 mapread 0x022380 thru 0x02d6f2 (0xb373 bytes) 000829 1787839010.160646 trunc from 0x040000 to 0x01ef6e (0x21091 bytes) 000831 1787839010.186030 mapread 0x007ca5 thru 0x01427e (0xc5da bytes) 000832 1787839010.200547 read 0x01cd70 thru 0x01ef6e (0x21ff bytes) 000835 1787839010.203231 write 0x034cb4 thru 0x038483 (0x37d0 bytes) 000836 1787839010.207492 trunc from 0x038484 to 0x006032 (0x32451 bytes) 000837 1787839010.233681 write 0x01a711 thru 0x020b22 (0x6412 bytes) 000838 1787839010.241332 MIRROR_RESYNC 5 mirrors 000840 1787839012.561127 trunc from 0x020b23 to 0x025c6c (0x514a bytes) 000842 1787839012.815402 write 0x01dde3 thru 0x01fcef (0x1f0d bytes) 000843 1787839012.838085 trunc from 0x025c6d to 0x020df2 (0x4e7a bytes) 000844 1787839012.885067 write 0x01b97b thru 0x02273a (0x6dc0 bytes) 000849 1787839012.901451 trunc from 0x02273b to 0x03d910 (0x1b1d6 bytes) 000854 1787839012.946562 mapwrite 0x00f06a thru 0x010dbb (0x1d52 bytes) 000855 1787839013.084941 mapwrite 0x00a8b9 thru 0x00c43e (0x1b86 bytes) 000856 1787839013.209930 mapwrite 0x0012b6 thru 0x00664f (0x539a bytes) 000858 1787839013.448172 mapwrite 0x01f366 thru 0x02d8a3 (0xe53e bytes) 000861 1787839013.672803 trunc from 0x03d911 to 0x03ca12 (0xefe bytes) 000864 1787839013.714608 mapread 0x037eba thru 0x038a59 (0xba0 bytes) 000865 1787839013.763033 trunc from 0x03ca13 to 0x028c63 (0x13daf bytes) 000866 1787839013.823809 mapread 0x027a32 thru 0x028c63 (0x1232 bytes) 000867 1787839013.825043 read 0x010178 thru 0x01ac5b (0xaae4 bytes) 000868 1787839013.857836 mapread 0x0237f8 thru 0x028c63 (0x546c bytes) 000869 1787839013.859046 read 0x010c48 thru 0x019c47 (0x9000 bytes) 000871 1787839013.871442 read 0x01005c thru 0x01eb7c (0xeb21 bytes) 000872 1787839013.875328 trunc from 0x028c64 to 0x0352c2 (0xc65f bytes) 000873 1787839013.914117 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x352c3): Mirror 24: 0xbb972061 (stale) Mirror 25: 0xbb972061 (stale) Mirror 26: 0xbb972061 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x352c3): Mirror 24: 0xbb972061 (stale) Mirror 25: 0xbb972061 (stale) Mirror 26: 0xbb972061 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x352c3: skipped 000874 1787839014.582131 read 0x028897 thru 0x0352c2 (0xca2c bytes) 000875 1787839014.612966 trunc from 0x0352c3 to 0x00c263 (0x2905f bytes) 000879 1787839014.648710 mapwrite 0x006e93 thru 0x0147b2 (0xd920 bytes) 000883 1787839014.832528 read 0x012e2e thru 0x0147b2 (0x1985 bytes) 000884 1787839014.833794 write 0x0331c0 thru 0x03bd8b (0x8bcc bytes) 000885 1787839014.835937 read 0x014e6c thru 0x01d25e (0x83f3 bytes) 000888 1787839014.854066 read 0x02877a thru 0x0333e9 (0xac70 bytes) 000891 1787839014.856241 read 0x00bc6a thru 0x00bd32 (0xc9 bytes) 000892 1787839014.858387 mapread 0x0087d9 thru 0x0117e9 (0x9011 bytes) 000894 1787839014.865166 MIRROR_RESYNC 4 mirrors 000896 1787839017.039613 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) 000898 1787839017.617276 MIRROR_RESYNC 3 mirrors 000900 1787839018.235249 mapwrite 0x03abed thru 0x03ffff (0x5413 bytes) 000901 1787839018.471536 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 25: 0x12764224 (stale) Mirror 26: 0x12764224 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x40c11d3b Mirror 27: 0x40c11d3b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000902 1787839019.864506 trunc from 0x040000 to 0x02c508 (0x13af7 bytes) 000903 1787839020.001555 write 0x02a50f thru 0x02f7a2 (0x5294 bytes) 000904 1787839020.017060 MIRROR_RESYNC 4 mirrors 000905 1787839021.917520 MIRROR_RESYNC 4 mirrors 000907 1787839022.613280 trunc from 0x02f7a3 to 0x005a71 (0x29d31 bytes) 000909 1787839022.752589 trunc from 0x005a72 to 0x00f996 (0x9f25 bytes) 000911 1787839022.785646 trunc from 0x00f997 to 0x01b848 (0xbeb2 bytes) 000912 1787839022.851498 write 0x0361d1 thru 0x03cf00 (0x6d30 bytes) 000915 1787839022.860160 mapwrite 0x012965 thru 0x017188 (0x4824 bytes) 000919 1787839023.035322 MIRROR_RESYNC 4 mirrors 000921 1787839025.835670 mapwrite 0x031418 thru 0x0389eb (0x75d4 bytes) 000922 1787839026.248265 trunc from 0x03cf01 to 0x03b605 (0x18fb bytes) 000923 1787839026.318489 mapread 0x024cbb thru 0x027a01 (0x2d47 bytes) 000925 1787839026.331801 mapread 0x01f626 thru 0x025359 (0x5d34 bytes) 000926 1787839026.332872 write 0x0061d3 thru 0x00a1df (0x400d bytes) 000928 1787839026.352314 mapwrite 0x0011cb thru 0x011133 (0xff69 bytes) 000933 1787839026.531948 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b606): Mirror 26: 0xc239adb6 (stale) Mirror 27: 0xc239adb6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b606): Mirror 26: 0xc239adb6 (stale) Mirror 27: 0xc239adb6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b606: skipped 000935 1787839027.395632 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b606): Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b606): Mirror 26: 0xc239adb6 (stale) Mirror 27: 0xc239adb6 (stale) CRC-32 checksum value for chunk [0, 0x3b606): Mirror 1: 0x8cc83112 Mirror 28: 0x8cc83112 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b606: skipped 000936 1787839029.731664 write 0x0356b4 thru 0x03ec9d (0x95ea bytes) 000937 1787839029.989418 read 0x023b62 thru 0x032a86 (0xef25 bytes) 000940 1787839030.022073 mapwrite 0x0204dc thru 0x02cab1 (0xc5d6 bytes) 000941 1787839030.196355 read 0x01a1dd thru 0x027d12 (0xdb36 bytes) 000942 1787839030.209159 mapwrite 0x0090b4 thru 0x0177e3 (0xe730 bytes) 000943 1787839030.280440 MIRROR_RESYNC 4 mirrors 000944 1787839032.432842 MIRROR_EXTEND to 5 mirrors 000945 1787839033.555546 read 0x01eac1 thru 0x0297a4 (0xace4 bytes) 000946 1787839033.568172 write 0x03deb6 thru 0x03ffff (0x214a bytes) 000947 1787839033.723646 MIRROR_RESYNC 5 mirrors 000948 1787839036.550447 write 0x030aa2 thru 0x03e59b (0xdafa bytes) 000950 1787839036.750276 mapwrite 0x00fee7 thru 0x01fcd0 (0xfdea bytes) 000951 1787839036.914320 mapwrite 0x009a6a thru 0x00fcef (0x6286 bytes) 000952 1787839036.996771 mapread 0x00c89b thru 0x018c1b (0xc381 bytes) 000954 1787839037.006204 read 0x03fede thru 0x03ffff (0x122 bytes) 000955 1787839037.014701 write 0x0072fd thru 0x010b87 (0x988b bytes) 000956 1787839037.016436 mapread 0x014fc6 thru 0x020aa0 (0xbadb bytes) 000957 1787839037.020529 trunc from 0x040000 to 0x013d39 (0x2c2c6 bytes) 000958 1787839037.065917 mapwrite 0x0036a0 thru 0x008e9b (0x57fc bytes) 000961 1787839037.177874 read 0x00fb7c thru 0x013d39 (0x41be bytes) 000963 1787839037.180538 write 0x02c41e thru 0x02ead4 (0x26b7 bytes) 000965 1787839037.182898 trunc from 0x02ead5 to 0x00d3a4 (0x21730 bytes) 000966 1787839037.209781 trunc from 0x00d3a5 to 0x01c507 (0xf163 bytes) 000967 1787839037.254302 read 0x0186bc thru 0x01c507 (0x3e4c bytes) 000969 1787839037.287124 write 0x017dde thru 0x01a03d (0x2260 bytes) 000973 1787839037.297908 mapread 0x017303 thru 0x01b427 (0x4125 bytes) 000975 1787839037.301526 write 0x002a20 thru 0x00fcbf (0xd2a0 bytes) 000976 1787839037.303054 mapwrite 0x018411 thru 0x0207ea (0x83da bytes) 000977 1787839037.473246 mapwrite 0x00de2a thru 0x018db8 (0xaf8f bytes) 000978 1787839037.649726 read 0x01af86 thru 0x01cbee (0x1c69 bytes) 000979 1787839037.650906 read 0x006bbc thru 0x010a37 (0x9e7c bytes) 000980 1787839037.652567 trunc from 0x0207eb to 0x000b00 (0x1fcea bytes) 000981 1787839037.685053 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xb01): Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb01): Mirror 26: 0xa31fd0b0 (stale) Mirror 27: 0xa31fd0b0 (stale) Mirror 28: 0xa31fd0b0 (stale) Mirror 29: 0xa31fd0b0 (stale) CRC-32 checksum value for chunk [0, 0xb01): Mirror 1: 0xa31fd0b0 Mirror 30: 0xa31fd0b0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb01: skipped 000983 1787839038.730515 read 0x0007e8 thru 0x000b00 (0x319 bytes) 000989 1787839038.738846 mapread 0x0004b9 thru 0x000b00 (0x648 bytes) 000990 1787839038.739568 trunc from 0x000b01 to 0x0327c5 (0x31cc5 bytes) 000994 1787839038.947383 mapread 0x01d156 thru 0x02235c (0x5207 bytes) 000995 1787839038.970415 read 0x02a1c0 thru 0x0327c5 (0x8606 bytes) 000996 1787839038.984870 trunc from 0x0327c6 to 0x024aa3 (0xdd22 bytes) 000997 1787839039.061962 trunc from 0x024aa4 to 0x017f78 (0xcb2b bytes) 000998 1787839039.112481 mapwrite 0x00eb19 thru 0x016282 (0x776a bytes) 000999 1787839039.256799 write 0x008cd7 thru 0x01871c (0xfa46 bytes) 001000 1787839039.260156 trunc from 0x01871d to 0x038550 (0x1fe34 bytes) All operations completed A-OK! PASS 70a (157s) == sanity-flr test 71: check mirror extend parent fid ==== 09:57:25 (1787839045) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 09:57:29 (1787839049) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.349745 s, 6.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38365 s, 2.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e8:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 09:57:40 (1787839060) Starting client: oleg105-client.virtnet: -o user_xattr,flock 192.168.201.105@tcp:/lustre /mnt/lustre2 Starting client: oleg105-client.virtnet: -o user_xattr,flock 192.168.201.105@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 4131552 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8906688 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4304896 .. Extending file size to 8666272 .. Extending file size to 9035232 .. Extending file size to 9085792 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9243328 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58792 58793 58796 58798 58799 failed 192.168.201.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg105-client.virtnet /mnt/lustre2 (opts:) 192.168.201.105@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg105-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (81s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:59:01 (1787839141) Starting client: oleg105-client.virtnet: -o user_xattr,flock 192.168.201.105@tcp:/lustre /mnt/lustre2 Starting client: oleg105-client.virtnet: -o user_xattr,flock 192.168.201.105@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7768928 .. Extending file size to 7985792 .. failed Extending file size to 7769440 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8336480 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8120480 .. Extending file size to 8682784 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8423360 .. Extending file size to 8709248 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8704064 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8934240 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 60201 60202 60205 60207 60208 failed 192.168.201.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg105-client.virtnet /mnt/lustre2 (opts:) 192.168.201.105@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg105-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 10:00:18 (1787839218) Starting client: oleg105-client.virtnet: -o user_xattr,flock 192.168.201.105@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.106332 s, 19.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.347249 s, 6.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0xcb2d439b (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.201.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg105-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 10:00:39 (1787839239) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0805524 s, 26.0 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 10:00:47 (1787839247) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x203:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31f:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31f:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x320:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x303:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 10:00:56 (1787839256) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.196878 s, 10.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x321:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x304:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x305:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 10:01:03 (1787839263) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.311144 s, 6.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x322:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x323:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x306:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x322:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x323:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x306:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x307:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 10:01:11 (1787839271) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.281607 s, 7.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x324:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x324:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x309:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 10:01:44 (1787839304) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg105-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121487 s, 8.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x325:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x325:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x326:0x0] } PASS 204d (64s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 10:02:48 (1787839368) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg105-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x327:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x328:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.273255 s, 7.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x327:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x328:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x327:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x328:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (58s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 10:03:46 (1787839426) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg105-server mds-ost sync done. Creating new pool test_204f oleg105-server: Pool lustre.test_204f created Adding targets to pool oleg105-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg105-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.251539 s, 8.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x329:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x329:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg105-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg105-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg105-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 10:05:06 (1787839506) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 10:05:15 (1787839515) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 10:05:24 (1787839524) oleg105-server: Pool lustre.test_206 created Adding targets to pool oleg105-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg105-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg105-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg105-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg105-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 10:05:51 (1787839551) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103651 s, 10.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869452 s, 12.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xd10d5568 Mirror 3: 0xd10d5568 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 10:06:03 (1787839563) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 10:06:07 (1787839567) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 10:06:11 (1787839571) Starting client: oleg105-client.virtnet: -o user_xattr,flock 192.168.201.105@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.174084 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xa60000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2330000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2330000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2330000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1620000 Read this (4 bytes): BBBB 192.168.201.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg105-client.virtnet /mnt/lustre2 (opts:) PASS 209a (45s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 10:06:56 (1787839616) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0266208 s, 308 kB/s the very first read read with warm cache PASS 209b (10s) == sanity-flr test 210a: handle broken mirrored lovea ==== 10:07:06 (1787839626) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116012 s, 9.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x228 lmm_fid: [0x200000404:0x228:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x228 lmm_fid: [0x200000404:0x228:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x318:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834472 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 10:07:08.000000000 -0400 Modify: 2026-08-27 10:07:08.000000000 -0400 Change: 2026-08-27 10:07:11.000000000 -0400 Birth: 2026-08-27 10:07:08.000000000 -0400 PASS 210a (13s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 10:07:19 (1787839639) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 10:07:24 (1787839644) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.987243 s, 41.5 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.110135 s, 37.2 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 10:07 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22a:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 10:07 /mnt/lustre/f211.sanity-flr PASS 211 (13s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 10:07:37 (1787839657) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 10:07:45 (1787839665) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1884 sec =========== 10:07:54 (1787839674) === sanity-flr: start cleanup 10:07:55 (1787839675) === === sanity-flr: finish cleanup 10:08:01 (1787839681) ===