-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 06:42:26 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg428-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 06:42:46 (1787827366) === oleg428-client.virtnet: executing check_config_client /mnt/lustre oleg428-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg428-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9ba390163000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9ba390163000.idle_timeout=debug disable quota as required oleg428-server: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 06:43:25 (1787827405) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7629] == sanity-flr test 0a: lfs mirror create with -N option == 06:43:28 (1787827408) PASS 0a (24s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 06:43:52 (1787827432) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 06:43:56 (1787827436) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 06:44:00 (1787827440) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 06:44:28 (1787827468) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 06:44:32 (1787827472) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 06:44:36 (1787827476) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9ba390163000.stats=clear osc.lustre-OST0001-osc-ffff9ba390163000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 06:44:44 (1787827484) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 06:44:56 (1787827496) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 06:45:05 (1787827505) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 06:45:09 (1787827509) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 06:45:18 (1787827518) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 06:45:27 (1787827527) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 06:45:35 (1787827535) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 06:45:38 (1787827538) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 06:45:46 (1787827546) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 06:45:58 (1787827558) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 06:46:04 (1787827564) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159136 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221305 s, 4.7 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 06:46:14 (1787827574) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 0.569448 s, 33.1 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 06:46:23 (1787827583) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 06:46:32 (1787827592) Stopping /mnt/lustre-ost1 (opts:) on oleg428-server oleg428-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ba390163000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ba390163000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg428-server: oleg428-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg428-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg428-server oleg428-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ba390163000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ba390163000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg428-server: oleg428-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg428-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (80s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 06:47:52 (1787827672) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.422209 s, 12.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg428-server oleg428-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ba390163000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ba390163000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg428-server: oleg428-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg428-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (46s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 06:48:39 (1787827719) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg428-server oleg428-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ba390163000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ba390163000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg428-server: oleg428-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg428-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg428-server oleg428-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ba390163000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ba390163000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg428-server: oleg428-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg428-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (83s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 06:50:02 (1787827802) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.50871 s, 9.5 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 06:50 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.991s user 0m0.000s sys 0m0.195s Stopping /mnt/lustre-ost1 (opts:) on oleg428-server oleg428-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ba390163000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ba390163000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.461s user 0m0.000s sys 0m0.182s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg428-server: oleg428-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg428-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg428-server oleg428-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ba390163000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ba390163000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.226s user 0m0.001s sys 0m0.184s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg428-server: oleg428-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg428-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg428-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ba390163000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ba390163000.ost_server_uuid in FULL state after 0 sec PASS 33b (103s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 06:51:45 (1787827905) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 06:51:48 (1787827908) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 06:51:51 (1787827911) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 06:51:54 (1787827914) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 06:52:01 (1787827921) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.570543 s, 22.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 06:52:15 (1787827935) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426731 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046452 s, 22.6 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 06:52:22 (1787827942) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 06:52:31 (1787827951) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00133003 s, 770 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00216933 s, 472 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00153508 s, 667 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0016352 s, 626 kB/s flv=10 olv=10 PASS 36d (14s) == sanity-flr test 37: mirror I/O API verification ======= 06:52:44 (1787827964) create test files with size 15749396 .. 8b2fc9487024db2edaf4b4001e920a21 - 4c852bf57a4e5b00d8848e161b3baac3 - 98fde601baac7d14c02f877818fa32f2 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg428-client.virtnet /mnt/lustre (opts:) Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 06:53:10 (1787827990) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (18s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 06:53:28 (1787828008) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 06:53:35 (1787828015) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15067 s, 7.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038096 s, 27.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 06:53:45 (1787828025) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.182031 s, 11.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.368022 s, 11.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.336569 s, 6.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 06:54:02 (1787828042) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 06:54:06 (1787828046) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 06:54:10 (1787828050) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432248 s, 24.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 43c: read prefer uncompressed mirror == 06:54:19 (1787828059) llite.lustre-ffff9ba38e4ed800.enable_compression=1 fail_loc=0x1425 fail_val=-1 PASS 43c (8s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 06:54:27 (1787828067) llite.lustre-ffff9ba38e4ed800.enable_compression=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031188 s, 33.6 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341277 s, 30.7 MB/s **verify components PASS 43d (10s) == sanity-flr test 44a: lfs mirror split check =========== 06:54:37 (1787828077) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 06:54:41 (1787828081) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xda:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xda:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 06:54:52 (1787828092) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.275344 s, 38.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x1f:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg428-client.virtnet /mnt/lustre (opts:) Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 06:55:05 (1787828105) llite.lustre-ffff9ba391af8000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.323927 s, 32.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg428-client.virtnet /mnt/lustre (opts:) Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9ba390163800.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 06:55:20 (1787828120) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (44s) == sanity-flr test 46: Verify setstripe --copy option ==== 06:56:04 (1787828164) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (38s) == sanity-flr test 47: Verify mirror obj alloc =========== 06:56:42 (1787828202) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 06:56:46 (1787828206) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.388173 s, 8.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.258449 s, 12.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x715e9e9c (nosync,stale) Mirror 2: 0x715e9e9c (nosync,stale) ** verify mirror 2 doesn't change original checksum: 0882feed65b68582141b818d6f1007c3 - mirror 1 checksum: d6690de151f6d2f1f1127e07e137f690 - mirror 2 checksum: 0882feed65b68582141b818d6f1007c3 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T06:56:51-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xef:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T06:56:51-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xf0:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 0882feed65b68582141b818d6f1007c3 - mirror 1 checksum: d6690de151f6d2f1f1127e07e137f690 - mirror 2 checksum: d6690de151f6d2f1f1127e07e137f690 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T06:56:51-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xef:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T06:56:51-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xf0:0x0] } PASS 48 (13s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 06:57:01 (1787828221) Failing mds1 on oleg428-server Stopping /mnt/lustre-mds1 (opts:) on oleg428-server 06:57:08 (1787828228) shut down facet: mds1 facet_host: oleg428-server facet_failover_host: oleg428-server Failover mds1 to oleg428-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg428-server: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-MDT0000 06:57:39 (1787828259) targets are mounted 06:57:39 (1787828259) facet_failover done oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702468 s, 14.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xf1:0x0] } PASS 50A (54s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 06:57:55 (1787828275) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 06:57:58 (1787828278) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 06:58:01 (1787828281) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00673198 s, 2.4 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 06:58:08 (1787828288) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 06:58:12 (1787828292) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00575135 s, 712 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.085244 s, 48.1 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 06:58:20 (1787828300) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 06:58:44 (1787828324) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 06:59:15 (1787828355) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 06:59:39 (1787828379) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 07:00:03 (1787828403) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 07:00:10 (1787828410) mirror create pid 55571 mirror split pid 55573 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 55571 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 55573 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 07:01:17 (1787828477) Seed set to 8479 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1787828479.620248 mapwrite 0x012f12 thru 0x014783 (0x1872 bytes) 000005 1787828479.741307 read 0x009940 thru 0x014783 (0xae44 bytes) 000009 1787828479.752341 read 0x00e4b6 thru 0x014783 (0x62ce bytes) truncating to largest ever: 0x2f8d 000011 1787828479.754983 trunc from 0x014784 to 0x002f8c (0x117f7 bytes) 000013 1787828479.776786 mapread 0x0011b8 thru 0x002f8c (0x1dd5 bytes) 000014 1787828479.787089 read 0x0008cc thru 0x002f8c (0x26c1 bytes) 000016 1787828479.791404 mapread 0x00140c thru 0x002f8c (0x1b81 bytes) 000017 1787828479.792906 mapread 0x0022f4 thru 0x002f8c (0xc99 bytes) truncating to largest ever: 0x2c309 000018 1787828479.794896 trunc from 0x002f8d to 0x02c308 (0x2937c bytes) 000019 1787828479.813249 mapread 0x01cbd9 thru 0x0297ac (0xcbd4 bytes) 000022 1787828479.837201 write 0x03a908 thru 0x03ad94 (0x48d bytes) 000023 1787828479.841156 mapread 0x009973 thru 0x013108 (0x9796 bytes) 000024 1787828479.843006 mapread 0x025934 thru 0x0293c0 (0x3a8d bytes) 000026 1787828479.844719 read 0x01bd78 thru 0x01d492 (0x171b bytes) 000027 1787828479.847156 write 0x005832 thru 0x011671 (0xbe40 bytes) 000030 1787828479.856005 mapwrite 0x0277fd thru 0x036c86 (0xf48a bytes) 000031 1787828479.956707 read 0x01a3ac thru 0x028a10 (0xe665 bytes) 000034 1787828479.960510 mapread 0x01735d thru 0x0258bb (0xe55f bytes) 000036 1787828479.962454 trunc from 0x03ad95 to 0x02a320 (0x10a74 bytes) 000039 1787828479.990782 mapwrite 0x0185a4 thru 0x020c54 (0x86b1 bytes) 000042 1787828480.061868 mapwrite 0x01ac58 thru 0x024298 (0x9641 bytes) 000043 1787828480.146932 read 0x009f1e thru 0x00d040 (0x3123 bytes) 000046 1787828480.148584 mapwrite 0x034bbf thru 0x03bd47 (0x7189 bytes) 000047 1787828480.307308 mapread 0x0018d4 thru 0x007b52 (0x627f bytes) 000049 1787828480.314548 read 0x01eb86 thru 0x02402d (0x54a8 bytes) 000051 1787828480.320367 mapwrite 0x02d676 thru 0x037e39 (0xa7c4 bytes) 000052 1787828480.413427 mapwrite 0x030717 thru 0x0318af (0x1199 bytes) 000053 1787828480.474084 mapread 0x024165 thru 0x0242ed (0x189 bytes) 000056 1787828480.477763 mapwrite 0x016796 thru 0x0229c5 (0xc230 bytes) 000057 1787828480.589247 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x37c63 000060 1787828481.534861 trunc from 0x03bd48 to 0x037c62 (0x40e5 bytes) 000062 1787828481.677373 mapwrite 0x0079ee thru 0x016c77 (0xf28a bytes) truncating to largest ever: 0x3a6c0 000064 1787828481.778953 trunc from 0x037c63 to 0x03a6bf (0x2a5d bytes) 000065 1787828481.822300 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3a6c0): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a6c0): Mirror 2: 0x77a541c6 (stale) CRC-32 checksum value for chunk [0, 0x3a6c0): Mirror 1: 0xcc9e03b0 Mirror 3: 0xcc9e03b0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a6c0: skipped 000070 1787828483.283053 MIRROR_RESYNC 3 mirrors 000072 1787828484.303959 write 0x030c69 thru 0x03e468 (0xd800 bytes) 000073 1787828484.383372 read 0x009dd1 thru 0x018891 (0xeac1 bytes) 000075 1787828484.408272 MIRROR_RESYNC 3 mirrors 000078 1787828485.314997 trunc from 0x03e469 to 0x032fd4 (0xb494 bytes) 000079 1787828485.463335 MIRROR_RESYNC 3 mirrors 000081 1787828486.448901 trunc from 0x032fd5 to 0x03a017 (0x7043 bytes) 000082 1787828486.547240 write 0x01abde thru 0x0269cf (0xbdf2 bytes) 000084 1787828486.586078 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a018): Mirror 2: 0x4261bf17 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a018): Mirror 2: 0x4261bf17 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a018: skipped 000086 1787828487.010376 trunc from 0x03a018 to 0x01eeed (0x1b12a bytes) 000087 1787828487.024383 mapwrite 0x02f44e thru 0x030747 (0x12fa bytes) 000089 1787828487.102497 mapread 0x00ae38 thru 0x0170a3 (0xc26c bytes) 000091 1787828487.136891 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x30748): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30748): Mirror 2: 0x1adb2593 (stale) CRC-32 checksum value for chunk [0, 0x30748): Mirror 1: 0xd280da9f Mirror 3: 0xd280da9f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30748: skipped 000094 1787828488.175171 trunc from 0x030748 to 0x02a4a5 (0x62a2 bytes) 000095 1787828488.290554 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a4a6): Mirror 3: 0xfb9bbf8c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a4a6): Mirror 3: 0xfb9bbf8c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a4a6: skipped 000097 1787828488.638067 write 0x01aefe thru 0x01d615 (0x2718 bytes) 000098 1787828488.650145 write 0x00ea4a thru 0x0164b6 (0x7a6d bytes) 000099 1787828488.664049 mapread 0x01c5e6 thru 0x0285b0 (0xbfcb bytes) 000100 1787828488.688505 trunc from 0x02a4a6 to 0x024ebb (0x55ea bytes) 000102 1787828488.723072 write 0x021a7c thru 0x025bbe (0x4143 bytes) 000104 1787828488.731392 trunc from 0x025bbf to 0x008d91 (0x1ce2d bytes) 000105 1787828488.769173 mapread 0x0082ab thru 0x008d91 (0xae7 bytes) 000106 1787828488.781003 mapread 0x00217f thru 0x007ec8 (0x5d4a bytes) 000108 1787828488.784229 write 0x007dfd thru 0x00edc1 (0x6fc5 bytes) 000109 1787828488.787241 trunc from 0x00edc2 to 0x01e895 (0xfad4 bytes) 000110 1787828488.818961 read 0x0152e2 thru 0x01e895 (0x95b4 bytes) 000111 1787828488.833061 mapread 0x000440 thru 0x00673b (0x62fc bytes) 000112 1787828488.844736 trunc from 0x01e896 to 0x032cb3 (0x1441e bytes) 000113 1787828488.886115 read 0x014f05 thru 0x017ebf (0x2fbb bytes) 000114 1787828488.902032 read 0x0286c3 thru 0x02a3d9 (0x1d17 bytes) 000115 1787828488.902999 mapread 0x030fe4 thru 0x032aa9 (0x1ac6 bytes) 000117 1787828488.913426 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x32cb4): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32cb4): Mirror 3: 0x980120f7 (stale) CRC-32 checksum value for chunk [0, 0x32cb4): Mirror 1: 0x544f22eb Mirror 4: 0x544f22eb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32cb4: skipped 000118 1787828490.213627 read 0x010988 thru 0x01a7c2 (0x9e3b bytes) 000120 1787828490.224208 mapwrite 0x013f2b thru 0x01b32c (0x7402 bytes) 000121 1787828490.474473 write 0x0305c4 thru 0x031664 (0x10a1 bytes) 000122 1787828490.492854 read 0x024733 thru 0x032cb3 (0xe581 bytes) 000123 1787828490.510147 mapread 0x01deb5 thru 0x0210e2 (0x322e bytes) 000124 1787828490.511533 mapread 0x00162c thru 0x00bb8f (0xa564 bytes) 000125 1787828490.513427 write 0x032ace thru 0x03ffff (0xd532 bytes) 000126 1787828490.515737 trunc from 0x040000 to 0x0124bc (0x2db43 bytes) 000127 1787828490.549012 mapread 0x0039bd thru 0x003c88 (0x2cc bytes) 000129 1787828490.550647 mapwrite 0x03ab0c thru 0x03ffff (0x54f4 bytes) 000130 1787828490.625952 mapread 0x032b52 thru 0x03ca7d (0x9f2c bytes) 000131 1787828490.638280 mapread 0x015cfe thru 0x01c9ba (0x6cbd bytes) 000134 1787828490.710255 read 0x00b82f thru 0x01b425 (0xfbf7 bytes) 000135 1787828490.727428 write 0x0106f7 thru 0x01488b (0x4195 bytes) 000136 1787828490.728609 mapwrite 0x01d5b5 thru 0x027b54 (0xa5a0 bytes) 000137 1787828490.844174 mapwrite 0x007b0c thru 0x00c02f (0x4524 bytes) 000139 1787828490.922996 read 0x00a5b5 thru 0x00e66e (0x40ba bytes) 000140 1787828490.923707 write 0x01fbaa thru 0x02747b (0x78d2 bytes) 000141 1787828490.928606 write 0x02c0e3 thru 0x03a6c7 (0xe5e5 bytes) 000143 1787828490.938066 read 0x03cfcf thru 0x03ffff (0x3031 bytes) 000144 1787828490.939304 write 0x011e11 thru 0x01728a (0x547a bytes) 000145 1787828490.943975 mapread 0x024ae5 thru 0x0323a0 (0xd8bc bytes) 000152 1787828490.953527 mapread 0x03435e thru 0x03ffff (0xbca2 bytes) 000153 1787828490.954939 mapwrite 0x003d27 thru 0x013c22 (0xfefc bytes) 000154 1787828491.045065 read 0x001855 thru 0x0055f9 (0x3da5 bytes) 000155 1787828491.051073 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf8f104af (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf8f104af (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000157 1787828491.493249 trunc from 0x040000 to 0x012a0f (0x2d5f0 bytes) 000158 1787828491.521236 write 0x02bf3a thru 0x0335eb (0x76b2 bytes) 000160 1787828491.524162 write 0x024729 thru 0x0336f5 (0xefcd bytes) 000161 1787828491.532052 mapwrite 0x001f6c thru 0x0100c4 (0xe159 bytes) 000162 1787828491.656583 write 0x03bb4c thru 0x03ffff (0x44b4 bytes) 000163 1787828491.658744 mapwrite 0x01d371 thru 0x0256f5 (0x8385 bytes) 000164 1787828491.775135 read 0x017df3 thru 0x025a2b (0xdc39 bytes) 000166 1787828491.776580 read 0x016bc5 thru 0x018280 (0x16bc bytes) 000168 1787828491.781650 read 0x0169fc thru 0x021ca0 (0xb2a5 bytes) 000169 1787828491.783159 write 0x030672 thru 0x03a64d (0x9fdc bytes) 000170 1787828491.805817 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf8f104af (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6edba20f Mirror 5: 0x6edba20f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000173 1787828493.154875 read 0x002da4 thru 0x011d49 (0xefa6 bytes) 000174 1787828493.167404 mapread 0x027d66 thru 0x02d5b5 (0x5850 bytes) 000175 1787828493.218487 write 0x018623 thru 0x026976 (0xe354 bytes) 000179 1787828493.309870 read 0x01d999 thru 0x01f43b (0x1aa3 bytes) 000180 1787828493.311045 mapread 0x022922 thru 0x022e5c (0x53b bytes) 000181 1787828493.312149 read 0x03ef0e thru 0x03ffff (0x10f2 bytes) 000183 1787828493.331401 mapread 0x023185 thru 0x025292 (0x210e bytes) 000184 1787828493.338432 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf8f104af (stale) Mirror 5: 0x6edba20f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x720e63c3 Mirror 6: 0x720e63c3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000188 1787828494.582738 write 0x018d32 thru 0x026d71 (0xe040 bytes) 000189 1787828494.683894 mapwrite 0x03f2be thru 0x03ffff (0xd42 bytes) 000191 1787828494.745981 mapwrite 0x005f99 thru 0x005faf (0x17 bytes) 000195 1787828494.797875 trunc from 0x040000 to 0x02b6c5 (0x1493a bytes) 000196 1787828494.863934 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2b6c6): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b6c6): Mirror 4: 0xcda33042 (stale) Mirror 5: 0x2bbacc3b (stale) Mirror 6: 0x445958e1 (stale) CRC-32 checksum value for chunk [0, 0x2b6c6): Mirror 1: 0xcd235111 Mirror 7: 0xcd235111 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b6c6: skipped 000198 1787828496.167578 trunc from 0x02b6c6 to 0x002233 (0x29492 bytes) 000199 1787828496.268315 MIRROR_RESYNC 5 mirrors 000202 1787828497.581380 mapread 0x000aa4 thru 0x002233 (0x1790 bytes) 000203 1787828497.605626 write 0x03c787 thru 0x03fe7b (0x36f5 bytes) 000204 1787828497.687001 trunc from 0x03fe7c to 0x029210 (0x16c6b bytes) 000205 1787828497.717120 MIRROR_RESYNC 5 mirrors 000206 1787828498.965934 mapwrite 0x03e007 thru 0x03ffff (0x1ff9 bytes) 000208 1787828499.120387 read 0x00f85c thru 0x01d9e0 (0xe185 bytes) 000209 1787828499.136978 mapread 0x007874 thru 0x009d3c (0x24c9 bytes) 000210 1787828499.163730 write 0x036bba thru 0x03e324 (0x776b bytes) 000214 1787828499.173580 write 0x001251 thru 0x010ee0 (0xfc90 bytes) 000219 1787828499.177704 read 0x00c909 thru 0x016269 (0x9961 bytes) 000221 1787828499.179664 mapread 0x023460 thru 0x0243c3 (0xf64 bytes) 000222 1787828499.198709 write 0x0249a3 thru 0x027544 (0x2ba2 bytes) 000223 1787828499.204031 mapread 0x00ea9f thru 0x01907c (0xa5de bytes) 000224 1787828499.205267 trunc from 0x040000 to 0x022569 (0x1da96 bytes) 000225 1787828499.239940 write 0x0035c0 thru 0x00928a (0x5ccb bytes) 000226 1787828499.241866 write 0x03e33c thru 0x03ffff (0x1cc4 bytes) 000227 1787828499.243776 mapwrite 0x02a081 thru 0x02b41d (0x139d bytes) 000228 1787828499.363612 mapread 0x02cbdd thru 0x03911d (0xc541 bytes) 000229 1787828499.471575 mapread 0x00bb65 thru 0x01450d (0x89a9 bytes) 000231 1787828499.473604 read 0x01a10f thru 0x020429 (0x631b bytes) 000233 1787828499.480358 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xd8cc95e7 (stale) Mirror 6: 0xd8cc95e7 (stale) Mirror 7: 0xd8cc95e7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xd8cc95e7 (stale) Mirror 6: 0xd8cc95e7 (stale) Mirror 7: 0xd8cc95e7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000235 1787828499.939552 write 0x0099f3 thru 0x0150a7 (0xb6b5 bytes) 000240 1787828499.964270 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xd8cc95e7 (stale) Mirror 6: 0xd8cc95e7 (stale) Mirror 7: 0xd8cc95e7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xeb367acb Mirror 8: 0xeb367acb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000241 1787828501.487006 MIRROR_RESYNC 5 mirrors 000242 1787828503.165509 MIRROR_EXTEND to 6 mirrors 000244 1787828504.459864 mapwrite 0x009712 thru 0x0146b4 (0xafa3 bytes) 000246 1787828504.754387 write 0x0012fc thru 0x00a14e (0x8e53 bytes) 000253 1787828504.769326 read 0x03ce05 thru 0x03ffff (0x31fb bytes) 000254 1787828504.787605 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xeb367acb (stale) Mirror 6: 0xeb367acb (stale) Mirror 7: 0xeb367acb (stale) Mirror 8: 0xeb367acb (stale) Mirror 9: 0xeb367acb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9a70eb57 Mirror 10: 0x9a70eb57 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000255 1787828506.934500 trunc from 0x040000 to 0x01bbaa (0x24455 bytes) 000256 1787828507.105070 write 0x00a97d thru 0x013cf2 (0x9376 bytes) 000257 1787828507.119991 mapwrite 0x030174 thru 0x038cec (0x8b79 bytes) 000259 1787828507.353726 write 0x025d02 thru 0x02fa2d (0x9d2c bytes) 000261 1787828507.368381 trunc from 0x038ced to 0x03089a (0x8452 bytes) 000267 1787828507.410526 mapwrite 0x03db60 thru 0x03fe96 (0x2337 bytes) 000268 1787828507.535125 trunc from 0x03fe97 to 0x0191ab (0x26ceb bytes) 000269 1787828507.566751 trunc from 0x0191ac to 0x01487c (0x492f bytes) 000270 1787828507.617253 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1487d): Mirror 5: 0x8adad1ea (stale) Mirror 6: 0x8adad1ea (stale) Mirror 7: 0x8adad1ea (stale) Mirror 8: 0x8adad1ea (stale) Mirror 9: 0x8adad1ea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1487d): Mirror 5: 0x8adad1ea (stale) Mirror 6: 0x8adad1ea (stale) Mirror 7: 0x8adad1ea (stale) Mirror 8: 0x8adad1ea (stale) Mirror 9: 0x8adad1ea (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1487d: skipped 000271 1787828508.237901 mapread 0x010a0a thru 0x01487c (0x3e73 bytes) 000275 1787828508.259436 mapread 0x00454c thru 0x00f97c (0xb431 bytes) 000276 1787828508.278501 mapwrite 0x017a18 thru 0x0194ff (0x1ae8 bytes) 000279 1787828508.384083 MIRROR_RESYNC 6 mirrors 000280 1787828510.264859 write 0x00e5db thru 0x00eb04 (0x52a bytes) 000281 1787828510.417075 trunc from 0x019500 to 0x034092 (0x1ab93 bytes) 000282 1787828510.448039 mapread 0x033e30 thru 0x034092 (0x263 bytes) 000283 1787828510.465813 write 0x00fc0a thru 0x011b74 (0x1f6b bytes) 000284 1787828510.480264 mapread 0x00ae26 thru 0x019727 (0xe902 bytes) 000285 1787828510.510718 trunc from 0x034093 to 0x00bf08 (0x2818a bytes) 000287 1787828510.535546 trunc from 0x00bf09 to 0x029bb2 (0x1dcaa bytes) 000288 1787828510.559107 mapread 0x00518d thru 0x006040 (0xeb4 bytes) 000289 1787828510.560237 read 0x028582 thru 0x029bb2 (0x1631 bytes) 000292 1787828510.573781 mapread 0x01e997 thru 0x029bb2 (0xb21c bytes) 000293 1787828510.599549 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x29bb3): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29bb3): Mirror 5: 0xa2029c4f (stale) Mirror 6: 0xa2029c4f (stale) Mirror 7: 0xa2029c4f (stale) Mirror 8: 0xa2029c4f (stale) Mirror 9: 0xa2029c4f (stale) CRC-32 checksum value for chunk [0, 0x29bb3): Mirror 1: 0xc15e637e Mirror 10: 0xc15e637e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29bb3: skipped 000294 1787828512.286485 mapread 0x01baaf thru 0x01e3a1 (0x28f3 bytes) 000295 1787828512.319804 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x29bb3): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29bb3): Mirror 5: 0xa2029c4f (stale) Mirror 6: 0xa2029c4f (stale) Mirror 7: 0xa2029c4f (stale) Mirror 8: 0xa2029c4f (stale) Mirror 9: 0xa2029c4f (stale) CRC-32 checksum value for chunk [0, 0x29bb3): Mirror 1: 0xc15e637e Mirror 10: 0xc15e637e Mirror 11: 0xc15e637e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29bb3: skipped 000296 1787828513.862394 mapread 0x028eb3 thru 0x029bb2 (0xd00 bytes) 000298 1787828513.883519 mapwrite 0x015a67 thru 0x01f2a3 (0x983d bytes) 000302 1787828514.266000 write 0x01d373 thru 0x01f374 (0x2002 bytes) 000305 1787828514.269785 read 0x0093f3 thru 0x0164b2 (0xd0c0 bytes) 000307 1787828514.287532 read 0x01d901 thru 0x021c0b (0x430b bytes) 000308 1787828514.288672 read 0x008c09 thru 0x01537e (0xc776 bytes) 000310 1787828514.316129 MIRROR_RESYNC 8 mirrors 000311 1787828517.385977 trunc from 0x029bb3 to 0x00fd4e (0x19e64 bytes) 000313 1787828517.521109 read 0x003af3 thru 0x0084d8 (0x49e6 bytes) 000314 1787828517.534637 write 0x03c33a thru 0x03e992 (0x2659 bytes) 000315 1787828517.539739 mapwrite 0x03ecdf thru 0x03ffff (0x1321 bytes) 000317 1787828517.734478 write 0x038c21 thru 0x03ff75 (0x7355 bytes) 000319 1787828517.755457 write 0x029214 thru 0x031b81 (0x896e bytes) 000322 1787828517.796127 trunc from 0x040000 to 0x02d730 (0x128cf bytes) 000323 1787828517.818286 mapwrite 0x03c38b thru 0x03ffff (0x3c75 bytes) 000324 1787828517.930464 write 0x03cdd8 thru 0x03ffff (0x3228 bytes) 000325 1787828517.931989 mapread 0x019c9f thru 0x025143 (0xb4a5 bytes) 000326 1787828517.960526 mapread 0x00ff44 thru 0x01a887 (0xa944 bytes) 000331 1787828517.995521 MIRROR_RESYNC 8 mirrors 000332 1787828521.000497 MIRROR_EXTEND to 9 mirrors truncating to largest ever: 0x3c698 000333 1787828522.043045 trunc from 0x040000 to 0x03c697 (0x3968 bytes) 000335 1787828522.189344 mapread 0x019e80 thru 0x0273ca (0xd54b bytes) 000336 1787828522.227447 read 0x010800 thru 0x0113ea (0xbeb bytes) 000338 1787828522.235005 trunc from 0x03c698 to 0x02e659 (0xe03e bytes) 000339 1787828522.301001 trunc from 0x02e65a to 0x02db17 (0xb42 bytes) 000340 1787828522.350070 mapwrite 0x020786 thru 0x02a5e9 (0x9e64 bytes) 000341 1787828522.561202 mapread 0x01a701 thru 0x0209d7 (0x62d7 bytes) 000342 1787828522.562195 write 0x00b946 thru 0x00c8ff (0xfba bytes) 000344 1787828522.588499 read 0x00d6da thru 0x01b49c (0xddc3 bytes) 000345 1787828522.614323 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2db18): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2db18): Mirror 5: 0x6819fffd (stale) Mirror 6: 0x6819fffd (stale) Mirror 7: 0x6819fffd (stale) Mirror 8: 0x6819fffd (stale) Mirror 9: 0x6819fffd (stale) Mirror 10: 0x6819fffd (stale) Mirror 11: 0x6819fffd (stale) Mirror 12: 0x6819fffd (stale) CRC-32 checksum value for chunk [0, 0x2db18): Mirror 1: 0x584e4e1c Mirror 13: 0x584e4e1c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2db18: skipped 000346 1787828524.287516 write 0x0034ad thru 0x012489 (0xefdd bytes) 000349 1787828524.437759 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2db18): Mirror 5: 0x6819fffd (stale) Mirror 6: 0x6819fffd (stale) Mirror 8: 0x6819fffd (stale) Mirror 9: 0x6819fffd (stale) Mirror 10: 0x6819fffd (stale) Mirror 11: 0x6819fffd (stale) Mirror 12: 0x6819fffd (stale) Mirror 13: 0x584e4e1c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2db18): Mirror 5: 0x6819fffd (stale) Mirror 6: 0x6819fffd (stale) Mirror 8: 0x6819fffd (stale) Mirror 9: 0x6819fffd (stale) Mirror 10: 0x6819fffd (stale) Mirror 11: 0x6819fffd (stale) Mirror 12: 0x6819fffd (stale) Mirror 13: 0x584e4e1c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2db18: skipped 000352 1787828525.371727 read 0x024dce thru 0x02db17 (0x8d4a bytes) 000353 1787828525.394316 trunc from 0x02db18 to 0x024797 (0x9380 bytes) 000354 1787828525.435453 read 0x00cfc1 thru 0x01220d (0x524d bytes) 000356 1787828525.450381 mapread 0x01989e thru 0x024797 (0xaefa bytes) 000358 1787828525.454376 mapread 0x018720 thru 0x024797 (0xc078 bytes) 000362 1787828525.459068 mapwrite 0x011fa3 thru 0x01c21f (0xa27d bytes) 000363 1787828525.632327 trunc from 0x024798 to 0x0226b1 (0x20e6 bytes) 000365 1787828525.670798 write 0x000e36 thru 0x008bd1 (0x7d9c bytes) 000366 1787828525.676234 trunc from 0x0226b2 to 0x020e1e (0x1893 bytes) 000367 1787828525.714646 mapread 0x006f71 thru 0x012120 (0xb1b0 bytes) 000369 1787828525.716121 mapread 0x008270 thru 0x00a4cc (0x225d bytes) 000372 1787828525.721119 mapread 0x00dda1 thru 0x0127a3 (0x4a03 bytes) 000374 1787828525.725082 trunc from 0x020e1f to 0x02734e (0x6530 bytes) 000375 1787828525.751670 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2734f): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2734f): Mirror 5: 0xb7662537 (stale) Mirror 6: 0xb7662537 (stale) Mirror 8: 0xb7662537 (stale) Mirror 9: 0xb7662537 (stale) Mirror 10: 0xb7662537 (stale) Mirror 11: 0xb7662537 (stale) Mirror 12: 0xb7662537 (stale) Mirror 13: 0x5925b2ba (stale) CRC-32 checksum value for chunk [0, 0x2734f): Mirror 1: 0xed24be8a Mirror 14: 0xed24be8a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2734f: skipped 000377 1787828527.517061 mapwrite 0x01a144 thru 0x022b04 (0x89c1 bytes) 000383 1787828527.857903 mapwrite 0x038ba1 thru 0x03ffff (0x745f bytes) 000384 1787828528.048677 write 0x01a078 thru 0x0212cd (0x7256 bytes) 000385 1787828528.063172 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x8df07f57 (stale) Mirror 6: 0x8df07f57 (stale) Mirror 8: 0x8df07f57 (stale) Mirror 9: 0x8df07f57 (stale) Mirror 10: 0x8df07f57 (stale) Mirror 11: 0x8df07f57 (stale) Mirror 12: 0x8df07f57 (stale) Mirror 13: 0x490e2ead (stale) Mirror 14: 0x4854f624 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf65a6788 Mirror 15: 0xf65a6788 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000387 1787828529.816395 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x8df07f57 (stale) Mirror 6: 0x8df07f57 (stale) Mirror 8: 0x8df07f57 (stale) Mirror 9: 0x8df07f57 (stale) Mirror 10: 0x8df07f57 (stale) Mirror 12: 0x8df07f57 (stale) Mirror 13: 0x490e2ead (stale) Mirror 14: 0x4854f624 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x8df07f57 (stale) Mirror 6: 0x8df07f57 (stale) Mirror 8: 0x8df07f57 (stale) Mirror 9: 0x8df07f57 (stale) Mirror 10: 0x8df07f57 (stale) Mirror 12: 0x8df07f57 (stale) Mirror 13: 0x490e2ead (stale) Mirror 14: 0x4854f624 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf65a6788 Mirror 15: 0xf65a6788 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000388 1787828530.761908 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8df07f57 (stale) Mirror 8: 0x8df07f57 (stale) Mirror 9: 0x8df07f57 (stale) Mirror 10: 0x8df07f57 (stale) Mirror 12: 0x8df07f57 (stale) Mirror 13: 0x490e2ead (stale) Mirror 14: 0x4854f624 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8df07f57 (stale) Mirror 8: 0x8df07f57 (stale) Mirror 9: 0x8df07f57 (stale) Mirror 10: 0x8df07f57 (stale) Mirror 12: 0x8df07f57 (stale) Mirror 13: 0x490e2ead (stale) Mirror 14: 0x4854f624 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf65a6788 Mirror 15: 0xf65a6788 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000389 1787828531.728623 MIRROR_RESYNC 9 mirrors 000392 1787828534.752759 read 0x00fc30 thru 0x0163a0 (0x6771 bytes) 000393 1787828534.768678 mapread 0x00f66d thru 0x014120 (0x4ab4 bytes) 000395 1787828534.770786 read 0x02c76c thru 0x02fb09 (0x339e bytes) 000396 1787828534.786630 write 0x026568 thru 0x031ede (0xb977 bytes) 000397 1787828534.938320 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xf65a6788 (stale) Mirror 8: 0xf65a6788 (stale) Mirror 9: 0xf65a6788 (stale) Mirror 10: 0xf65a6788 (stale) Mirror 12: 0xf65a6788 (stale) Mirror 13: 0xf65a6788 (stale) Mirror 14: 0xf65a6788 (stale) Mirror 15: 0xf65a6788 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd975eef9 Mirror 16: 0xd975eef9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000402 1787828536.524880 mapread 0x0152ca thru 0x023a03 (0xe73a bytes) 000406 1787828536.557055 mapread 0x011d99 thru 0x01c262 (0xa4ca bytes) 000408 1787828536.558235 write 0x000de8 thru 0x00b96b (0xab84 bytes) 000409 1787828536.683259 read 0x00f1bd thru 0x0177f0 (0x8634 bytes) 000410 1787828536.704192 trunc from 0x040000 to 0x02fccc (0x10333 bytes) 000416 1787828536.749753 read 0x022dea thru 0x025fcd (0x31e4 bytes) 000417 1787828536.764898 trunc from 0x02fccd to 0x0316c5 (0x19f9 bytes) 000418 1787828536.814263 trunc from 0x0316c6 to 0x01a621 (0x170a4 bytes) 000419 1787828536.849707 MIRROR_RESYNC 10 mirrors 000420 1787828538.854298 read 0x008586 thru 0x00a938 (0x23b3 bytes) 000421 1787828538.866462 read 0x009f29 thru 0x00efdb (0x50b3 bytes) 000422 1787828538.884888 trunc from 0x01a622 to 0x013283 (0x739e bytes) 000423 1787828539.014999 mapread 0x007c59 thru 0x009cc8 (0x2070 bytes) 000425 1787828539.045593 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x13284): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13284): Mirror 6: 0x4e5a1993 (stale) Mirror 8: 0x4e5a1993 (stale) Mirror 9: 0x4e5a1993 (stale) Mirror 10: 0x4e5a1993 (stale) Mirror 12: 0x4e5a1993 (stale) Mirror 13: 0x4e5a1993 (stale) Mirror 14: 0x4e5a1993 (stale) Mirror 15: 0x4e5a1993 (stale) Mirror 16: 0x4e5a1993 (stale) CRC-32 checksum value for chunk [0, 0x13284): Mirror 1: 0x4e5a1993 Mirror 17: 0x4e5a1993 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13284: skipped 000427 1787828540.070204 trunc from 0x013284 to 0x00b7be (0x7ac5 bytes) 000429 1787828540.241568 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb7bf): Mirror 6: 0xfbbb113 (stale) Mirror 8: 0xfbbb113 (stale) Mirror 9: 0xfbbb113 (stale) Mirror 10: 0xfbbb113 (stale) Mirror 12: 0xfbbb113 (stale) Mirror 13: 0xfbbb113 (stale) Mirror 14: 0xfbbb113 (stale) Mirror 15: 0xfbbb113 (stale) Mirror 16: 0xfbbb113 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb7bf): Mirror 6: 0xfbbb113 (stale) Mirror 8: 0xfbbb113 (stale) Mirror 9: 0xfbbb113 (stale) Mirror 10: 0xfbbb113 (stale) Mirror 12: 0xfbbb113 (stale) Mirror 13: 0xfbbb113 (stale) Mirror 14: 0xfbbb113 (stale) Mirror 15: 0xfbbb113 (stale) Mirror 16: 0xfbbb113 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb7bf: skipped 000431 1787828540.936527 mapread 0x006fd2 thru 0x00b7be (0x47ed bytes) 000433 1787828540.971199 mapread 0x009817 thru 0x00b7be (0x1fa8 bytes) 000435 1787828540.976268 trunc from 0x00b7bf to 0x02a7e2 (0x1f024 bytes) 000436 1787828541.038011 write 0x032e83 thru 0x03ceda (0xa058 bytes) 000438 1787828541.042176 trunc from 0x03cedb to 0x003e8b (0x3904f bytes) 000439 1787828541.075201 mapwrite 0x02342e thru 0x02e180 (0xad53 bytes) 000440 1787828541.211167 mapread 0x012cb7 thru 0x01c672 (0x99bc bytes) 000441 1787828541.239310 write 0x02c717 thru 0x03bcca (0xf5b4 bytes) 000448 1787828541.263173 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3bccb): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bccb): Mirror 6: 0x2fa8cb55 (stale) Mirror 8: 0x2fa8cb55 (stale) Mirror 9: 0x2fa8cb55 (stale) Mirror 10: 0x2fa8cb55 (stale) Mirror 12: 0x2fa8cb55 (stale) Mirror 13: 0x2fa8cb55 (stale) Mirror 14: 0x2fa8cb55 (stale) Mirror 15: 0x2fa8cb55 (stale) Mirror 16: 0x2fa8cb55 (stale) CRC-32 checksum value for chunk [0, 0x3bccb): Mirror 1: 0x203574ae Mirror 17: 0x203574ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bccb: skipped 000449 1787828542.572977 write 0x03866b thru 0x03ddba (0x5750 bytes) 000450 1787828542.696116 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3ddbb): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ddbb): Mirror 6: 0x23d674ad (stale) Mirror 8: 0x23d674ad (stale) Mirror 9: 0x23d674ad (stale) Mirror 10: 0x23d674ad (stale) Mirror 12: 0x23d674ad (stale) Mirror 13: 0x23d674ad (stale) Mirror 14: 0x23d674ad (stale) Mirror 15: 0x23d674ad (stale) Mirror 16: 0x23d674ad (stale) Mirror 17: 0xb6f84a41 (stale) CRC-32 checksum value for chunk [0, 0x3ddbb): Mirror 1: 0x58ae34f6 Mirror 18: 0x58ae34f6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ddbb: skipped 000451 1787828544.238142 write 0x00e86f thru 0x019377 (0xab09 bytes) 000452 1787828544.400112 write 0x00f50e thru 0x017e04 (0x88f7 bytes) 000454 1787828544.403175 mapread 0x0181d0 thru 0x022a55 (0xa886 bytes) 000455 1787828544.447040 MIRROR_RESYNC 12 mirrors 000456 1787828549.000714 mapwrite 0x01615a thru 0x01caec (0x6993 bytes) 000457 1787828549.189888 mapread 0x01de9b thru 0x01fd43 (0x1ea9 bytes) 000458 1787828549.191119 read 0x038474 thru 0x03ddba (0x5947 bytes) 000459 1787828549.212697 mapwrite 0x019f12 thru 0x023628 (0x9717 bytes) 000464 1787828549.286364 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3ddbb): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ddbb): Mirror 6: 0xafc412de (stale) Mirror 8: 0xafc412de (stale) Mirror 9: 0xafc412de (stale) Mirror 10: 0xafc412de (stale) Mirror 12: 0xafc412de (stale) Mirror 13: 0xafc412de (stale) Mirror 14: 0xafc412de (stale) Mirror 15: 0xafc412de (stale) Mirror 16: 0xafc412de (stale) Mirror 17: 0xafc412de (stale) Mirror 18: 0xafc412de (stale) CRC-32 checksum value for chunk [0, 0x3ddbb): Mirror 1: 0xec527f5e Mirror 19: 0xec527f5e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ddbb: skipped 000465 1787828550.887455 trunc from 0x03ddbb to 0x030090 (0xdd2a bytes) 000466 1787828551.103419 read 0x0232e5 thru 0x024ef7 (0x1c13 bytes) 000467 1787828551.109481 mapread 0x02b285 thru 0x030090 (0x4e0c bytes) 000468 1787828551.132672 mapread 0x02d944 thru 0x030090 (0x274d bytes) 000469 1787828551.139685 MIRROR_SPLIT mirror 3 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30091): Mirror 6: 0x79c389f1 (stale) Mirror 8: 0x79c389f1 (stale) Mirror 10: 0x79c389f1 (stale) Mirror 12: 0x79c389f1 (stale) Mirror 13: 0x79c389f1 (stale) Mirror 14: 0x79c389f1 (stale) Mirror 15: 0x79c389f1 (stale) Mirror 16: 0x79c389f1 (stale) Mirror 17: 0x79c389f1 (stale) Mirror 18: 0x79c389f1 (stale) Mirror 19: 0xbb2311b3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30091): Mirror 6: 0x79c389f1 (stale) Mirror 8: 0x79c389f1 (stale) Mirror 10: 0x79c389f1 (stale) Mirror 12: 0x79c389f1 (stale) Mirror 13: 0x79c389f1 (stale) Mirror 14: 0x79c389f1 (stale) Mirror 15: 0x79c389f1 (stale) Mirror 16: 0x79c389f1 (stale) Mirror 17: 0x79c389f1 (stale) Mirror 18: 0x79c389f1 (stale) Mirror 19: 0xbb2311b3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30091: skipped 000472 1787828552.057750 read 0x02a363 thru 0x030090 (0x5d2e bytes) 000473 1787828552.068040 write 0x000d9c thru 0x00a405 (0x966a bytes) 000474 1787828552.083775 mapread 0x001138 thru 0x00d821 (0xc6ea bytes) 000475 1787828552.116218 mapwrite 0x02685b thru 0x02d341 (0x6ae7 bytes) 000477 1787828552.259427 MIRROR_RESYNC 12 mirrors 000478 1787828554.979695 MIRROR_EXTEND to 13 mirrors 000482 1787828556.179323 trunc from 0x030091 to 0x017774 (0x1891c bytes) 000483 1787828556.279224 write 0x020167 thru 0x027fd7 (0x7e71 bytes) 000484 1787828556.281413 mapwrite 0x0158d6 thru 0x0195e0 (0x3d0b bytes) 000486 1787828556.427673 MIRROR_SPLIT mirror 7 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27fd8): Mirror 6: 0x3272187 (stale) Mirror 8: 0x3272187 (stale) Mirror 10: 0x3272187 (stale) Mirror 12: 0x3272187 (stale) Mirror 13: 0x3272187 (stale) Mirror 14: 0x3272187 (stale) Mirror 16: 0x3272187 (stale) Mirror 17: 0x3272187 (stale) Mirror 18: 0x3272187 (stale) Mirror 19: 0x3272187 (stale) Mirror 20: 0x3272187 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27fd8): Mirror 6: 0x3272187 (stale) Mirror 8: 0x3272187 (stale) Mirror 10: 0x3272187 (stale) Mirror 12: 0x3272187 (stale) Mirror 13: 0x3272187 (stale) Mirror 14: 0x3272187 (stale) Mirror 16: 0x3272187 (stale) Mirror 17: 0x3272187 (stale) Mirror 18: 0x3272187 (stale) Mirror 19: 0x3272187 (stale) Mirror 20: 0x3272187 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27fd8: skipped 000488 1787828557.202237 trunc from 0x027fd8 to 0x01ad16 (0xd2c1 bytes) 000489 1787828557.238421 write 0x0270ee thru 0x027182 (0x95 bytes) 000490 1787828557.239556 mapwrite 0x02d584 thru 0x033419 (0x5e96 bytes) 000491 1787828557.380639 mapwrite 0x0331ca thru 0x03b240 (0x8077 bytes) 000493 1787828557.505462 MIRROR_RESYNC 12 mirrors 000494 1787828561.601937 read 0x01ee19 thru 0x027e51 (0x9039 bytes) 000495 1787828561.621697 mapwrite 0x01b9f3 thru 0x025c81 (0xa28f bytes) 000496 1787828561.944484 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3b241): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b241): Mirror 6: 0x4d13cf13 (stale) Mirror 8: 0x4d13cf13 (stale) Mirror 10: 0x4d13cf13 (stale) Mirror 12: 0x4d13cf13 (stale) Mirror 13: 0x4d13cf13 (stale) Mirror 14: 0x4d13cf13 (stale) Mirror 16: 0x4d13cf13 (stale) Mirror 17: 0x4d13cf13 (stale) Mirror 18: 0x4d13cf13 (stale) Mirror 19: 0x4d13cf13 (stale) Mirror 20: 0x4d13cf13 (stale) CRC-32 checksum value for chunk [0, 0x3b241): Mirror 1: 0x37aba9a2 Mirror 21: 0x37aba9a2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b241: skipped 000498 1787828563.365753 trunc from 0x03b241 to 0x024933 (0x1690d bytes) 000499 1787828563.467312 read 0x0073ba thru 0x016eca (0xfb11 bytes) 000500 1787828563.482161 read 0x013e4a thru 0x022b28 (0xecdf bytes) 000501 1787828563.501606 MIRROR_SPLIT mirror 5 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24934): Mirror 6: 0xca7f276 (stale) Mirror 8: 0xca7f276 (stale) Mirror 10: 0xca7f276 (stale) Mirror 12: 0xca7f276 (stale) Mirror 14: 0xca7f276 (stale) Mirror 16: 0xca7f276 (stale) Mirror 17: 0xca7f276 (stale) Mirror 18: 0xca7f276 (stale) Mirror 19: 0xca7f276 (stale) Mirror 20: 0xca7f276 (stale) Mirror 21: 0xd355f473 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24934): Mirror 6: 0xca7f276 (stale) Mirror 8: 0xca7f276 (stale) Mirror 10: 0xca7f276 (stale) Mirror 12: 0xca7f276 (stale) Mirror 14: 0xca7f276 (stale) Mirror 16: 0xca7f276 (stale) Mirror 17: 0xca7f276 (stale) Mirror 18: 0xca7f276 (stale) Mirror 19: 0xca7f276 (stale) Mirror 20: 0xca7f276 (stale) Mirror 21: 0xd355f473 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24934: skipped 000502 1787828564.190477 mapread 0x012d2b thru 0x012db5 (0x8b bytes) 000503 1787828564.197660 read 0x01cd5c thru 0x024933 (0x7bd8 bytes) 000504 1787828564.206445 write 0x02c53e thru 0x02e8b6 (0x2379 bytes) 000505 1787828564.208736 mapread 0x0112d0 thru 0x019f6e (0x8c9f bytes) 000506 1787828564.226223 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x2e8b7): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e8b7): Mirror 6: 0x4d5ae087 (stale) Mirror 8: 0x4d5ae087 (stale) Mirror 10: 0x4d5ae087 (stale) Mirror 12: 0x4d5ae087 (stale) Mirror 14: 0x4d5ae087 (stale) Mirror 16: 0x4d5ae087 (stale) Mirror 17: 0x4d5ae087 (stale) Mirror 18: 0x4d5ae087 (stale) Mirror 19: 0x4d5ae087 (stale) Mirror 20: 0x4d5ae087 (stale) Mirror 21: 0xc362777c (stale) CRC-32 checksum value for chunk [0, 0x2e8b7): Mirror 1: 0x4d9315c1 Mirror 22: 0x4d9315c1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e8b7: skipped 000507 1787828565.719427 write 0x03f923 thru 0x03ffff (0x6dd bytes) 000508 1787828565.818269 MIRROR_RESYNC 13 mirrors 000513 1787828570.502621 mapread 0x017032 thru 0x017386 (0x355 bytes) 000517 1787828570.518365 MIRROR_EXTEND to 14 mirrors 000518 1787828571.909084 read 0x011b82 thru 0x01caee (0xaf6d bytes) 000520 1787828571.924648 mapread 0x007b89 thru 0x008176 (0x5ee bytes) 000522 1787828571.942242 mapread 0x00f66a thru 0x014e65 (0x57fc bytes) 000525 1787828571.954282 MIRROR_SPLIT mirror 7 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) 000526 1787828572.467174 read 0x02c205 thru 0x0350c1 (0x8ebd bytes) 000527 1787828572.507474 MIRROR_RESYNC 13 mirrors 000530 1787828573.388316 write 0x01d654 thru 0x02028d (0x2c3a bytes) 000531 1787828573.517118 read 0x010a26 thru 0x01e97e (0xdf59 bytes) 000532 1787828573.529121 trunc from 0x040000 to 0x00a7c7 (0x35838 bytes) 000536 1787828573.562733 trunc from 0x00a7c8 to 0x03a444 (0x2fc7d bytes) 000539 1787828573.588856 write 0x03d282 thru 0x03d870 (0x5ef bytes) 000540 1787828573.590535 read 0x00c899 thru 0x00ea51 (0x21b9 bytes) 000541 1787828573.613201 MIRROR_SPLIT mirror 11 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d871): Mirror 6: 0x196cd277 (stale) Mirror 8: 0x196cd277 (stale) Mirror 10: 0x196cd277 (stale) Mirror 12: 0x196cd277 (stale) Mirror 14: 0x196cd277 (stale) Mirror 16: 0x196cd277 (stale) Mirror 18: 0x196cd277 (stale) Mirror 19: 0x196cd277 (stale) Mirror 20: 0x196cd277 (stale) Mirror 21: 0x196cd277 (stale) Mirror 23: 0x196cd277 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d871): Mirror 6: 0x196cd277 (stale) Mirror 8: 0x196cd277 (stale) Mirror 10: 0x196cd277 (stale) Mirror 12: 0x196cd277 (stale) Mirror 14: 0x196cd277 (stale) Mirror 16: 0x196cd277 (stale) Mirror 18: 0x196cd277 (stale) Mirror 19: 0x196cd277 (stale) Mirror 20: 0x196cd277 (stale) Mirror 21: 0x196cd277 (stale) Mirror 23: 0x196cd277 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d871: skipped 000542 1787828574.382215 read 0x00a30f thru 0x013bc8 (0x98ba bytes) 000543 1787828574.397393 trunc from 0x03d871 to 0x028018 (0x15858 bytes) 000544 1787828574.423171 mapwrite 0x011793 thru 0x015019 (0x3887 bytes) 000546 1787828574.561175 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x28019): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28019): Mirror 6: 0xcf32cd5a (stale) Mirror 8: 0xcf32cd5a (stale) Mirror 10: 0xcf32cd5a (stale) Mirror 12: 0xcf32cd5a (stale) Mirror 14: 0xcf32cd5a (stale) Mirror 16: 0xcf32cd5a (stale) Mirror 18: 0xcf32cd5a (stale) Mirror 19: 0xcf32cd5a (stale) Mirror 20: 0xcf32cd5a (stale) Mirror 21: 0xcf32cd5a (stale) Mirror 23: 0xcf32cd5a (stale) CRC-32 checksum value for chunk [0, 0x28019): Mirror 1: 0x7815c69d Mirror 24: 0x7815c69d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28019: skipped 000547 1787828576.222894 mapwrite 0x001fbd thru 0x00b8ff (0x9943 bytes) 000550 1787828576.518577 MIRROR_RESYNC 13 mirrors 000552 1787828580.238300 trunc from 0x028019 to 0x00464d (0x239cb bytes) 000553 1787828580.347531 MIRROR_RESYNC 13 mirrors 000555 1787828582.365054 mapread 0x003947 thru 0x00464d (0xd07 bytes) 000556 1787828582.386150 read 0x000fed thru 0x00464d (0x3661 bytes) 000557 1787828582.393032 mapread 0x001c53 thru 0x00464d (0x29fb bytes) 000558 1787828582.394061 mapread 0x004140 thru 0x00464d (0x50e bytes) 000559 1787828582.396247 mapwrite 0x01bbc6 thru 0x02abaa (0xefe5 bytes) 000564 1787828582.594454 write 0x00c35c thru 0x019525 (0xd1ca bytes) 000565 1787828582.609802 trunc from 0x02abab to 0x026ef6 (0x3cb4 bytes) 000567 1787828582.638294 write 0x010d13 thru 0x01d06d (0xc35b bytes) 000568 1787828582.641034 read 0x011690 thru 0x01b2bb (0x9c2c bytes) 000569 1787828582.642404 mapwrite 0x0283b8 thru 0x036221 (0xde6a bytes) 000570 1787828582.763180 MIRROR_RESYNC 13 mirrors 000572 1787828586.352134 trunc from 0x036222 to 0x00ffe1 (0x26240 bytes) 000573 1787828586.465929 mapread 0x00785a thru 0x00d1c7 (0x596e bytes) 000578 1787828586.497187 write 0x028a71 thru 0x0339e8 (0xaf78 bytes) 000579 1787828586.502611 trunc from 0x0339e9 to 0x025d77 (0xdc71 bytes) 000580 1787828586.541731 mapread 0x019c0b thru 0x023ab5 (0x9eab bytes) 000581 1787828586.567283 mapwrite 0x022a60 thru 0x029426 (0x69c7 bytes) 000582 1787828586.712484 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x29427): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29427): Mirror 6: 0x92a1cdc9 (stale) Mirror 8: 0x92a1cdc9 (stale) Mirror 10: 0x92a1cdc9 (stale) Mirror 12: 0x92a1cdc9 (stale) Mirror 14: 0x92a1cdc9 (stale) Mirror 16: 0x92a1cdc9 (stale) Mirror 18: 0x92a1cdc9 (stale) Mirror 19: 0x92a1cdc9 (stale) Mirror 20: 0x92a1cdc9 (stale) Mirror 21: 0x92a1cdc9 (stale) Mirror 23: 0x92a1cdc9 (stale) Mirror 24: 0x92a1cdc9 (stale) CRC-32 checksum value for chunk [0, 0x29427): Mirror 1: 0x3aaec171 Mirror 25: 0x3aaec171 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29427: skipped 000583 1787828588.206484 write 0x036fc0 thru 0x03ffff (0x9040 bytes) 000584 1787828588.291634 write 0x028726 thru 0x0326e9 (0x9fc4 bytes) 000586 1787828588.303732 trunc from 0x040000 to 0x014734 (0x2b8cb bytes) 000587 1787828588.340212 write 0x03f240 thru 0x03ffff (0xdc0 bytes) 000588 1787828588.343937 trunc from 0x040000 to 0x022981 (0x1d67e bytes) 000589 1787828588.375433 mapread 0x011fc0 thru 0x0212ec (0xf32d bytes) 000591 1787828588.424888 trunc from 0x022982 to 0x038c08 (0x16287 bytes) 000594 1787828588.461513 trunc from 0x038c09 to 0x00587a (0x3338e bytes) 000596 1787828588.508478 MIRROR_SPLIT mirror 5 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x587b): Mirror 6: 0x79eed2d0 (stale) Mirror 8: 0x79eed2d0 (stale) Mirror 10: 0x79eed2d0 (stale) Mirror 12: 0x79eed2d0 (stale) Mirror 16: 0x79eed2d0 (stale) Mirror 18: 0x79eed2d0 (stale) Mirror 19: 0x79eed2d0 (stale) Mirror 20: 0x79eed2d0 (stale) Mirror 21: 0x79eed2d0 (stale) Mirror 23: 0x79eed2d0 (stale) Mirror 24: 0x79eed2d0 (stale) Mirror 25: 0x79eed2d0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x587b): Mirror 6: 0x79eed2d0 (stale) Mirror 8: 0x79eed2d0 (stale) Mirror 10: 0x79eed2d0 (stale) Mirror 12: 0x79eed2d0 (stale) Mirror 16: 0x79eed2d0 (stale) Mirror 18: 0x79eed2d0 (stale) Mirror 19: 0x79eed2d0 (stale) Mirror 20: 0x79eed2d0 (stale) Mirror 21: 0x79eed2d0 (stale) Mirror 23: 0x79eed2d0 (stale) Mirror 24: 0x79eed2d0 (stale) Mirror 25: 0x79eed2d0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x587b: skipped 000598 1787828589.054183 MIRROR_RESYNC 13 mirrors 000600 1787828591.028275 trunc from 0x00587b to 0x00f8c7 (0xa04d bytes) 000601 1787828591.203440 mapwrite 0x017212 thru 0x020257 (0x9046 bytes) 000603 1787828591.335420 trunc from 0x020258 to 0x022ce5 (0x2a8e bytes) 000605 1787828591.374618 read 0x022233 thru 0x022ce5 (0xab3 bytes) 000606 1787828591.390458 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x22ce6): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22ce6): Mirror 6: 0x2f10d74 (stale) Mirror 8: 0x2f10d74 (stale) Mirror 10: 0x2f10d74 (stale) Mirror 12: 0x2f10d74 (stale) Mirror 16: 0x2f10d74 (stale) Mirror 18: 0x2f10d74 (stale) Mirror 19: 0x2f10d74 (stale) Mirror 20: 0x2f10d74 (stale) Mirror 21: 0x2f10d74 (stale) Mirror 23: 0x2f10d74 (stale) Mirror 24: 0x2f10d74 (stale) Mirror 25: 0x2f10d74 (stale) CRC-32 checksum value for chunk [0, 0x22ce6): Mirror 1: 0xd369757c Mirror 26: 0xd369757c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22ce6: skipped 000607 1787828592.595887 mapwrite 0x008e71 thru 0x015386 (0xc516 bytes) 000609 1787828592.906005 trunc from 0x022ce6 to 0x014a41 (0xe2a4 bytes) 000610 1787828592.957012 mapread 0x001b84 thru 0x008453 (0x68d0 bytes) 000613 1787828592.971577 MIRROR_SPLIT mirror 9 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14a42): Mirror 6: 0x4c830134 (stale) Mirror 8: 0x4c830134 (stale) Mirror 10: 0x4c830134 (stale) Mirror 12: 0x4c830134 (stale) Mirror 16: 0x4c830134 (stale) Mirror 18: 0x4c830134 (stale) Mirror 19: 0x4c830134 (stale) Mirror 20: 0x4c830134 (stale) Mirror 23: 0x4c830134 (stale) Mirror 24: 0x4c830134 (stale) Mirror 25: 0x4c830134 (stale) Mirror 26: 0x4c830134 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14a42): Mirror 6: 0x4c830134 (stale) Mirror 8: 0x4c830134 (stale) Mirror 10: 0x4c830134 (stale) Mirror 12: 0x4c830134 (stale) Mirror 16: 0x4c830134 (stale) Mirror 18: 0x4c830134 (stale) Mirror 19: 0x4c830134 (stale) Mirror 20: 0x4c830134 (stale) Mirror 23: 0x4c830134 (stale) Mirror 24: 0x4c830134 (stale) Mirror 25: 0x4c830134 (stale) Mirror 26: 0x4c830134 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14a42: skipped 000615 1787828593.716874 write 0x03b693 thru 0x03ffff (0x496d bytes) 000617 1787828593.723028 trunc from 0x040000 to 0x0209e8 (0x1f617 bytes) 000618 1787828593.739566 mapread 0x004230 thru 0x01169c (0xd46d bytes) 000619 1787828593.765053 write 0x01da38 thru 0x0286b5 (0xac7e bytes) 000623 1787828593.773456 trunc from 0x0286b6 to 0x003aa9 (0x24c0c bytes) 000624 1787828593.790630 mapread 0x0033aa thru 0x003aa9 (0x700 bytes) 000625 1787828593.792023 write 0x028513 thru 0x03331b (0xae09 bytes) 000626 1787828593.803414 MIRROR_RESYNC 13 mirrors 000627 1787828596.981043 mapwrite 0x030033 thru 0x0352fc (0x52ca bytes) 000628 1787828597.203443 write 0x00638b thru 0x011a7d (0xb6f3 bytes) 000630 1787828597.221883 read 0x03064e thru 0x0352fc (0x4caf bytes) 000631 1787828597.222508 trunc from 0x0352fd to 0x03595e (0x662 bytes) 000632 1787828597.256440 mapread 0x00044d thru 0x00c76a (0xc31e bytes) 000633 1787828597.269645 mapwrite 0x005b0d thru 0x01214f (0xc643 bytes) 000638 1787828597.408544 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x3595f): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3595f): Mirror 6: 0xc8b3cf91 (stale) Mirror 8: 0xc8b3cf91 (stale) Mirror 10: 0xc8b3cf91 (stale) Mirror 12: 0xc8b3cf91 (stale) Mirror 16: 0xc8b3cf91 (stale) Mirror 18: 0xc8b3cf91 (stale) Mirror 19: 0xc8b3cf91 (stale) Mirror 20: 0xc8b3cf91 (stale) Mirror 23: 0xc8b3cf91 (stale) Mirror 24: 0xc8b3cf91 (stale) Mirror 25: 0xc8b3cf91 (stale) Mirror 26: 0xc8b3cf91 (stale) CRC-32 checksum value for chunk [0, 0x3595f): Mirror 1: 0x322d4311 Mirror 27: 0x322d4311 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3595f: skipped 000639 1787828599.053308 MIRROR_RESYNC 14 mirrors 000640 1787828602.245081 MIRROR_EXTEND to 15 mirrors 000641 1787828603.525413 mapwrite 0x0029a1 thru 0x00755f (0x4bbf bytes) 000643 1787828603.731245 MIRROR_SPLIT mirror 2 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3595f): Mirror 6: 0x322d4311 (stale) Mirror 10: 0x322d4311 (stale) Mirror 12: 0x322d4311 (stale) Mirror 16: 0x322d4311 (stale) Mirror 18: 0x322d4311 (stale) Mirror 19: 0x322d4311 (stale) Mirror 20: 0x322d4311 (stale) Mirror 23: 0x322d4311 (stale) Mirror 24: 0x322d4311 (stale) Mirror 25: 0x322d4311 (stale) Mirror 26: 0x322d4311 (stale) Mirror 27: 0x322d4311 (stale) Mirror 28: 0x322d4311 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3595f): Mirror 6: 0x322d4311 (stale) Mirror 10: 0x322d4311 (stale) Mirror 12: 0x322d4311 (stale) Mirror 16: 0x322d4311 (stale) Mirror 18: 0x322d4311 (stale) Mirror 19: 0x322d4311 (stale) Mirror 20: 0x322d4311 (stale) Mirror 23: 0x322d4311 (stale) Mirror 24: 0x322d4311 (stale) Mirror 25: 0x322d4311 (stale) Mirror 26: 0x322d4311 (stale) Mirror 27: 0x322d4311 (stale) Mirror 28: 0x322d4311 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3595f: skipped 000644 1787828604.610801 write 0x031042 thru 0x039318 (0x82d7 bytes) 000645 1787828604.617293 mapread 0x00dc10 thru 0x00f4cb (0x18bc bytes) 000646 1787828604.645283 mapwrite 0x00e38d thru 0x00f32a (0xf9e bytes) 000648 1787828604.723139 mapwrite 0x0107b3 thru 0x0135aa (0x2df8 bytes) 000649 1787828604.768108 mapread 0x00d975 thru 0x00e1d9 (0x865 bytes) 000650 1787828604.770982 trunc from 0x039319 to 0x00f914 (0x29a04 bytes) 000652 1787828604.798855 mapwrite 0x039f4c thru 0x03ffff (0x60b4 bytes) 000655 1787828604.900742 trunc from 0x040000 to 0x006e44 (0x391bb bytes) 000656 1787828604.931362 MIRROR_RESYNC 14 mirrors 000658 1787828607.204294 write 0x02534a thru 0x02db0e (0x87c5 bytes) 000659 1787828607.283737 write 0x00f1db thru 0x01ce06 (0xdc2c bytes) 000660 1787828607.308405 read 0x01b38e thru 0x02a096 (0xed09 bytes) 000661 1787828607.319775 mapwrite 0x02e6b7 thru 0x03602f (0x7979 bytes) 000662 1787828607.473388 write 0x0134f4 thru 0x020501 (0xd00e bytes) 000663 1787828607.478353 write 0x00c1a6 thru 0x014531 (0x838c bytes) 000665 1787828607.490663 trunc from 0x036030 to 0x036a27 (0x9f8 bytes) 000667 1787828607.514239 mapread 0x0226f7 thru 0x02c31a (0x9c24 bytes) 000670 1787828607.516861 read 0x011054 thru 0x016c36 (0x5be3 bytes) 000671 1787828607.517564 read 0x00b17b thru 0x018fa1 (0xde27 bytes) 000672 1787828607.532164 read 0x0245ce thru 0x02fc8a (0xb6bd bytes) 000673 1787828607.532900 read 0x01665a thru 0x01c2e1 (0x5c88 bytes) 000676 1787828607.539116 MIRROR_RESYNC 14 mirrors 000678 1787828610.890623 write 0x034e66 thru 0x037968 (0x2b03 bytes) 000679 1787828611.019437 mapwrite 0x0080f2 thru 0x008a96 (0x9a5 bytes) 000680 1787828611.095186 mapread 0x036841 thru 0x037968 (0x1128 bytes) 000683 1787828611.098585 trunc from 0x037969 to 0x02ac6a (0xccfe bytes) 000685 1787828611.143712 read 0x028f13 thru 0x02ac6a (0x1d58 bytes) 000691 1787828611.154884 trunc from 0x02ac6b to 0x034c6d (0xa003 bytes) 000692 1787828611.173614 trunc from 0x034c6e to 0x0052b7 (0x2f9b6 bytes) 000695 1787828611.204065 MIRROR_RESYNC 14 mirrors 000696 1787828613.080259 trunc from 0x0052b8 to 0x03a9a6 (0x356ef bytes) 000698 1787828613.223951 write 0x015d72 thru 0x01f0bd (0x934c bytes) 000699 1787828613.239154 write 0x03c30f thru 0x03ffff (0x3cf1 bytes) 000701 1787828613.240804 mapwrite 0x0354b5 thru 0x03ffff (0xab4b bytes) 000702 1787828613.417215 write 0x038888 thru 0x03ffff (0x7778 bytes) 000704 1787828613.426088 mapread 0x015b5a thru 0x020575 (0xaa1c bytes) 000705 1787828613.447785 write 0x037404 thru 0x03ffff (0x8bfc bytes) 000708 1787828613.451781 trunc from 0x040000 to 0x008b1d (0x374e2 bytes) 000709 1787828613.501345 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x8b1e): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8b1e): Mirror 6: 0x50510d5 (stale) Mirror 10: 0x50510d5 (stale) Mirror 12: 0x50510d5 (stale) Mirror 16: 0x50510d5 (stale) Mirror 18: 0x50510d5 (stale) Mirror 19: 0x50510d5 (stale) Mirror 20: 0x50510d5 (stale) Mirror 23: 0x50510d5 (stale) Mirror 24: 0x50510d5 (stale) Mirror 25: 0x50510d5 (stale) Mirror 26: 0x50510d5 (stale) Mirror 27: 0x50510d5 (stale) Mirror 28: 0x50510d5 (stale) CRC-32 checksum value for chunk [0, 0x8b1e): Mirror 1: 0x50510d5 Mirror 29: 0x50510d5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8b1e: skipped 000710 1787828615.131551 read 0x0011d0 thru 0x008b1d (0x794e bytes) 000712 1787828615.142607 write 0x01cdd1 thru 0x01ce00 (0x30 bytes) 000713 1787828615.269796 trunc from 0x01ce01 to 0x020412 (0x3612 bytes) 000714 1787828615.307532 mapread 0x019d96 thru 0x020412 (0x667d bytes) 000717 1787828615.332127 MIRROR_SPLIT mirror 10 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20413): Mirror 6: 0x544d8edc (stale) Mirror 10: 0x544d8edc (stale) Mirror 12: 0x544d8edc (stale) Mirror 16: 0x544d8edc (stale) Mirror 18: 0x544d8edc (stale) Mirror 19: 0x544d8edc (stale) Mirror 20: 0x544d8edc (stale) Mirror 23: 0x544d8edc (stale) Mirror 24: 0x544d8edc (stale) Mirror 26: 0x544d8edc (stale) Mirror 27: 0x544d8edc (stale) Mirror 28: 0x544d8edc (stale) Mirror 29: 0x544d8edc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20413): Mirror 6: 0x544d8edc (stale) Mirror 10: 0x544d8edc (stale) Mirror 12: 0x544d8edc (stale) Mirror 16: 0x544d8edc (stale) Mirror 18: 0x544d8edc (stale) Mirror 19: 0x544d8edc (stale) Mirror 20: 0x544d8edc (stale) Mirror 23: 0x544d8edc (stale) Mirror 24: 0x544d8edc (stale) Mirror 26: 0x544d8edc (stale) Mirror 27: 0x544d8edc (stale) Mirror 28: 0x544d8edc (stale) Mirror 29: 0x544d8edc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20413: skipped 000719 1787828616.225002 read 0x012d40 thru 0x0149e8 (0x1ca9 bytes) 000720 1787828616.233728 write 0x03d877 thru 0x03ffff (0x2789 bytes) 000721 1787828616.235583 mapwrite 0x00ddb4 thru 0x01216f (0x43bc bytes) 000722 1787828616.316977 read 0x02298d thru 0x02b01a (0x868e bytes) 000723 1787828616.330321 trunc from 0x040000 to 0x0159d5 (0x2a62a bytes) 000724 1787828616.350981 trunc from 0x0159d6 to 0x0098c2 (0xc113 bytes) 000727 1787828616.382732 trunc from 0x0098c3 to 0x00bdc1 (0x24ff bytes) 000730 1787828616.399409 mapread 0x00081f thru 0x007878 (0x705a bytes) 000731 1787828616.401802 mapwrite 0x03d89c thru 0x03ffff (0x2764 bytes) 000734 1787828616.520671 trunc from 0x040000 to 0x0261ee (0x19e11 bytes) 000736 1787828616.540219 mapwrite 0x01d0f4 thru 0x029f84 (0xce91 bytes) 000737 1787828616.788482 MIRROR_SPLIT mirror 9 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29f85): Mirror 6: 0x8a77486e (stale) Mirror 10: 0x8a77486e (stale) Mirror 12: 0x8a77486e (stale) Mirror 16: 0x8a77486e (stale) Mirror 18: 0x8a77486e (stale) Mirror 19: 0x8a77486e (stale) Mirror 20: 0x8a77486e (stale) Mirror 23: 0x8a77486e (stale) Mirror 26: 0x8a77486e (stale) Mirror 27: 0x8a77486e (stale) Mirror 28: 0x8a77486e (stale) Mirror 29: 0x8a77486e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29f85): Mirror 6: 0x8a77486e (stale) Mirror 10: 0x8a77486e (stale) Mirror 12: 0x8a77486e (stale) Mirror 16: 0x8a77486e (stale) Mirror 18: 0x8a77486e (stale) Mirror 19: 0x8a77486e (stale) Mirror 20: 0x8a77486e (stale) Mirror 23: 0x8a77486e (stale) Mirror 26: 0x8a77486e (stale) Mirror 27: 0x8a77486e (stale) Mirror 28: 0x8a77486e (stale) Mirror 29: 0x8a77486e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29f85: skipped 000740 1787828617.591215 MIRROR_SPLIT mirror 12 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29f85): Mirror 6: 0x8a77486e (stale) Mirror 10: 0x8a77486e (stale) Mirror 12: 0x8a77486e (stale) Mirror 16: 0x8a77486e (stale) Mirror 18: 0x8a77486e (stale) Mirror 19: 0x8a77486e (stale) Mirror 20: 0x8a77486e (stale) Mirror 23: 0x8a77486e (stale) Mirror 26: 0x8a77486e (stale) Mirror 27: 0x8a77486e (stale) Mirror 28: 0x8a77486e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29f85): Mirror 6: 0x8a77486e (stale) Mirror 10: 0x8a77486e (stale) Mirror 12: 0x8a77486e (stale) Mirror 16: 0x8a77486e (stale) Mirror 18: 0x8a77486e (stale) Mirror 19: 0x8a77486e (stale) Mirror 20: 0x8a77486e (stale) Mirror 23: 0x8a77486e (stale) Mirror 26: 0x8a77486e (stale) Mirror 27: 0x8a77486e (stale) Mirror 28: 0x8a77486e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29f85: skipped 000741 1787828618.545543 trunc from 0x029f85 to 0x02bd6c (0x1de8 bytes) 000743 1787828618.569762 trunc from 0x02bd6d to 0x035f72 (0xa206 bytes) 000744 1787828618.593405 mapwrite 0x030c40 thru 0x03e183 (0xd544 bytes) 000745 1787828618.729899 trunc from 0x03e184 to 0x031eda (0xc2a9 bytes) 000746 1787828618.771495 mapwrite 0x0315b8 thru 0x03ffff (0xea48 bytes) 000747 1787828618.971357 read 0x03a9e7 thru 0x03ffff (0x5619 bytes) 000748 1787828618.989410 MIRROR_SPLIT mirror 9 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x53e6da5e (stale) Mirror 10: 0x53e6da5e (stale) Mirror 12: 0x53e6da5e (stale) Mirror 16: 0x53e6da5e (stale) Mirror 18: 0x53e6da5e (stale) Mirror 19: 0x53e6da5e (stale) Mirror 20: 0x53e6da5e (stale) Mirror 23: 0x53e6da5e (stale) Mirror 27: 0x53e6da5e (stale) Mirror 28: 0x53e6da5e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x53e6da5e (stale) Mirror 10: 0x53e6da5e (stale) Mirror 12: 0x53e6da5e (stale) Mirror 16: 0x53e6da5e (stale) Mirror 18: 0x53e6da5e (stale) Mirror 19: 0x53e6da5e (stale) Mirror 20: 0x53e6da5e (stale) Mirror 23: 0x53e6da5e (stale) Mirror 27: 0x53e6da5e (stale) Mirror 28: 0x53e6da5e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000749 1787828620.009668 mapwrite 0x0002b5 thru 0x00ac3f (0xa98b bytes) 000750 1787828620.153491 mapwrite 0x009c90 thru 0x00ba77 (0x1de8 bytes) 000751 1787828620.226551 mapwrite 0x010555 thru 0x01d84f (0xd2fb bytes) 000752 1787828620.341081 trunc from 0x040000 to 0x017fe3 (0x2801c bytes) 000753 1787828620.370967 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17fe4): Mirror 6: 0x721fbcdb (stale) Mirror 10: 0x721fbcdb (stale) Mirror 12: 0x721fbcdb (stale) Mirror 16: 0x721fbcdb (stale) Mirror 18: 0x721fbcdb (stale) Mirror 20: 0x721fbcdb (stale) Mirror 23: 0x721fbcdb (stale) Mirror 27: 0x721fbcdb (stale) Mirror 28: 0x721fbcdb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17fe4): Mirror 6: 0x721fbcdb (stale) Mirror 10: 0x721fbcdb (stale) Mirror 12: 0x721fbcdb (stale) Mirror 16: 0x721fbcdb (stale) Mirror 18: 0x721fbcdb (stale) Mirror 20: 0x721fbcdb (stale) Mirror 23: 0x721fbcdb (stale) Mirror 27: 0x721fbcdb (stale) Mirror 28: 0x721fbcdb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17fe4: skipped 000754 1787828621.053088 trunc from 0x017fe4 to 0x02c133 (0x14150 bytes) 000755 1787828621.072591 read 0x023a8e thru 0x02c133 (0x86a6 bytes) 000756 1787828621.085189 trunc from 0x02c134 to 0x005503 (0x26c30 bytes) 000757 1787828621.121949 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x5504): Mirror 6: 0xd016f524 (stale) Mirror 10: 0xd016f524 (stale) Mirror 12: 0xd016f524 (stale) Mirror 18: 0xd016f524 (stale) Mirror 20: 0xd016f524 (stale) Mirror 23: 0xd016f524 (stale) Mirror 27: 0xd016f524 (stale) Mirror 28: 0xd016f524 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5504): Mirror 6: 0xd016f524 (stale) Mirror 10: 0xd016f524 (stale) Mirror 12: 0xd016f524 (stale) Mirror 18: 0xd016f524 (stale) Mirror 20: 0xd016f524 (stale) Mirror 23: 0xd016f524 (stale) Mirror 27: 0xd016f524 (stale) Mirror 28: 0xd016f524 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5504: skipped 000758 1787828621.834277 read 0x0009b3 thru 0x005503 (0x4b51 bytes) 000759 1787828621.860172 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x5504): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5504): Mirror 6: 0xd016f524 (stale) Mirror 10: 0xd016f524 (stale) Mirror 12: 0xd016f524 (stale) Mirror 18: 0xd016f524 (stale) Mirror 20: 0xd016f524 (stale) Mirror 23: 0xd016f524 (stale) Mirror 27: 0xd016f524 (stale) Mirror 28: 0xd016f524 (stale) CRC-32 checksum value for chunk [0, 0x5504): Mirror 1: 0x2f282358 Mirror 29: 0x2f282358 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5504: skipped 000760 1787828623.127342 trunc from 0x005504 to 0x035ad6 (0x305d3 bytes) 000762 1787828623.260582 mapwrite 0x0219f6 thru 0x02625f (0x486a bytes) 000763 1787828623.389545 read 0x030a23 thru 0x035ad6 (0x50b4 bytes) 000764 1787828623.415005 read 0x02680b thru 0x03412c (0xd922 bytes) 000765 1787828623.437136 trunc from 0x035ad7 to 0x029500 (0xc5d6 bytes) 000766 1787828623.477338 mapread 0x01e86a thru 0x029500 (0xac97 bytes) 000767 1787828623.480059 trunc from 0x029501 to 0x0218c2 (0x7c3e bytes) 000768 1787828623.502502 read 0x014741 thru 0x0218c2 (0xd182 bytes) 000769 1787828623.504013 mapwrite 0x034f82 thru 0x03e434 (0x94b3 bytes) 000771 1787828623.622099 read 0x00ab39 thru 0x019b1f (0xefe7 bytes) 000772 1787828623.622755 mapread 0x0315a5 thru 0x03a864 (0x92c0 bytes) 000773 1787828623.624506 write 0x00737e thru 0x011972 (0xa5f5 bytes) 000776 1787828623.635943 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e435): Mirror 6: 0x3fe4e6c6 (stale) Mirror 10: 0x3fe4e6c6 (stale) Mirror 12: 0x3fe4e6c6 (stale) Mirror 18: 0x3fe4e6c6 (stale) Mirror 20: 0x3fe4e6c6 (stale) Mirror 23: 0x3fe4e6c6 (stale) Mirror 27: 0x3fe4e6c6 (stale) Mirror 29: 0xc41869db (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e435): Mirror 6: 0x3fe4e6c6 (stale) Mirror 10: 0x3fe4e6c6 (stale) Mirror 12: 0x3fe4e6c6 (stale) Mirror 18: 0x3fe4e6c6 (stale) Mirror 20: 0x3fe4e6c6 (stale) Mirror 23: 0x3fe4e6c6 (stale) Mirror 27: 0x3fe4e6c6 (stale) Mirror 29: 0xc41869db (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e435: skipped 000777 1787828624.306502 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e435): Mirror 6: 0x3fe4e6c6 (stale) Mirror 10: 0x3fe4e6c6 (stale) Mirror 12: 0x3fe4e6c6 (stale) Mirror 18: 0x3fe4e6c6 (stale) Mirror 23: 0x3fe4e6c6 (stale) Mirror 27: 0x3fe4e6c6 (stale) Mirror 29: 0xc41869db (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e435): Mirror 6: 0x3fe4e6c6 (stale) Mirror 10: 0x3fe4e6c6 (stale) Mirror 12: 0x3fe4e6c6 (stale) Mirror 18: 0x3fe4e6c6 (stale) Mirror 23: 0x3fe4e6c6 (stale) Mirror 27: 0x3fe4e6c6 (stale) Mirror 29: 0xc41869db (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e435: skipped 000778 1787828624.942269 write 0x03fbc2 thru 0x03ffff (0x43e bytes) 000779 1787828624.943209 mapread 0x022d02 thru 0x023e1a (0x1119 bytes) 000780 1787828624.952414 mapwrite 0x0227b4 thru 0x02e34b (0xbb98 bytes) 000781 1787828625.065417 MIRROR_RESYNC 8 mirrors 000784 1787828628.331509 read 0x01d53f thru 0x023b93 (0x6655 bytes) 000786 1787828628.343443 write 0x001c12 thru 0x0041c9 (0x25b8 bytes) 000788 1787828628.446068 trunc from 0x040000 to 0x02c177 (0x13e88 bytes) 000789 1787828628.473588 read 0x01b114 thru 0x01ee07 (0x3cf4 bytes) 000790 1787828628.491262 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2c178): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 18: (stale - read error) Mirror 23: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c178): Mirror 6: 0xbf13f04d (stale) Mirror 10: 0xbf13f04d (stale) Mirror 12: 0xbf13f04d (stale) Mirror 18: 0xbf13f04d (stale) Mirror 23: 0xbf13f04d (stale) Mirror 27: 0xbf13f04d (stale) Mirror 29: 0xbf13f04d (stale) CRC-32 checksum value for chunk [0, 0x2c178): Mirror 1: 0xc7927f47 Mirror 30: 0xc7927f47 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c178: skipped 000791 1787828629.880015 read 0x00b34a thru 0x01ad40 (0xf9f7 bytes) 000792 1787828629.891518 trunc from 0x02c178 to 0x0187f8 (0x1397f bytes) 000793 1787828630.021298 mapwrite 0x007236 thru 0x012366 (0xb131 bytes) 000796 1787828630.109439 write 0x02ab9c thru 0x0314f8 (0x695d bytes) 000799 1787828630.113375 read 0x02c9f9 thru 0x0314f8 (0x4b00 bytes) 000800 1787828630.114404 mapwrite 0x00e235 thru 0x010426 (0x21f2 bytes) 000801 1787828630.166370 mapread 0x01be6e thru 0x020c27 (0x4dba bytes) 000802 1787828630.180882 mapread 0x00ce10 thru 0x01714d (0xa33e bytes) 000803 1787828630.183440 mapwrite 0x02405f thru 0x031c45 (0xdbe7 bytes) 000807 1787828630.289390 mapwrite 0x002ff3 thru 0x00dcc9 (0xacd7 bytes) 000808 1787828630.365976 write 0x02ca4e thru 0x038b4d (0xc100 bytes) 000809 1787828630.368208 mapread 0x025fb4 thru 0x028abb (0x2b08 bytes) 000810 1787828630.373448 write 0x016404 thru 0x017aa6 (0x16a3 bytes) 000811 1787828630.378033 write 0x01be42 thru 0x020ba4 (0x4d63 bytes) 000812 1787828630.380417 mapwrite 0x007a47 thru 0x008878 (0xe32 bytes) 000813 1787828630.449420 mapwrite 0x02275f thru 0x0289a3 (0x6245 bytes) 000815 1787828630.527361 trunc from 0x038b4e to 0x000b46 (0x38007 bytes) 000816 1787828630.559279 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0xb47): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 18: (stale - read error) Mirror 23: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb47): Mirror 6: 0xf28cdebf (stale) Mirror 10: 0xf28cdebf (stale) Mirror 12: 0xf28cdebf (stale) Mirror 18: 0xf28cdebf (stale) Mirror 23: 0xf28cdebf (stale) Mirror 27: 0xf28cdebf (stale) Mirror 29: 0xf28cdebf (stale) Mirror 30: 0xf28cdebf (stale) CRC-32 checksum value for chunk [0, 0xb47): Mirror 1: 0xf28cdebf Mirror 31: 0xf28cdebf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb47: skipped 000818 1787828631.563488 MIRROR_RESYNC 10 mirrors 000820 1787828632.933682 write 0x00bee6 thru 0x0178dd (0xb9f8 bytes) 000822 1787828633.041282 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x178de): Mirror 6: 0x37a34ca3 (stale) Mirror 12: 0x37a34ca3 (stale) Mirror 18: 0x37a34ca3 (stale) Mirror 23: 0x37a34ca3 (stale) Mirror 27: 0x37a34ca3 (stale) Mirror 29: 0x37a34ca3 (stale) Mirror 30: 0x37a34ca3 (stale) Mirror 31: 0x37a34ca3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x178de): Mirror 6: 0x37a34ca3 (stale) Mirror 12: 0x37a34ca3 (stale) Mirror 18: 0x37a34ca3 (stale) Mirror 23: 0x37a34ca3 (stale) Mirror 27: 0x37a34ca3 (stale) Mirror 29: 0x37a34ca3 (stale) Mirror 30: 0x37a34ca3 (stale) Mirror 31: 0x37a34ca3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x178de: skipped 000824 1787828633.826021 read 0x00981c thru 0x014dc5 (0xb5aa bytes) 000825 1787828633.833371 mapread 0x009ede thru 0x012624 (0x8747 bytes) 000826 1787828633.836337 mapread 0x00102d thru 0x00ce88 (0xbe5c bytes) 000832 1787828633.884333 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x178de): Mirror 6: 0x37a34ca3 (stale) Mirror 12: 0x37a34ca3 (stale) Mirror 18: 0x37a34ca3 (stale) Mirror 27: 0x37a34ca3 (stale) Mirror 29: 0x37a34ca3 (stale) Mirror 30: 0x37a34ca3 (stale) Mirror 31: 0x37a34ca3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x178de): Mirror 6: 0x37a34ca3 (stale) Mirror 12: 0x37a34ca3 (stale) Mirror 18: 0x37a34ca3 (stale) Mirror 27: 0x37a34ca3 (stale) Mirror 29: 0x37a34ca3 (stale) Mirror 30: 0x37a34ca3 (stale) Mirror 31: 0x37a34ca3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x178de: skipped 000833 1787828634.531436 read 0x003a57 thru 0x00c4b7 (0x8a61 bytes) 000834 1787828634.553212 write 0x025029 thru 0x027e5b (0x2e33 bytes) 000835 1787828634.558288 read 0x01e2ca thru 0x027e5b (0x9b92 bytes) 000836 1787828634.592138 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27e5c): Mirror 6: 0x2de4987 (stale) Mirror 18: 0x2de4987 (stale) Mirror 27: 0x2de4987 (stale) Mirror 29: 0x2de4987 (stale) Mirror 30: 0x2de4987 (stale) Mirror 31: 0x2de4987 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27e5c): Mirror 6: 0x2de4987 (stale) Mirror 18: 0x2de4987 (stale) Mirror 27: 0x2de4987 (stale) Mirror 29: 0x2de4987 (stale) Mirror 30: 0x2de4987 (stale) Mirror 31: 0x2de4987 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27e5c: skipped 000837 1787828635.140006 trunc from 0x027e5c to 0x02e3dd (0x6582 bytes) 000838 1787828635.156299 write 0x030486 thru 0x03c2a2 (0xbe1d bytes) 000839 1787828635.159963 trunc from 0x03c2a3 to 0x02ca28 (0xf87a bytes) 000840 1787828635.177808 mapwrite 0x0249fe thru 0x030b1e (0xc121 bytes) 000841 1787828635.285108 trunc from 0x030b1f to 0x038f45 (0x8427 bytes) 000842 1787828635.305156 trunc from 0x038f46 to 0x025090 (0x13eb5 bytes) 000843 1787828635.328283 write 0x00e271 thru 0x01c5e5 (0xe375 bytes) 000844 1787828635.339327 mapwrite 0x000737 thru 0x00ccea (0xc5b4 bytes) 000845 1787828635.398633 mapread 0x016cd8 thru 0x024b7f (0xdea8 bytes) 000846 1787828635.407032 trunc from 0x025091 to 0x01b4a7 (0x9be9 bytes) 000847 1787828635.430150 MIRROR_RESYNC 7 mirrors 000848 1787828636.891763 write 0x036099 thru 0x03d30a (0x7272 bytes) 000849 1787828636.964932 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3d30b): Mirror 6: (stale - read error) Mirror 18: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d30b): Mirror 6: 0xa81bec52 (stale) Mirror 18: 0xa81bec52 (stale) Mirror 27: 0xa81bec52 (stale) Mirror 29: 0xa81bec52 (stale) Mirror 30: 0xa81bec52 (stale) Mirror 31: 0xa81bec52 (stale) CRC-32 checksum value for chunk [0, 0x3d30b): Mirror 1: 0x493da799 Mirror 32: 0x493da799 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d30b: skipped 000850 1787828638.466818 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d30b): Mirror 18: 0xa81bec52 (stale) Mirror 27: 0xa81bec52 (stale) Mirror 29: 0xa81bec52 (stale) Mirror 30: 0xa81bec52 (stale) Mirror 31: 0xa81bec52 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d30b): Mirror 18: 0xa81bec52 (stale) Mirror 27: 0xa81bec52 (stale) Mirror 29: 0xa81bec52 (stale) Mirror 30: 0xa81bec52 (stale) Mirror 31: 0xa81bec52 (stale) CRC-32 checksum value for chunk [0, 0x3d30b): Mirror 1: 0x493da799 Mirror 32: 0x493da799 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d30b: skipped truncating to largest ever: 0x3fb88 000851 1787828639.245814 trunc from 0x03d30b to 0x03fb87 (0x287d bytes) 000852 1787828639.379790 mapread 0x03f829 thru 0x03fb87 (0x35f bytes) 000856 1787828639.395233 trunc from 0x03fb88 to 0x02f0ef (0x10a98 bytes) 000859 1787828639.443498 trunc from 0x02f0f0 to 0x038b16 (0x9a27 bytes) 000860 1787828639.478058 write 0x023e36 thru 0x029066 (0x5231 bytes) 000861 1787828639.512238 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x38b17): Mirror 18: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38b17): Mirror 18: 0x62d1a70d (stale) Mirror 27: 0x62d1a70d (stale) Mirror 29: 0x62d1a70d (stale) Mirror 30: 0x62d1a70d (stale) Mirror 31: 0x62d1a70d (stale) Mirror 32: 0xbca356f6 (stale) CRC-32 checksum value for chunk [0, 0x38b17): Mirror 1: 0x5cba4404 Mirror 33: 0x5cba4404 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38b17: skipped 000862 1787828641.188584 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38b17): Mirror 18: 0x62d1a70d (stale) Mirror 27: 0x62d1a70d (stale) Mirror 29: 0x62d1a70d (stale) Mirror 31: 0x62d1a70d (stale) Mirror 32: 0xbca356f6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38b17): Mirror 18: 0x62d1a70d (stale) Mirror 27: 0x62d1a70d (stale) Mirror 29: 0x62d1a70d (stale) Mirror 31: 0x62d1a70d (stale) Mirror 32: 0xbca356f6 (stale) CRC-32 checksum value for chunk [0, 0x38b17): Mirror 1: 0x5cba4404 Mirror 33: 0x5cba4404 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38b17: skipped 000863 1787828642.065415 write 0x02923b thru 0x03738d (0xe153 bytes) 000865 1787828642.192008 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x38b17): Mirror 18: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38b17): Mirror 18: 0x62d1a70d (stale) Mirror 27: 0x62d1a70d (stale) Mirror 29: 0x62d1a70d (stale) Mirror 31: 0x62d1a70d (stale) Mirror 32: 0xbca356f6 (stale) Mirror 33: 0x5cba4404 (stale) CRC-32 checksum value for chunk [0, 0x38b17): Mirror 1: 0xfc73c699 Mirror 34: 0xfc73c699 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38b17: skipped 000867 1787828643.655141 read 0x028060 thru 0x035ef0 (0xde91 bytes) 000869 1787828643.669588 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x38b17): Mirror 18: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38b17): Mirror 18: 0x62d1a70d (stale) Mirror 27: 0x62d1a70d (stale) Mirror 29: 0x62d1a70d (stale) Mirror 31: 0x62d1a70d (stale) Mirror 32: 0xbca356f6 (stale) Mirror 33: 0x5cba4404 (stale) CRC-32 checksum value for chunk [0, 0x38b17): Mirror 1: 0xfc73c699 Mirror 34: 0xfc73c699 Mirror 35: 0xfc73c699 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38b17: skipped 000872 1787828645.055100 trunc from 0x038b17 to 0x0126b9 (0x2645d bytes) 000873 1787828645.152316 MIRROR_RESYNC 9 mirrors 000874 1787828647.167194 write 0x02179d thru 0x0245e2 (0x2e46 bytes) 000875 1787828647.272013 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x245e3): Mirror 18: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x245e3): Mirror 18: 0x94c4e18e (stale) Mirror 27: 0x94c4e18e (stale) Mirror 29: 0x94c4e18e (stale) Mirror 31: 0x94c4e18e (stale) Mirror 32: 0x94c4e18e (stale) Mirror 33: 0x94c4e18e (stale) Mirror 34: 0x94c4e18e (stale) Mirror 35: 0x94c4e18e (stale) CRC-32 checksum value for chunk [0, 0x245e3): Mirror 1: 0xe1394d42 Mirror 36: 0xe1394d42 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x245e3: skipped 000876 1787828648.747122 write 0x011407 thru 0x01c9f8 (0xb5f2 bytes) 000881 1787828648.875089 trunc from 0x0245e3 to 0x0080df (0x1c503 bytes) 000883 1787828648.901903 mapwrite 0x03e2bf thru 0x03ffff (0x1d41 bytes) 000884 1787828648.992058 trunc from 0x040000 to 0x00c7d3 (0x3382c bytes) 000885 1787828649.024295 read 0x00c03b thru 0x00c7d3 (0x799 bytes) 000886 1787828649.033640 read 0x00a6e9 thru 0x00c7d3 (0x20eb bytes) 000887 1787828649.042377 mapwrite 0x02288d thru 0x031d03 (0xf477 bytes) 000889 1787828649.169486 mapread 0x014626 thru 0x02446a (0xfe45 bytes) 000890 1787828649.171938 trunc from 0x031d04 to 0x03e79a (0xca97 bytes) 000891 1787828649.191267 mapread 0x035ff2 thru 0x03bab1 (0x5ac0 bytes) 000892 1787828649.201779 mapwrite 0x01cc52 thru 0x02998b (0xcd3a bytes) 000895 1787828649.266889 MIRROR_RESYNC 10 mirrors 000899 1787828652.712972 read 0x028343 thru 0x02d4d3 (0x5191 bytes) 000900 1787828652.724270 mapwrite 0x02059e thru 0x029cc4 (0x9727 bytes) 000901 1787828652.972392 mapread 0x02523a thru 0x0260a4 (0xe6b bytes) 000902 1787828652.974164 mapwrite 0x00a358 thru 0x0100cc (0x5d75 bytes) 000904 1787828653.054667 read 0x038971 thru 0x03e79a (0x5e2a bytes) 000906 1787828653.079345 mapread 0x01f9b8 thru 0x02f8c9 (0xff12 bytes) 000907 1787828653.081370 mapread 0x01f232 thru 0x029b02 (0xa8d1 bytes) 000909 1787828653.085092 trunc from 0x03e79b to 0x024be3 (0x19bb7 bytes) 000910 1787828653.124470 write 0x00ecd5 thru 0x0121d4 (0x3500 bytes) 000911 1787828653.126135 write 0x01295f thru 0x0147ed (0x1e8f bytes) 000917 1787828653.132082 read 0x01d0d9 thru 0x024be3 (0x7b0b bytes) 000918 1787828653.140219 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24be4): Mirror 18: 0xbb3c3ebc (stale) Mirror 27: 0xbb3c3ebc (stale) Mirror 29: 0xbb3c3ebc (stale) Mirror 31: 0xbb3c3ebc (stale) Mirror 32: 0xbb3c3ebc (stale) Mirror 33: 0xbb3c3ebc (stale) Mirror 35: 0xbb3c3ebc (stale) Mirror 36: 0xbb3c3ebc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24be4): Mirror 18: 0xbb3c3ebc (stale) Mirror 27: 0xbb3c3ebc (stale) Mirror 29: 0xbb3c3ebc (stale) Mirror 31: 0xbb3c3ebc (stale) Mirror 32: 0xbb3c3ebc (stale) Mirror 33: 0xbb3c3ebc (stale) Mirror 35: 0xbb3c3ebc (stale) Mirror 36: 0xbb3c3ebc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24be4: skipped 000920 1787828653.741662 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24be4): Mirror 18: 0xbb3c3ebc (stale) Mirror 29: 0xbb3c3ebc (stale) Mirror 31: 0xbb3c3ebc (stale) Mirror 32: 0xbb3c3ebc (stale) Mirror 33: 0xbb3c3ebc (stale) Mirror 35: 0xbb3c3ebc (stale) Mirror 36: 0xbb3c3ebc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24be4): Mirror 18: 0xbb3c3ebc (stale) Mirror 29: 0xbb3c3ebc (stale) Mirror 31: 0xbb3c3ebc (stale) Mirror 32: 0xbb3c3ebc (stale) Mirror 33: 0xbb3c3ebc (stale) Mirror 35: 0xbb3c3ebc (stale) Mirror 36: 0xbb3c3ebc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24be4: skipped 000921 1787828654.464399 mapwrite 0x00bb95 thru 0x01057c (0x49e8 bytes) 000922 1787828654.641187 write 0x00e4ba thru 0x013480 (0x4fc7 bytes) 000923 1787828654.649018 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x24be4): Mirror 18: (stale - read error) Mirror 29: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24be4): Mirror 18: 0xbb3c3ebc (stale) Mirror 29: 0xbb3c3ebc (stale) Mirror 31: 0xbb3c3ebc (stale) Mirror 32: 0xbb3c3ebc (stale) Mirror 33: 0xbb3c3ebc (stale) Mirror 35: 0xbb3c3ebc (stale) Mirror 36: 0xbb3c3ebc (stale) CRC-32 checksum value for chunk [0, 0x24be4): Mirror 1: 0x3ad85487 Mirror 37: 0x3ad85487 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24be4: skipped 000924 1787828656.520239 trunc from 0x024be4 to 0x0040e0 (0x20b03 bytes) 000927 1787828656.656679 MIRROR_RESYNC 9 mirrors 000928 1787828658.660209 mapread 0x002f0a thru 0x0040e0 (0x11d7 bytes) 000929 1787828658.689859 trunc from 0x0040e1 to 0x03f41b (0x3b33b bytes) 000930 1787828658.911422 read 0x004b8a thru 0x00629e (0x1715 bytes) 000932 1787828658.927391 read 0x01f08e thru 0x02ceaa (0xde1d bytes) 000933 1787828658.963452 write 0x01112a thru 0x015318 (0x41ef bytes) 000936 1787828658.968217 read 0x025471 thru 0x02721a (0x1daa bytes) 000937 1787828658.968795 read 0x02e3bd thru 0x03bfc5 (0xdc09 bytes) 000939 1787828658.969897 mapwrite 0x019d43 thru 0x021b86 (0x7e44 bytes) 000942 1787828659.059574 mapwrite 0x01eca7 thru 0x01f819 (0xb73 bytes) 000948 1787828659.178662 write 0x02e08d thru 0x03a90b (0xc87f bytes) 000950 1787828659.182587 mapwrite 0x003e8e thru 0x00412f (0x2a2 bytes) 000955 1787828659.285910 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3f41c): Mirror 18: (stale - read error) Mirror 29: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) Mirror 37: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f41c): Mirror 18: 0xdb02d62c (stale) Mirror 29: 0xdb02d62c (stale) Mirror 31: 0xdb02d62c (stale) Mirror 32: 0xdb02d62c (stale) Mirror 33: 0xdb02d62c (stale) Mirror 35: 0xdb02d62c (stale) Mirror 36: 0xdb02d62c (stale) Mirror 37: 0xdb02d62c (stale) CRC-32 checksum value for chunk [0, 0x3f41c): Mirror 1: 0x5f520101 Mirror 38: 0x5f520101 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f41c: skipped 000957 1787828661.012659 read 0x03c95c thru 0x03f41b (0x2ac0 bytes) 000958 1787828661.022271 mapread 0x01eaac thru 0x01fb21 (0x1076 bytes) 000959 1787828661.040567 MIRROR_RESYNC 10 mirrors 000960 1787828665.019572 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) 000961 1787828665.667608 mapread 0x0266df thru 0x0358c8 (0xf1ea bytes) 000962 1787828665.730420 write 0x002ce2 thru 0x00b7b3 (0x8ad2 bytes) 000963 1787828665.901405 trunc from 0x03f41c to 0x01f2b7 (0x20164 bytes) 000964 1787828665.932571 write 0x03eaae thru 0x03ff3c (0x148f bytes) 000966 1787828665.935724 write 0x0231f0 thru 0x02ab57 (0x7968 bytes) 000968 1787828665.951724 mapread 0x026ea1 thru 0x02fb6f (0x8ccf bytes) 000969 1787828665.966718 trunc from 0x03ff3d to 0x00e300 (0x31c3c bytes) 000971 1787828666.000073 mapwrite 0x02188e thru 0x027c52 (0x63c5 bytes) 000972 1787828666.141508 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27c53): Mirror 29: 0x2532bb8b (stale) Mirror 31: 0x2532bb8b (stale) Mirror 32: 0x2532bb8b (stale) Mirror 33: 0x2532bb8b (stale) Mirror 36: 0x2532bb8b (stale) Mirror 37: 0x2532bb8b (stale) Mirror 38: 0x2532bb8b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27c53): Mirror 29: 0x2532bb8b (stale) Mirror 31: 0x2532bb8b (stale) Mirror 32: 0x2532bb8b (stale) Mirror 33: 0x2532bb8b (stale) Mirror 36: 0x2532bb8b (stale) Mirror 37: 0x2532bb8b (stale) Mirror 38: 0x2532bb8b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27c53: skipped 000974 1787828666.932963 mapwrite 0x027162 thru 0x02f6d8 (0x8577 bytes) 000975 1787828667.043082 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2f6d9): Mirror 29: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 36: (stale - read error) Mirror 37: (stale - read error) Mirror 38: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f6d9): Mirror 29: 0x5cb9e012 (stale) Mirror 31: 0x5cb9e012 (stale) Mirror 32: 0x5cb9e012 (stale) Mirror 33: 0x5cb9e012 (stale) Mirror 36: 0x5cb9e012 (stale) Mirror 37: 0x5cb9e012 (stale) Mirror 38: 0x5cb9e012 (stale) CRC-32 checksum value for chunk [0, 0x2f6d9): Mirror 1: 0x2f970445 Mirror 39: 0x2f970445 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f6d9: skipped 000980 1787828668.334493 mapread 0x00cb53 thru 0x0195fa (0xcaa8 bytes) 000982 1787828668.368265 mapread 0x022679 thru 0x02cc7f (0xa607 bytes) 000983 1787828668.386442 read 0x021bd8 thru 0x0296fd (0x7b26 bytes) 000984 1787828668.387082 trunc from 0x02f6d9 to 0x02e148 (0x1590 bytes) 000985 1787828668.525543 mapread 0x02d024 thru 0x02db11 (0xaee bytes) 000987 1787828668.538326 trunc from 0x02e149 to 0x0000ce (0x2e07a bytes) 000988 1787828668.566254 trunc from 0x0000cf to 0x0103e4 (0x10316 bytes) 000989 1787828668.614377 mapwrite 0x031316 thru 0x039c57 (0x8942 bytes) 000990 1787828668.830327 read 0x0156a2 thru 0x01f04f (0x99ae bytes) 000996 1787828668.874053 trunc from 0x039c58 to 0x033c71 (0x5fe6 bytes) 000997 1787828668.932173 MIRROR_RESYNC 9 mirrors 001000 1787828672.211115 trunc from 0x033c72 to 0x00b1d8 (0x28a99 bytes) All operations completed A-OK! PASS 70a (201s) == sanity-flr test 71: check mirror extend parent fid ==== 07:04:38 (1787828678) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 07:04:42 (1787828682) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.284757 s, 7.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x301:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132736 s, 7.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x301:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x301:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 07:04:52 (1787828692) Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre2 Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8025376 .. Extending file size to 5173792 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6968640 .. failed Extending file size to 8970240 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8901920 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9175296 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9054272 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 58817 58819 58820 58822 58823 done 192.168.204.128@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg428-client.virtnet /mnt/lustre2 (opts:) 192.168.204.128@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg428-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 07:06:11 (1787828771) Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre2 Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7768384 .. Extending file size to 1441824 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5008640 .. failed Extending file size to 6321824 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7386560 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7722048 .. Extending file size to 8633888 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8681408 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8697664 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9257440 .. Extending file size to 8986848 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 60617 60618 60619 60621 60622 done 192.168.204.128@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg428-client.virtnet /mnt/lustre2 (opts:) 192.168.204.128@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg428-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 07:07:30 (1787828850) Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3072 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3072 2205440 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104712 s, 20.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.241379 s, 8.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x75e8e228 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.204.128@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg428-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 07:07:50 (1787828870) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.112252 s, 18.7 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 07:07:56 (1787828876) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x205:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x301:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x301:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x302:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x31d:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 07:08:03 (1787828883) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.19972 s, 10.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x303:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 07:08:11 (1787828891) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.157396 s, 13.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x304:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x305:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x320:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x304:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x305:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x320:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x321:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 07:08:20 (1787828900) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.415282 s, 5.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x306:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x322:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x306:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x322:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x323:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 07:08:54 (1787828934) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg428-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129692 s, 8.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x307:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x325:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x324:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x307:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x325:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x324:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } PASS 204d (62s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 07:09:56 (1787828996) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg428-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x326:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x327:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.362172 s, 5.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x326:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x327:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x309:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x326:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x327:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x309:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (61s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 07:10:57 (1787829057) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg428-server mds-ost sync done. Creating new pool test_204f oleg428-server: Pool lustre.test_204f created Adding targets to pool oleg428-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg428-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.193959 s, 10.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x328:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x328:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg428-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg428-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg428-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 07:12:17 (1787829137) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 07:12:24 (1787829144) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 07:12:31 (1787829151) oleg428-server: Pool lustre.test_206 created Adding targets to pool oleg428-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg428-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg428-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg428-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg428-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 07:12:57 (1787829177) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157694 s, 6.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876825 s, 12.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x7b8fc125 Mirror 3: 0x7b8fc125 PASS 207 (14s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 07:13:11 (1787829191) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 07:13:16 (1787829196) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 07:13:20 (1787829200) Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.157489 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1ee0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x23a0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x23a0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x23a0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x16a0000 Read this (4 bytes): BBBB 192.168.204.128@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg428-client.virtnet /mnt/lustre2 (opts:) PASS 209a (41s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 07:14:00 (1787829240) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0280873 s, 292 kB/s the very first read read with warm cache PASS 209b (10s) == sanity-flr test 210a: handle broken mirrored lovea ==== 07:14:10 (1787829250) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536536 s, 19.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x22a lmm_fid: [0x200000404:0x22a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x22a lmm_fid: [0x200000404:0x22a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x315:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834474 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 07:14:11.000000000 -0400 Modify: 2026-08-27 07:14:12.000000000 -0400 Change: 2026-08-27 07:14:14.000000000 -0400 Birth: 2026-08-27 07:14:12.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 07:14:20 (1787829260) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 07:14:23 (1787829263) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.908481 s, 45.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0636069 s, 64.4 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 07:14 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22c:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 07:14 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 07:14:32 (1787829272) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 07:14:40 (1787829280) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (13s) == sanity-flr test complete, duration 1945 sec =========== 07:14:53 (1787829293) === sanity-flr: start cleanup 07:14:55 (1787829295) === === sanity-flr: finish cleanup 07:15:05 (1787829305) ===