-----============= acceptance-small: sanity-flr ============----- Fri Jun 26 09:31:34 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg409-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 09:31:57 (1782480717) === oleg409-client.virtnet: executing check_config_client /mnt/lustre oleg409-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg409-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8f0b4bb4d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8f0b4bb4d800.idle_timeout=debug disable quota as required oleg409-server: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 09:32:37 (1782480757) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7613] == sanity-flr test 0a: lfs mirror create with -N option == 09:32:40 (1782480760) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:33:01 (1782480781) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:33:04 (1782480784) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:33:08 (1782480788) PASS 0d (27s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:33:35 (1782480815) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:33:38 (1782480818) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 09:33:42 (1782480822) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8f0b4bb4d800.stats=clear osc.lustre-OST0001-osc-ffff8f0b4bb4d800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:33:50 (1782480830) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:34:01 (1782480841) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:34:10 (1782480850) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 09:34:13 (1782480853) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:34:21 (1782480861) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 09:34:30 (1782480870) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:34:37 (1782480877) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:34:40 (1782480880) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:34:46 (1782480886) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:34:56 (1782480896) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:35:03 (1782480903) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.696587 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147241 s, 7.1 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:35:12 (1782480912) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0609069 s, 34.4 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 09:35:19 (1782480919) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:35:29 (1782480929) Stopping /mnt/lustre-ost1 (opts:) on oleg409-server oleg409-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f0b4bb4d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f0b4bb4d800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg409-server: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg409-server: oleg409-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg409-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg409-server oleg409-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f0b4bb4d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f0b4bb4d800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg409-server: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg409-server: oleg409-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg409-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (68s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:36:37 (1782480997) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.196223 s, 16.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg409-server oleg409-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f0b4bb4d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f0b4bb4d800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg409-server: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg409-server: oleg409-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg409-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (39s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:37:17 (1782481037) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg409-server oleg409-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f0b4bb4d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f0b4bb4d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg409-server: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg409-server: oleg409-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg409-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg409-server oleg409-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f0b4bb4d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f0b4bb4d800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg409-server: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg409-server: oleg409-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg409-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (77s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:38:34 (1782481114) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.57353 s, 11.5 MB/s -rw-r--r-- 1 root root 52428800 Jun 26 09:38 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.086s user 0m0.000s sys 0m0.178s Stopping /mnt/lustre-ost1 (opts:) on oleg409-server oleg409-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f0b4bb4d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f0b4bb4d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.987s user 0m0.001s sys 0m0.161s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg409-server: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg409-server: oleg409-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg409-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg409-server oleg409-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f0b4bb4d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f0b4bb4d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.179s user 0m0.001s sys 0m0.195s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg409-server: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg409-server: oleg409-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg409-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg409-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f0b4bb4d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f0b4bb4d800.ost_server_uuid in FULL state after 0 sec PASS 33b (90s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:40:04 (1782481204) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:40:08 (1782481208) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:40:11 (1782481211) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 09:40:14 (1782481214) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 09:40:22 (1782481222) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.490604 s, 25.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:40:33 (1782481233) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720212 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350791 s, 29.9 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:40:41 (1782481241) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:40:49 (1782481249) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000947325 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00152715 s, 671 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00082693 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000782658 s, 1.3 MB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 09:41:00 (1782481260) create test files with size 15747601 .. d8d403e2565c7c755031a1712646bf63 - 15ed5d2ba2a93b7a9835430ec07ba5db - 1458851fc2fa9e088de932f02b335efb - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg409-client.virtnet /mnt/lustre (opts:) Starting client: oleg409-client.virtnet: -o user_xattr,flock 192.168.204.109@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 09:41:24 (1782481284) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:41:39 (1782481299) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:41:46 (1782481306) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187251 s, 5.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xca:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xc8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579618 s, 18.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xc9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xca:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 09:41:56 (1782481316) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.180808 s, 11.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.472478 s, 8.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.188569 s, 11.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 09:42:12 (1782481332) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 09:42:16 (1782481336) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:42:20 (1782481340) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373971 s, 28.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 09:42:28 (1782481348) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:42:31 (1782481351) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd1:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd3:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd3:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:42:40 (1782481360) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.35319 s, 29.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg409-client.virtnet /mnt/lustre (opts:) Starting client: oleg409-client.virtnet: -o user_xattr,flock 192.168.204.109@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:42:52 (1782481372) llite.lustre-ffff8f0b43120000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.337693 s, 31.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg409-client.virtnet /mnt/lustre (opts:) Starting client: oleg409-client.virtnet: -o user_xattr,flock 192.168.204.109@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8f0b45062800.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:43:07 (1782481387) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:43:45 (1782481425) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (34s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:44:19 (1782481459) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 09:44:22 (1782481462) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.209807 s, 15.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.140429 s, 22.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xa776d545 (nosync,stale) Mirror 2: 0xa776d545 (nosync,stale) ** verify mirror 2 doesn't change original checksum: c1c2f07c2cdbc28a35dd7009b8d67bb3 - mirror 1 checksum: 6f1400ead4cd66ef92460a4058d319ea - mirror 2 checksum: c1c2f07c2cdbc28a35dd7009b8d67bb3 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-26T09:44:24-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-26T09:44:24-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe9:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: c1c2f07c2cdbc28a35dd7009b8d67bb3 - mirror 1 checksum: 6f1400ead4cd66ef92460a4058d319ea - mirror 2 checksum: 6f1400ead4cd66ef92460a4058d319ea - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-26T09:44:24-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-26T09:44:24-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe9:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:44:33 (1782481473) Failing mds1 on oleg409-server Stopping /mnt/lustre-mds1 (opts:) on oleg409-server 09:44:39 (1782481479) shut down facet: mds1 facet_host: oleg409-server facet_failover_host: oleg409-server Failover mds1 to oleg409-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 09:45:11 (1782481511) targets are mounted 09:45:11 (1782481511) facet_failover done oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350523 s, 29.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xea:0x0] } PASS 50A (52s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:45:25 (1782481525) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:45:28 (1782481528) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:45:31 (1782481531) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0156494 s, 1.0 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:45:38 (1782481538) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:45:41 (1782481541) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:45:44 (1782481544) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (18s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:46:02 (1782481562) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:46:29 (1782481589) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 09:46:51 (1782481611) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 09:47:14 (1782481634) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (5s) == sanity-flr test 70: mirror create and split race ====== 09:47:19 (1782481639) mirror create pid 53747 mirror split pid 53748 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7768: 53748 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53747 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 09:48:25 (1782481705) Seed set to 1707 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x23c02 000002 1782481707.383151 trunc from 00000000 to 0x023c01 (0x23c02 bytes) 000004 1782481707.400424 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x3a0c5 000005 1782481707.872878 trunc from 0x023c02 to 0x03a0c4 (0x164c3 bytes) 000006 1782481707.990368 mapwrite 0x024691 thru 0x02cd7f (0x86ef bytes) 000007 1782481708.075964 trunc from 0x03a0c5 to 0x00d081 (0x2d043 bytes) 000008 1782481708.105332 mapwrite 0x03c740 thru 0x03e485 (0x1d46 bytes) 000009 1782481708.275086 write 0x004560 thru 0x009300 (0x4da1 bytes) 000010 1782481708.292643 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) 000011 1782481708.534867 read 0x02cd19 thru 0x03135e (0x4646 bytes) 000012 1782481708.546000 mapread 0x00b5f7 thru 0x01855d (0xcf67 bytes) 000014 1782481708.572706 read 0x03cda1 thru 0x03e485 (0x16e5 bytes) 000015 1782481708.583308 mapread 0x03cf4f thru 0x03e485 (0x1537 bytes) 000017 1782481708.586072 write 0x0338a1 thru 0x0377c9 (0x3f29 bytes) 000019 1782481708.588184 trunc from 0x03e486 to 0x017eeb (0x2659a bytes) 000021 1782481708.606561 write 0x007d93 thru 0x014403 (0xc671 bytes) 000022 1782481708.609042 trunc from 0x017eec to 0x002b12 (0x153d9 bytes) 000023 1782481708.624937 write 0x022d60 thru 0x026683 (0x3924 bytes) 000025 1782481708.627004 read 0x002667 thru 0x010fe5 (0xe97f bytes) 000026 1782481708.635624 mapread 0x00a1fe thru 0x00c754 (0x2557 bytes) 000028 1782481708.638990 mapwrite 0x00f357 thru 0x019a51 (0xa6fb bytes) truncating to largest ever: 0x3ee25 000030 1782481708.718183 trunc from 0x026684 to 0x03ee24 (0x187a1 bytes) 000031 1782481708.740481 read 0x03687b thru 0x03a919 (0x409f bytes) 000032 1782481708.753267 write 0x00c3f0 thru 0x014050 (0x7c61 bytes) 000035 1782481708.758592 mapread 0x00dbc5 thru 0x016cf9 (0x9135 bytes) 000036 1782481708.760077 read 0x017f53 thru 0x018457 (0x505 bytes) 000037 1782481708.760922 read 0x009476 thru 0x017389 (0xdf14 bytes) 000040 1782481708.762745 read 0x031767 thru 0x0336f8 (0x1f92 bytes) 000042 1782481708.763949 trunc from 0x03ee25 to 0x0319e1 (0xd443 bytes) 000045 1782481708.786474 read 0x030f86 thru 0x0319e1 (0xa5c bytes) 000047 1782481708.788089 read 0x02cef6 thru 0x0319e1 (0x4aec bytes) 000048 1782481708.789167 write 0x009490 thru 0x0163d6 (0xcf47 bytes) 000049 1782481708.791445 mapwrite 0x037ff1 thru 0x03dafc (0x5b0c bytes) 000054 1782481708.895541 mapwrite 0x02ed05 thru 0x03b71e (0xca1a bytes) 000055 1782481708.944419 read 0x00cffc thru 0x01c8ea (0xf8ef bytes) 000056 1782481708.947422 trunc from 0x03dafd to 0x015cd0 (0x27e2c bytes) 000057 1782481708.976983 mapread 0x002138 thru 0x004737 (0x2600 bytes) 000058 1782481708.981967 mapwrite 0x0234e1 thru 0x02fddd (0xc8fd bytes) 000064 1782481709.083456 write 0x02d052 thru 0x02d10d (0xbc bytes) 000071 1782481709.092198 trunc from 0x02fdde to 0x013064 (0x1cd79 bytes) 000072 1782481709.112914 mapwrite 0x03831d thru 0x03ce9a (0x4b7e bytes) 000073 1782481709.237371 mapread 0x001985 thru 0x01057e (0xebfa bytes) 000075 1782481709.244133 write 0x025f08 thru 0x02bd53 (0x5e4c bytes) 000076 1782481709.261378 MIRROR_EXTEND to 2 mirrors 000077 1782481709.830078 mapwrite 0x01f7d5 thru 0x02405b (0x4887 bytes) 000079 1782481709.994991 read 0x00f425 thru 0x011d0a (0x28e6 bytes) 000082 1782481710.010300 MIRROR_RESYNC 2 mirrors 000083 1782481710.664108 write 0x02b69a thru 0x034867 (0x91ce bytes) 000086 1782481710.724646 mapwrite 0x000c35 thru 0x007066 (0x6432 bytes) 000088 1782481710.790060 mapwrite 0x03ea57 thru 0x03ffff (0x15a9 bytes) 000090 1782481710.848941 read 0x00a44b thru 0x010029 (0x5bdf bytes) 000091 1782481710.850093 trunc from 0x040000 to 0x03e9a1 (0x165e bytes) 000093 1782481710.864061 mapread 0x014eae thru 0x01be7d (0x6fd0 bytes) 000094 1782481710.872418 mapread 0x033891 thru 0x0364d7 (0x2c47 bytes) 000095 1782481710.877432 mapwrite 0x021fd9 thru 0x028da3 (0x6dcb bytes) 000096 1782481710.928727 mapwrite 0x02a776 thru 0x0346a4 (0x9f2f bytes) 000097 1782481710.983354 mapread 0x016e71 thru 0x020816 (0x99a6 bytes) 000099 1782481710.987306 write 0x01de88 thru 0x02c6c2 (0xe83b bytes) 000101 1782481710.999126 MIRROR_RESYNC 2 mirrors 000102 1782481711.625804 MIRROR_EXTEND to 3 mirrors 000104 1782481712.319341 trunc from 0x03e9a2 to 0x0240e7 (0x1a8ba bytes) 000105 1782481712.410826 read 0x005a86 thru 0x00a016 (0x4591 bytes) 000106 1782481712.423916 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x240e8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x240e8): Mirror 2: 0x13074168 (stale) Mirror 3: 0x13074168 (stale) CRC-32 checksum value for chunk [0, 0x240e8): Mirror 1: 0x13074168 Mirror 4: 0x13074168 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x240e8: skipped 000107 1782481713.152102 trunc from 0x0240e8 to 0x02ea8c (0xa9a5 bytes) 000108 1782481713.276284 mapwrite 0x035ee2 thru 0x03ffff (0xa11e bytes) 000110 1782481713.358258 mapwrite 0x03ffab thru 0x03ffff (0x55 bytes) 000112 1782481713.410244 write 0x00435f thru 0x00be1c (0x7abe bytes) 000114 1782481713.429297 mapread 0x017be5 thru 0x01c68b (0x4aa7 bytes) 000117 1782481713.443324 write 0x003b6a thru 0x00d3c1 (0x9858 bytes) 000118 1782481713.454083 mapwrite 0x0032fa thru 0x012b4e (0xf855 bytes) 000120 1782481713.524994 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2c426a9d (stale) Mirror 4: 0x3fa04eb0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2c426a9d (stale) Mirror 4: 0x3fa04eb0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000122 1782481713.816714 mapread 0x01ed82 thru 0x0285be (0x983d bytes) 000124 1782481713.838459 MIRROR_RESYNC 3 mirrors 000125 1782481714.694443 mapwrite 0x00e0c4 thru 0x01d5db (0xf518 bytes) 000127 1782481714.940533 trunc from 0x040000 to 0x00bce5 (0x3431a bytes) 000128 1782481714.964198 mapread 0x0037e4 thru 0x0082b5 (0x4ad2 bytes) 000129 1782481714.982456 MIRROR_RESYNC 3 mirrors 000132 1782481715.752006 MIRROR_EXTEND to 4 mirrors 000134 1782481716.535809 write 0x031566 thru 0x03c840 (0xb2db bytes) 000136 1782481716.674670 mapread 0x00f67b thru 0x0123a7 (0x2d2d bytes) 000138 1782481716.726922 MIRROR_RESYNC 4 mirrors 000142 1782481717.857016 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) 000143 1782481718.272828 read 0x034ab3 thru 0x03c840 (0x7d8e bytes) 000144 1782481718.289333 write 0x012713 thru 0x020379 (0xdc67 bytes) 000146 1782481718.369501 read 0x01dfa3 thru 0x026e6b (0x8ec9 bytes) 000148 1782481718.392539 trunc from 0x03c841 to 0x01f7dc (0x1d064 bytes) 000149 1782481718.423122 trunc from 0x01f7dd to 0x005547 (0x1a295 bytes) 000151 1782481718.451494 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x5548): Mirror 5: 0x486d24fb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5548): Mirror 5: 0x486d24fb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5548: skipped 000153 1782481718.758561 mapwrite 0x02b9b0 thru 0x035f66 (0xa5b7 bytes) 000154 1782481718.922933 mapread 0x01d5ed thru 0x0286df (0xb0f3 bytes) 000160 1782481718.946893 write 0x0336ab thru 0x037174 (0x3aca bytes) 000162 1782481718.952305 mapread 0x031023 thru 0x031e44 (0xe22 bytes) 000164 1782481718.954800 write 0x02425c thru 0x02ab5a (0x68ff bytes) 000167 1782481718.960554 mapread 0x022c5e thru 0x02c02a (0x93cd bytes) 000171 1782481718.974464 MIRROR_RESYNC 2 mirrors 000174 1782481719.619129 mapwrite 0x01a7c8 thru 0x029391 (0xebca bytes) 000175 1782481719.797274 MIRROR_RESYNC 2 mirrors 000176 1782481720.514242 read 0x02a40e thru 0x02e0de (0x3cd1 bytes) 000177 1782481720.527935 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) 000179 1782481720.718864 read 0x030f59 thru 0x037174 (0x621c bytes) 000180 1782481720.732419 write 0x0087b7 thru 0x014dd6 (0xc620 bytes) 000181 1782481720.773956 mapread 0x011e58 thru 0x01985c (0x7a05 bytes) 000182 1782481720.801039 write 0x005cd4 thru 0x015098 (0xf3c5 bytes) 000183 1782481720.810142 trunc from 0x037175 to 0x011f3f (0x25235 bytes) 000188 1782481720.843143 trunc from 0x011f40 to 0x0367c7 (0x24888 bytes) 000189 1782481720.862169 read 0x0346b0 thru 0x0367c7 (0x2118 bytes) 000190 1782481720.871818 mapread 0x0033cc thru 0x007cb6 (0x48eb bytes) 000193 1782481720.876008 trunc from 0x0367c8 to 0x0250e1 (0x116e6 bytes) 000194 1782481720.911895 MIRROR_EXTEND to 2 mirrors 000196 1782481721.522121 trunc from 0x0250e2 to 0x004e2d (0x202b4 bytes) 000197 1782481721.666675 read 0x001f17 thru 0x004e2d (0x2f17 bytes) 000198 1782481721.677233 mapread 0x00398e thru 0x004e2d (0x14a0 bytes) 000201 1782481721.694330 mapwrite 0x015c79 thru 0x023b0c (0xde94 bytes) 000203 1782481721.836566 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x23b0d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23b0d): Mirror 2: 0x5314fb6e (stale) CRC-32 checksum value for chunk [0, 0x23b0d): Mirror 1: 0x9568a86b Mirror 3: 0x9568a86b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23b0d: skipped 000205 1782481722.673852 mapwrite 0x03999a thru 0x03ffff (0x6666 bytes) 000206 1782481722.823355 write 0x014b9c thru 0x021380 (0xc7e5 bytes) 000208 1782481722.836252 write 0x037d52 thru 0x03ffff (0x82ae bytes) 000211 1782481722.844116 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x65cf9d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x65cf9d6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000212 1782481723.109621 trunc from 0x040000 to 0x0126e7 (0x2d918 bytes) 000213 1782481723.131286 write 0x00ccc7 thru 0x018385 (0xb6bf bytes) 000215 1782481723.142840 mapwrite 0x033cbf thru 0x036546 (0x2888 bytes) 000216 1782481723.241886 read 0x002869 thru 0x007dfb (0x5593 bytes) 000217 1782481723.250624 write 0x03fc67 thru 0x03ffff (0x399 bytes) 000218 1782481723.252727 mapread 0x02c685 thru 0x031f80 (0x58fc bytes) 000220 1782481723.266228 mapread 0x037362 thru 0x03f3c4 (0x8063 bytes) 000221 1782481723.280131 write 0x0339f1 thru 0x0351c1 (0x17d1 bytes) 000222 1782481723.283867 trunc from 0x040000 to 0x029e98 (0x16167 bytes) 000223 1782481723.307550 write 0x0253b2 thru 0x034ef7 (0xfb46 bytes) 000224 1782481723.315720 write 0x0080a8 thru 0x01794c (0xf8a5 bytes) 000225 1782481723.333380 mapwrite 0x034db2 thru 0x03715e (0x23ad bytes) 000228 1782481723.442428 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3715f): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3715f): Mirror 3: 0x32ac3f7a (stale) CRC-32 checksum value for chunk [0, 0x3715f): Mirror 1: 0x5c2f093 Mirror 4: 0x5c2f093 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3715f: skipped 000229 1782481724.183089 mapwrite 0x005a90 thru 0x007f67 (0x24d8 bytes) 000235 1782481724.382830 mapread 0x00ef09 thru 0x014586 (0x567e bytes) 000236 1782481724.410626 write 0x00d07d thru 0x00dbd9 (0xb5d bytes) 000237 1782481724.412220 trunc from 0x03715f to 0x02f212 (0x7f4c bytes) 000238 1782481724.479213 MIRROR_RESYNC 3 mirrors 000239 1782481725.292463 mapread 0x0274f1 thru 0x02f212 (0x7d22 bytes) 000240 1782481725.321612 mapread 0x02be5b thru 0x02d04a (0x11f0 bytes) 000243 1782481725.327845 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) 000244 1782481725.656836 write 0x03dbc4 thru 0x03e2f2 (0x72f bytes) 000248 1782481725.745556 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) 000249 1782481725.926935 mapread 0x033adf thru 0x03d37b (0x989d bytes) 000252 1782481725.956522 mapread 0x02171d thru 0x02410d (0x29f1 bytes) 000254 1782481725.966145 mapread 0x039196 thru 0x03e2f2 (0x515d bytes) 000255 1782481725.973056 trunc from 0x03e2f3 to 0x006ac3 (0x3782f bytes) 000256 1782481725.990048 read 0x0054f1 thru 0x006ac3 (0x15d3 bytes) 000258 1782481725.999874 mapread 0x004ff2 thru 0x006ac3 (0x1ad2 bytes) 000261 1782481726.018654 mapwrite 0x00096a thru 0x00d947 (0xcfde bytes) 000262 1782481726.120508 MIRROR_EXTEND to 2 mirrors 000263 1782481726.713854 write 0x0216c6 thru 0x02a9d2 (0x930d bytes) 000266 1782481726.837970 read 0x028da7 thru 0x029cc5 (0xf1f bytes) 000267 1782481726.838612 read 0x0232ab thru 0x02a9d2 (0x7728 bytes) 000268 1782481726.851637 write 0x021c2b thru 0x026a4d (0x4e23 bytes) 000270 1782481726.853326 write 0x025cdd thru 0x03073b (0xaa5f bytes) 000273 1782481726.865492 write 0x000608 thru 0x0009ea (0x3e3 bytes) 000276 1782481726.887726 read 0x02b40c thru 0x02f41c (0x4011 bytes) 000277 1782481726.889191 trunc from 0x03073c to 0x034b43 (0x4408 bytes) 000278 1782481726.986485 trunc from 0x034b44 to 0x011198 (0x239ab bytes) 000279 1782481727.010951 trunc from 0x011199 to 0x032352 (0x211ba bytes) 000282 1782481727.043778 mapwrite 0x026795 thru 0x031a04 (0xb270 bytes) 000283 1782481727.149605 trunc from 0x032353 to 0x02e0e9 (0x4269 bytes) 000284 1782481727.178498 write 0x02143a thru 0x02a31b (0x8ee2 bytes) 000286 1782481727.186310 MIRROR_RESYNC 2 mirrors 000291 1782481727.975561 mapwrite 0x03b428 thru 0x03ffff (0x4bd8 bytes) 000293 1782481728.134431 write 0x02096f thru 0x02fcaa (0xf33c bytes) 000294 1782481728.150338 write 0x010aa1 thru 0x01442b (0x398b bytes) 000295 1782481728.163814 mapwrite 0x00029a thru 0x008c73 (0x89da bytes) 000297 1782481728.219861 MIRROR_RESYNC 2 mirrors 000298 1782481728.859758 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) 000301 1782481729.010265 mapread 0x000338 thru 0x004ed2 (0x4b9b bytes) 000303 1782481729.052628 mapwrite 0x038a55 thru 0x03ffff (0x75ab bytes) 000304 1782481729.127546 read 0x020458 thru 0x02b5a2 (0xb14b bytes) 000305 1782481729.147872 mapwrite 0x00cec1 thru 0x011933 (0x4a73 bytes) 000306 1782481729.211046 write 0x009646 thru 0x010dd9 (0x7794 bytes) 000308 1782481729.218292 MIRROR_EXTEND to 2 mirrors 000309 1782481729.732459 mapwrite 0x02a508 thru 0x03134e (0x6e47 bytes) 000312 1782481729.894889 trunc from 0x040000 to 0x02eb43 (0x114bc bytes) 000313 1782481729.968252 trunc from 0x02eb44 to 0x039d64 (0xb221 bytes) 000314 1782481730.005128 write 0x036525 thru 0x03f6b4 (0x9190 bytes) 000315 1782481730.011220 trunc from 0x03f6b5 to 0x0072cb (0x383e9 bytes) 000319 1782481730.055216 MIRROR_RESYNC 2 mirrors 000321 1782481730.791100 mapread 0x000bfc thru 0x0072cb (0x66d0 bytes) 000324 1782481730.814522 read 0x0014f0 thru 0x005a59 (0x456a bytes) 000326 1782481730.815674 mapread 0x004282 thru 0x006f88 (0x2d07 bytes) 000328 1782481730.817447 read 0x0029ce thru 0x0072cb (0x48fe bytes) 000329 1782481730.818367 mapread 0x0062c4 thru 0x0072cb (0x1008 bytes) 000330 1782481730.819431 read 0x004df9 thru 0x0072cb (0x24d3 bytes) 000331 1782481730.823403 MIRROR_EXTEND to 3 mirrors 000332 1782481731.410205 MIRROR_RESYNC 3 mirrors 000338 1782481731.800510 read 0x0055aa thru 0x0072cb (0x1d22 bytes) 000339 1782481731.807259 mapwrite 0x02dae1 thru 0x02fc56 (0x2176 bytes) 000342 1782481732.003145 mapread 0x01013f thru 0x01d38f (0xd251 bytes) 000348 1782481732.035921 read 0x0014c6 thru 0x0056c5 (0x4200 bytes) 000349 1782481732.046021 trunc from 0x02fc57 to 0x02b505 (0x4751 bytes) 000353 1782481732.067989 write 0x01ac77 thru 0x02a137 (0xf4c1 bytes) 000355 1782481732.076669 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2b506): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b506): Mirror 2: 0xd2843101 (stale) Mirror 3: 0xd2843101 (stale) CRC-32 checksum value for chunk [0, 0x2b506): Mirror 1: 0xe171dec0 Mirror 4: 0xe171dec0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b506: skipped 000356 1782481732.933567 mapwrite 0x030ab9 thru 0x03ffff (0xf547 bytes) 000357 1782481733.183951 MIRROR_RESYNC 4 mirrors 000359 1782481733.959191 trunc from 0x040000 to 0x01d37f (0x22c80 bytes) 000360 1782481734.047733 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1d380): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d380): Mirror 2: 0x99beb4db (stale) Mirror 3: 0x99beb4db (stale) Mirror 4: 0x99beb4db (stale) CRC-32 checksum value for chunk [0, 0x1d380): Mirror 1: 0x99beb4db Mirror 5: 0x99beb4db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d380: skipped 000361 1782481734.772667 mapwrite 0x025267 thru 0x02eea1 (0x9c3b bytes) 000363 1782481734.944397 mapwrite 0x001d76 thru 0x0053ad (0x3638 bytes) 000364 1782481734.998903 trunc from 0x02eea2 to 0x0358bf (0x6a1e bytes) 000365 1782481735.019203 mapwrite 0x014f95 thru 0x024150 (0xf1bc bytes) 000366 1782481735.083352 read 0x004824 thru 0x00954c (0x4d29 bytes) 000367 1782481735.084093 trunc from 0x0358c0 to 0x013fb0 (0x2190f bytes) 000369 1782481735.102956 mapread 0x00bcef thru 0x00e3a4 (0x26b6 bytes) 000370 1782481735.105277 write 0x013f4d thru 0x021480 (0xd534 bytes) 000371 1782481735.111413 mapread 0x00daa8 thru 0x01988f (0xbde8 bytes) 000372 1782481735.121492 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21481): Mirror 3: 0xadb653e5 (stale) Mirror 4: 0xadb653e5 (stale) Mirror 5: 0x7aff65c2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21481): Mirror 3: 0xadb653e5 (stale) Mirror 4: 0xadb653e5 (stale) Mirror 5: 0x7aff65c2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21481: skipped 000373 1782481735.490932 trunc from 0x021481 to 0x029475 (0x7ff5 bytes) 000374 1782481735.507776 trunc from 0x029476 to 0x03e1fc (0x14d87 bytes) 000376 1782481735.535628 trunc from 0x03e1fd to 0x0168d5 (0x27927 bytes) 000379 1782481735.557038 trunc from 0x0168d6 to 0x00c4b8 (0xa41d bytes) 000380 1782481735.575851 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc4b9): Mirror 3: 0x92e0ce1 (stale) Mirror 5: 0x92e0ce1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc4b9): Mirror 3: 0x92e0ce1 (stale) Mirror 5: 0x92e0ce1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc4b9: skipped 000381 1782481735.877198 trunc from 0x00c4b9 to 0x002398 (0xa120 bytes) 000383 1782481735.897964 write 0x00d0c4 thru 0x01731f (0xa25c bytes) 000384 1782481735.900488 trunc from 0x017320 to 0x02b19f (0x13e80 bytes) 000386 1782481735.916656 mapwrite 0x025993 thru 0x02a09c (0x470a bytes) 000388 1782481735.974968 write 0x0080d3 thru 0x00c5e2 (0x4510 bytes) 000389 1782481735.985202 write 0x03d72d thru 0x03ffff (0x28d3 bytes) 000391 1782481735.986820 mapwrite 0x00ccbd thru 0x01b6c2 (0xea06 bytes) 000392 1782481736.046494 read 0x03e000 thru 0x03ffff (0x2000 bytes) 000393 1782481736.047699 trunc from 0x040000 to 0x01e275 (0x21d8a bytes) 000394 1782481736.077762 mapread 0x00d215 thru 0x010f23 (0x3d0f bytes) 000395 1782481736.078918 read 0x00a49a thru 0x010158 (0x5cbf bytes) 000397 1782481736.080114 trunc from 0x01e276 to 0x015d99 (0x84dc bytes) 000402 1782481736.103865 read 0x007f57 thru 0x00d955 (0x59ff bytes) 000403 1782481736.115453 read 0x001866 thru 0x00f624 (0xddbf bytes) 000405 1782481736.117644 mapwrite 0x004062 thru 0x00edf1 (0xad90 bytes) 000406 1782481736.189512 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15d9a): Mirror 5: 0x8781b63b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15d9a): Mirror 5: 0x8781b63b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15d9a: skipped 000408 1782481736.463473 trunc from 0x015d9a to 0x0159ff (0x39a bytes) 000409 1782481736.483658 mapwrite 0x0334c3 thru 0x03ffff (0xcb3d bytes) 000410 1782481736.580925 read 0x02507b thru 0x026b2c (0x1ab2 bytes) 000412 1782481736.588627 mapread 0x00d908 thru 0x012356 (0x4a4f bytes) 000413 1782481736.602868 read 0x002957 thru 0x00b423 (0x8acd bytes) 000414 1782481736.612112 trunc from 0x040000 to 0x00c5e6 (0x33a19 bytes) 000415 1782481736.635645 trunc from 0x00c5e7 to 0x02461c (0x18036 bytes) 000416 1782481736.649839 read 0x01ab98 thru 0x021981 (0x6dea bytes) 000417 1782481736.665541 mapread 0x00bb58 thru 0x01ac46 (0xf0ef bytes) 000418 1782481736.668304 mapread 0x01d683 thru 0x01e71e (0x109c bytes) 000421 1782481736.673990 MIRROR_RESYNC 2 mirrors 000422 1782481737.153904 trunc from 0x02461d to 0x01765e (0xcfbe bytes) 000426 1782481737.273390 trunc from 0x01765f to 0x02fe2e (0x187d0 bytes) 000427 1782481737.296169 read 0x00a709 thru 0x00ddf9 (0x36f1 bytes) 000428 1782481737.308221 mapwrite 0x0312ad thru 0x032cca (0x1a1e bytes) 000429 1782481737.402555 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) 000430 1782481737.541767 write 0x0112a2 thru 0x01792c (0x668b bytes) 000432 1782481737.556536 read 0x021910 thru 0x026ca1 (0x5392 bytes) 000433 1782481737.569047 mapread 0x0174d0 thru 0x018bba (0x16eb bytes) 000434 1782481737.583042 trunc from 0x032ccb to 0x029098 (0x9c32 bytes) 000435 1782481737.603399 mapwrite 0x02bd0c thru 0x02f65e (0x3953 bytes) 000436 1782481737.697887 write 0x03e67a thru 0x03ffff (0x1986 bytes) 000440 1782481737.704276 trunc from 0x040000 to 0x01323e (0x2cdc1 bytes) 000441 1782481737.717817 mapwrite 0x01bb35 thru 0x01cc9a (0x1166 bytes) 000442 1782481737.775741 mapread 0x00cf2c thru 0x01a93d (0xda12 bytes) 000443 1782481737.783104 write 0x03f1e1 thru 0x03ffff (0xe1f bytes) 000444 1782481737.784362 mapread 0x02d47e thru 0x03846c (0xafef bytes) 000447 1782481737.803085 trunc from 0x040000 to 0x01cb9c (0x23463 bytes) 000449 1782481737.823388 read 0x01697e thru 0x018b4a (0x21cd bytes) 000454 1782481737.826141 write 0x036da7 thru 0x03ffff (0x9259 bytes) 000456 1782481737.828167 trunc from 0x040000 to 0x03d166 (0x2e99 bytes) 000460 1782481737.849324 mapread 0x01f78a thru 0x02dff4 (0xe86b bytes) 000462 1782481737.869619 write 0x0067e9 thru 0x015cef (0xf507 bytes) 000464 1782481737.874927 write 0x0342a7 thru 0x0355b6 (0x1310 bytes) 000467 1782481737.883241 mapread 0x025df4 thru 0x02a3b9 (0x45c6 bytes) 000468 1782481737.888205 mapwrite 0x00e9cb thru 0x0199e0 (0xb016 bytes) 000472 1782481737.968058 write 0x037555 thru 0x03777f (0x22b bytes) 000473 1782481737.969067 mapread 0x0298af thru 0x038261 (0xe9b3 bytes) 000474 1782481737.972415 trunc from 0x03d167 to 0x00acdd (0x32489 bytes) 000475 1782481737.989350 read 0x005cfc thru 0x00acdd (0x4fe2 bytes) 000477 1782481737.990749 mapread 0x002546 thru 0x00acdd (0x8798 bytes) 000478 1782481737.993663 write 0x028966 thru 0x02ca56 (0x40f1 bytes) 000479 1782481737.997535 mapread 0x029082 thru 0x02ca56 (0x39d5 bytes) 000485 1782481738.010107 mapwrite 0x02cbe7 thru 0x03beae (0xf2c8 bytes) 000490 1782481738.123160 trunc from 0x03beaf to 0x01be2a (0x20084 bytes) 000494 1782481738.140220 mapwrite 0x01b8c4 thru 0x028946 (0xd083 bytes) 000496 1782481738.272656 trunc from 0x028947 to 0x0169af (0x11f97 bytes) 000499 1782481738.323382 mapread 0x00011b thru 0x00d424 (0xd30a bytes) 000501 1782481738.325952 mapwrite 0x03bafe thru 0x03ffff (0x4502 bytes) 000503 1782481738.404350 mapread 0x009af7 thru 0x00edf4 (0x52fe bytes) 000505 1782481738.407406 mapread 0x0126fd thru 0x01ba85 (0x9389 bytes) 000507 1782481738.428282 read 0x009342 thru 0x00a148 (0xe07 bytes) 000510 1782481738.430484 trunc from 0x040000 to 0x012f72 (0x2d08d bytes) 000511 1782481738.445042 mapread 0x01080a thru 0x012f72 (0x2769 bytes) 000514 1782481738.448398 trunc from 0x012f73 to 0x02f8ed (0x1c97b bytes) 000515 1782481738.458787 read 0x026f86 thru 0x02a47e (0x34f9 bytes) 000516 1782481738.467832 read 0x006d0a thru 0x01273a (0xba31 bytes) 000518 1782481738.469649 mapwrite 0x00cc38 thru 0x01a5d0 (0xd999 bytes) 000519 1782481738.519304 write 0x009377 thru 0x00da32 (0x46bc bytes) 000521 1782481738.520966 mapwrite 0x0341b5 thru 0x03ffff (0xbe4b bytes) 000522 1782481738.586837 write 0x023819 thru 0x0283ee (0x4bd6 bytes) 000527 1782481738.594010 mapread 0x0047d9 thru 0x005fa2 (0x17ca bytes) 000528 1782481738.595193 write 0x00c53d thru 0x01b5c7 (0xf08b bytes) 000530 1782481738.598618 mapread 0x021645 thru 0x02f70c (0xe0c8 bytes) 000531 1782481738.600319 mapread 0x02dfc1 thru 0x037736 (0x9776 bytes) 000538 1782481738.616753 MIRROR_EXTEND to 2 mirrors 000541 1782481739.138160 MIRROR_RESYNC 2 mirrors 000544 1782481739.387769 mapwrite 0x016a6b thru 0x024e21 (0xe3b7 bytes) 000546 1782481739.569804 write 0x01a1d5 thru 0x024087 (0x9eb3 bytes) 000547 1782481739.572244 mapwrite 0x023240 thru 0x02690f (0x36d0 bytes) 000550 1782481739.625857 write 0x03010e thru 0x03d020 (0xcf13 bytes) 000551 1782481739.636980 write 0x00a81a thru 0x00faa1 (0x5288 bytes) 000553 1782481739.647784 trunc from 0x040000 to 0x0292c5 (0x16d3a bytes) 000554 1782481739.691653 trunc from 0x0292c6 to 0x01c841 (0xca84 bytes) 000555 1782481739.718352 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1c842): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c842): Mirror 2: 0x9d611d8 (stale) CRC-32 checksum value for chunk [0, 0x1c842): Mirror 1: 0x3dfbe470 Mirror 3: 0x3dfbe470 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c842: skipped 000557 1782481740.353117 mapwrite 0x003712 thru 0x004db2 (0x16a1 bytes) 000558 1782481740.513030 read 0x00570c thru 0x015495 (0xfd8a bytes) 000559 1782481740.536057 trunc from 0x01c842 to 0x03c3bb (0x1fb7a bytes) 000560 1782481740.555376 mapwrite 0x00653c thru 0x013b15 (0xd5da bytes) 000562 1782481740.596949 trunc from 0x03c3bc to 0x005484 (0x36f37 bytes) 000563 1782481740.613522 read 0x004a4b thru 0x005484 (0xa3a bytes) 000567 1782481740.616057 trunc from 0x005485 to 0x01465f (0xf1db bytes) 000568 1782481740.635821 mapread 0x007680 thru 0x00df9e (0x691f bytes) 000571 1782481740.647907 mapwrite 0x001b0d thru 0x00c765 (0xac59 bytes) 000572 1782481740.690593 mapwrite 0x011133 thru 0x011965 (0x833 bytes) 000573 1782481740.737439 mapwrite 0x012a63 thru 0x01c256 (0x97f4 bytes) 000576 1782481740.821891 trunc from 0x01c257 to 0x0388ea (0x1c694 bytes) 000580 1782481740.838501 read 0x0170ef thru 0x01fd3e (0x8c50 bytes) 000581 1782481740.845997 read 0x01e09c thru 0x02ceb2 (0xee17 bytes) 000582 1782481740.847047 write 0x0234b2 thru 0x02641b (0x2f6a bytes) 000584 1782481740.849024 trunc from 0x0388eb to 0x035c7f (0x2c6b bytes) 000588 1782481740.866402 mapread 0x023138 thru 0x02926c (0x6135 bytes) 000591 1782481740.869723 write 0x03bfd4 thru 0x03ffff (0x402c bytes) 000593 1782481740.873085 trunc from 0x040000 to 0x032a46 (0xd5b9 bytes) 000596 1782481740.896961 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x32a47): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32a47): Mirror 2: 0xfec235e2 (stale) Mirror 3: 0xc3df8a24 (stale) CRC-32 checksum value for chunk [0, 0x32a47): Mirror 1: 0x7c67fb97 Mirror 4: 0x7c67fb97 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32a47: skipped 000597 1782481741.661535 read 0x0155b0 thru 0x0251f2 (0xfc43 bytes) 000599 1782481741.681647 read 0x018fea thru 0x01b458 (0x246f bytes) 000601 1782481741.705337 read 0x02b874 thru 0x032a46 (0x71d3 bytes) 000602 1782481741.724729 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x32a47): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32a47): Mirror 2: 0xfec235e2 (stale) Mirror 3: 0xc3df8a24 (stale) CRC-32 checksum value for chunk [0, 0x32a47): Mirror 1: 0x7c67fb97 Mirror 4: 0x7c67fb97 Mirror 5: 0x7c67fb97 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32a47: skipped 000603 1782481742.453646 mapwrite 0x01b955 thru 0x0294be (0xdb6a bytes) 000604 1782481742.613158 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32a47): Mirror 3: 0xc3df8a24 (stale) Mirror 4: 0x7c67fb97 (stale) Mirror 5: 0x7c67fb97 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32a47): Mirror 3: 0xc3df8a24 (stale) Mirror 4: 0x7c67fb97 (stale) Mirror 5: 0x7c67fb97 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32a47: skipped 000605 1782481742.957649 MIRROR_RESYNC 4 mirrors 000607 1782481743.733817 trunc from 0x032a47 to 0x03e3cc (0xb986 bytes) 000608 1782481743.809874 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3e3cd): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e3cd): Mirror 3: 0x1e1d7faf (stale) Mirror 4: 0x1e1d7faf (stale) Mirror 5: 0x1e1d7faf (stale) CRC-32 checksum value for chunk [0, 0x3e3cd): Mirror 1: 0x1e1d7faf Mirror 6: 0x1e1d7faf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e3cd: skipped 000610 1782481744.574846 trunc from 0x03e3cd to 0x00b388 (0x33044 bytes) 000611 1782481744.691175 trunc from 0x00b389 to 0x011ec6 (0x6b3e bytes) 000612 1782481744.728466 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x11ec7): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11ec7): Mirror 3: 0x14448f6 (stale) Mirror 4: 0x14448f6 (stale) Mirror 5: 0x14448f6 (stale) Mirror 6: 0x14448f6 (stale) CRC-32 checksum value for chunk [0, 0x11ec7): Mirror 1: 0x3b202ace Mirror 7: 0x3b202ace lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11ec7: skipped 000614 1782481745.448150 mapwrite 0x03127b thru 0x038649 (0x73cf bytes) 000615 1782481745.658428 read 0x02d53e thru 0x02f4bc (0x1f7f bytes) 000616 1782481745.660327 read 0x016021 thru 0x01ff83 (0x9f63 bytes) 000619 1782481745.671866 mapwrite 0x0321e4 thru 0x03735c (0x5179 bytes) 000620 1782481745.718626 read 0x020afa thru 0x025ec7 (0x53ce bytes) 000621 1782481745.724518 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3864a): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3864a): Mirror 3: 0xec95e779 (stale) Mirror 4: 0xec95e779 (stale) Mirror 5: 0xec95e779 (stale) Mirror 6: 0xec95e779 (stale) Mirror 7: 0xf8cbf35d (stale) CRC-32 checksum value for chunk [0, 0x3864a): Mirror 1: 0x578b4707 Mirror 8: 0x578b4707 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3864a: skipped 000623 1782481746.644122 read 0x0352c0 thru 0x038649 (0x338a bytes) 000625 1782481746.677042 MIRROR_RESYNC 7 mirrors 000626 1782481747.742391 mapwrite 0x03f49b thru 0x03ffff (0xb65 bytes) 000627 1782481747.868469 write 0x02d3da thru 0x036f8b (0x9bb2 bytes) 000628 1782481747.891190 MIRROR_RESYNC 7 mirrors 000629 1782481748.933893 trunc from 0x040000 to 0x0336d3 (0xc92c bytes) 000631 1782481749.013549 mapwrite 0x01ad60 thru 0x028e8d (0xe12e bytes) 000632 1782481749.092226 mapread 0x01cfc4 thru 0x01f478 (0x24b5 bytes) 000633 1782481749.093738 write 0x031eee thru 0x03feda (0xdfed bytes) 000634 1782481749.099823 trunc from 0x03fedb to 0x035ec0 (0xa01a bytes) 000635 1782481749.121952 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35ec1): Mirror 4: 0xa9c14402 (stale) Mirror 5: 0xa9c14402 (stale) Mirror 6: 0xa9c14402 (stale) Mirror 7: 0xa9c14402 (stale) Mirror 8: 0xa9c14402 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35ec1): Mirror 4: 0xa9c14402 (stale) Mirror 5: 0xa9c14402 (stale) Mirror 6: 0xa9c14402 (stale) Mirror 7: 0xa9c14402 (stale) Mirror 8: 0xa9c14402 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35ec1: skipped 000636 1782481749.520979 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x35ec1): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35ec1): Mirror 4: 0xa9c14402 (stale) Mirror 5: 0xa9c14402 (stale) Mirror 6: 0xa9c14402 (stale) Mirror 7: 0xa9c14402 (stale) Mirror 8: 0xa9c14402 (stale) CRC-32 checksum value for chunk [0, 0x35ec1): Mirror 1: 0xe73a2536 Mirror 9: 0xe73a2536 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35ec1: skipped 000641 1782481750.309224 mapread 0x021692 thru 0x02cc52 (0xb5c1 bytes) 000642 1782481750.372034 read 0x00c6d1 thru 0x013fd3 (0x7903 bytes) 000645 1782481750.400187 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35ec1): Mirror 4: 0xa9c14402 (stale) Mirror 5: 0xa9c14402 (stale) Mirror 6: 0xa9c14402 (stale) Mirror 7: 0xa9c14402 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35ec1): Mirror 4: 0xa9c14402 (stale) Mirror 5: 0xa9c14402 (stale) Mirror 6: 0xa9c14402 (stale) Mirror 7: 0xa9c14402 (stale) CRC-32 checksum value for chunk [0, 0x35ec1): Mirror 1: 0xe73a2536 Mirror 9: 0xe73a2536 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35ec1: skipped 000646 1782481751.086482 MIRROR_RESYNC 6 mirrors 000647 1782481751.912053 write 0x00e9f7 thru 0x01b458 (0xca62 bytes) 000650 1782481752.001506 read 0x03156e thru 0x035ec0 (0x4953 bytes) 000651 1782481752.009178 mapwrite 0x025ec7 thru 0x033482 (0xd5bc bytes) 000652 1782481752.099459 write 0x0051e6 thru 0x012ee9 (0xdd04 bytes) 000653 1782481752.124367 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x35ec1): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35ec1): Mirror 4: 0xe73a2536 (stale) Mirror 5: 0xe73a2536 (stale) Mirror 6: 0xe73a2536 (stale) Mirror 7: 0xe73a2536 (stale) Mirror 9: 0xe73a2536 (stale) CRC-32 checksum value for chunk [0, 0x35ec1): Mirror 1: 0xf2535c5c Mirror 10: 0xf2535c5c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35ec1: skipped 000656 1782481752.985029 mapread 0x0163fd thru 0x024c05 (0xe809 bytes) 000657 1782481753.041938 read 0x025b09 thru 0x0344c9 (0xe9c1 bytes) 000658 1782481753.068147 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35ec1): Mirror 4: 0xe73a2536 (stale) Mirror 5: 0xe73a2536 (stale) Mirror 7: 0xe73a2536 (stale) Mirror 9: 0xe73a2536 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35ec1): Mirror 4: 0xe73a2536 (stale) Mirror 5: 0xe73a2536 (stale) Mirror 7: 0xe73a2536 (stale) Mirror 9: 0xe73a2536 (stale) CRC-32 checksum value for chunk [0, 0x35ec1): Mirror 1: 0xf2535c5c Mirror 10: 0xf2535c5c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35ec1: skipped 000659 1782481753.657513 mapwrite 0x015fd8 thru 0x02078f (0xa7b8 bytes) 000661 1782481753.852768 MIRROR_RESYNC 6 mirrors 000663 1782481754.767849 read 0x007af5 thru 0x013200 (0xb70c bytes) 000664 1782481754.777226 mapwrite 0x0381a5 thru 0x03b217 (0x3073 bytes) 000665 1782481754.895544 write 0x03d975 thru 0x03e0d3 (0x75f bytes) 000666 1782481754.896762 mapread 0x03d07d thru 0x03e0d3 (0x1057 bytes) 000667 1782481754.897796 mapread 0x003c99 thru 0x010043 (0xc3ab bytes) 000668 1782481754.906296 mapwrite 0x012c44 thru 0x014e2b (0x21e8 bytes) 000669 1782481754.957135 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3e0d4): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e0d4): Mirror 4: 0x1460c5e4 (stale) Mirror 5: 0x1460c5e4 (stale) Mirror 7: 0x1460c5e4 (stale) Mirror 9: 0x1460c5e4 (stale) Mirror 10: 0x1460c5e4 (stale) CRC-32 checksum value for chunk [0, 0x3e0d4): Mirror 1: 0xe791bdf7 Mirror 11: 0xe791bdf7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e0d4: skipped 000670 1782481755.791160 read 0x0059fb thru 0x00731e (0x1924 bytes) 000671 1782481755.812963 trunc from 0x03e0d4 to 0x03bdeb (0x22e8 bytes) 000673 1782481755.903948 mapwrite 0x0273f7 thru 0x032492 (0xb09c bytes) 000674 1782481755.998563 mapwrite 0x026d67 thru 0x0296c8 (0x2962 bytes) 000675 1782481756.047412 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3bdec): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bdec): Mirror 4: 0x6d5affa3 (stale) Mirror 5: 0x6d5affa3 (stale) Mirror 7: 0x6d5affa3 (stale) Mirror 9: 0x6d5affa3 (stale) Mirror 10: 0x6d5affa3 (stale) Mirror 11: 0x44586e (stale) CRC-32 checksum value for chunk [0, 0x3bdec): Mirror 1: 0xf0c6b0fa Mirror 12: 0xf0c6b0fa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bdec: skipped 000676 1782481756.802101 write 0x021e1d thru 0x0318b0 (0xfa94 bytes) 000677 1782481756.875637 read 0x0125e6 thru 0x01d404 (0xae1f bytes) 000678 1782481756.886467 trunc from 0x03bdec to 0x0355d0 (0x681b bytes) 000679 1782481756.920509 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x355d1): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x355d1): Mirror 4: 0xf9cf493d (stale) Mirror 5: 0xf9cf493d (stale) Mirror 7: 0xf9cf493d (stale) Mirror 9: 0xf9cf493d (stale) Mirror 10: 0xf9cf493d (stale) Mirror 11: 0xf7b1b096 (stale) Mirror 12: 0x601c8a46 (stale) CRC-32 checksum value for chunk [0, 0x355d1): Mirror 1: 0x77569da0 Mirror 13: 0x77569da0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x355d1: skipped 000680 1782481757.771858 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x355d1): Mirror 4: 0xf9cf493d (stale) Mirror 5: 0xf9cf493d (stale) Mirror 7: 0xf9cf493d (stale) Mirror 9: 0xf9cf493d (stale) Mirror 11: 0xf7b1b096 (stale) Mirror 12: 0x601c8a46 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x355d1): Mirror 4: 0xf9cf493d (stale) Mirror 5: 0xf9cf493d (stale) Mirror 7: 0xf9cf493d (stale) Mirror 9: 0xf9cf493d (stale) Mirror 11: 0xf7b1b096 (stale) Mirror 12: 0x601c8a46 (stale) CRC-32 checksum value for chunk [0, 0x355d1): Mirror 1: 0x77569da0 Mirror 13: 0x77569da0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x355d1: skipped 000682 1782481758.631226 MIRROR_RESYNC 8 mirrors 000685 1782481759.796967 mapwrite 0x011933 thru 0x019a4d (0x811b bytes) 000687 1782481759.976441 read 0x0244b4 thru 0x03403f (0xfb8c bytes) 000689 1782481759.986095 mapwrite 0x02531e thru 0x02ec80 (0x9963 bytes) 000693 1782481760.042567 trunc from 0x0355d1 to 0x00aa4c (0x2ab84 bytes) 000695 1782481760.060531 mapread 0x001e21 thru 0x00aa4c (0x8c2c bytes) 000696 1782481760.069375 trunc from 0x00aa4d to 0x03219b (0x2774f bytes) 000697 1782481760.082275 write 0x00f3ed thru 0x01148a (0x209e bytes) 000699 1782481760.092733 mapread 0x0059e7 thru 0x00bde6 (0x6400 bytes) 000701 1782481760.107447 read 0x00ada1 thru 0x011cc3 (0x6f23 bytes) 000703 1782481760.108748 mapwrite 0x03486d thru 0x03d756 (0x8eea bytes) 000704 1782481760.221671 write 0x01479b thru 0x0235a8 (0xee0e bytes) 000705 1782481760.226485 read 0x0252b8 thru 0x029b0d (0x4856 bytes) 000707 1782481760.228020 trunc from 0x03d757 to 0x01a242 (0x23514 bytes) 000708 1782481760.253313 mapwrite 0x028868 thru 0x036960 (0xe0f9 bytes) 000710 1782481760.357153 trunc from 0x036961 to 0x01ccbb (0x19ca5 bytes) 000711 1782481760.377106 read 0x00cef4 thru 0x00eae3 (0x1bf0 bytes) 000713 1782481760.380040 mapread 0x00f077 thru 0x013df3 (0x4d7d bytes) 000715 1782481760.382130 read 0x013758 thru 0x01ccbb (0x9564 bytes) 000717 1782481760.383825 write 0x03219c thru 0x03ffff (0xde64 bytes) 000718 1782481760.385482 read 0x02d9f6 thru 0x02f3cb (0x19d6 bytes) 000720 1782481760.401827 mapwrite 0x01aa21 thru 0x027db3 (0xd393 bytes) 000724 1782481760.469593 MIRROR_RESYNC 8 mirrors 000727 1782481761.573129 MIRROR_RESYNC 8 mirrors 000728 1782481762.081473 read 0x0211da thru 0x0217fe (0x625 bytes) 000732 1782481762.091557 trunc from 0x040000 to 0x009173 (0x36e8c bytes) 000733 1782481762.216715 read 0x001501 thru 0x009173 (0x7c73 bytes) 000734 1782481762.225752 write 0x01d388 thru 0x0201ce (0x2e47 bytes) 000738 1782481762.230067 mapwrite 0x003db3 thru 0x0067a7 (0x29f5 bytes) 000741 1782481762.279793 write 0x03cf07 thru 0x03ffff (0x30f9 bytes) 000742 1782481762.281454 write 0x00b187 thru 0x00c338 (0x11b2 bytes) 000743 1782481762.291597 mapread 0x01f18a thru 0x022b03 (0x397a bytes) 000746 1782481762.304707 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x641c04b6 (stale) Mirror 5: 0x641c04b6 (stale) Mirror 7: 0x641c04b6 (stale) Mirror 9: 0x641c04b6 (stale) Mirror 11: 0x641c04b6 (stale) Mirror 12: 0x641c04b6 (stale) Mirror 13: 0x641c04b6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5f0c3a0c Mirror 14: 0x5f0c3a0c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000747 1782481763.190129 mapread 0x01f38a thru 0x02ee95 (0xfb0c bytes) 000752 1782481763.261410 trunc from 0x040000 to 0x037a50 (0x85af bytes) 000753 1782481763.388741 write 0x01646b thru 0x0214b9 (0xb04f bytes) 000754 1782481763.401050 read 0x014665 thru 0x01e10c (0x9aa8 bytes) 000759 1782481763.413070 read 0x02438f thru 0x027dc1 (0x3a33 bytes) 000760 1782481763.422781 mapread 0x00c364 thru 0x01911b (0xcdb8 bytes) 000762 1782481763.425509 write 0x0067b6 thru 0x0137ce (0xd019 bytes) 000764 1782481763.428056 write 0x01536e thru 0x01e8fd (0x9590 bytes) 000765 1782481763.430274 mapread 0x00b153 thru 0x01a270 (0xf11e bytes) 000766 1782481763.431639 mapread 0x00c78d thru 0x00e8fc (0x2170 bytes) 000769 1782481763.435306 mapwrite 0x00bdb3 thru 0x01278f (0x69dd bytes) 000770 1782481763.485655 write 0x0144f0 thru 0x02257e (0xe08f bytes) 000771 1782481763.487164 read 0x0309c3 thru 0x032c37 (0x2275 bytes) 000772 1782481763.488810 mapread 0x01d58d thru 0x0267eb (0x925f bytes) 000776 1782481763.492128 write 0x029245 thru 0x03671a (0xd4d6 bytes) 000777 1782481763.493995 write 0x0311c2 thru 0x034db1 (0x3bf0 bytes) 000780 1782481763.496540 mapread 0x0322d8 thru 0x032b1f (0x848 bytes) 000781 1782481763.498124 mapread 0x00e3b4 thru 0x01791b (0x9568 bytes) 000783 1782481763.504056 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37a51): Mirror 4: 0x86377b6b (stale) Mirror 5: 0x86377b6b (stale) Mirror 7: 0x86377b6b (stale) Mirror 9: 0x86377b6b (stale) Mirror 11: 0x86377b6b (stale) Mirror 12: 0x86377b6b (stale) Mirror 13: 0x86377b6b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a51): Mirror 4: 0x86377b6b (stale) Mirror 5: 0x86377b6b (stale) Mirror 7: 0x86377b6b (stale) Mirror 9: 0x86377b6b (stale) Mirror 11: 0x86377b6b (stale) Mirror 12: 0x86377b6b (stale) Mirror 13: 0x86377b6b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a51: skipped 000784 1782481764.046330 mapread 0x005528 thru 0x00f5b0 (0xa089 bytes) 000785 1782481764.074119 mapwrite 0x03974d thru 0x03d76f (0x4023 bytes) 000786 1782481764.164558 mapread 0x005c0e thru 0x00c335 (0x6728 bytes) 000787 1782481764.173987 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d770): Mirror 4: 0xa4a635fc (stale) Mirror 5: 0xa4a635fc (stale) Mirror 7: 0xa4a635fc (stale) Mirror 9: 0xa4a635fc (stale) Mirror 12: 0xa4a635fc (stale) Mirror 13: 0xa4a635fc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d770): Mirror 4: 0xa4a635fc (stale) Mirror 5: 0xa4a635fc (stale) Mirror 7: 0xa4a635fc (stale) Mirror 9: 0xa4a635fc (stale) Mirror 12: 0xa4a635fc (stale) Mirror 13: 0xa4a635fc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d770: skipped 000789 1782481764.666711 trunc from 0x03d770 to 0x01893b (0x24e34 bytes) 000793 1782481764.689856 read 0x00cb0a thru 0x01668f (0x9b86 bytes) 000794 1782481764.697647 mapread 0x00bbcc thru 0x015d60 (0xa195 bytes) 000795 1782481764.702846 trunc from 0x01893c to 0x020dde (0x84a3 bytes) 000797 1782481764.723219 write 0x003831 thru 0x00b20b (0x79db bytes) 000798 1782481764.729398 trunc from 0x020ddf to 0x00fef2 (0x10eec bytes) 000800 1782481764.745018 read 0x007c33 thru 0x00fef2 (0x82c0 bytes) 000803 1782481764.746819 read 0x00fa89 thru 0x00fef2 (0x46a bytes) 000804 1782481764.749437 mapwrite 0x02d08c thru 0x03c17f (0xf0f4 bytes) 000805 1782481764.853087 write 0x0064a9 thru 0x012cb6 (0xc80e bytes) 000807 1782481764.858135 write 0x035679 thru 0x038fe7 (0x396f bytes) 000811 1782481764.863938 mapwrite 0x034042 thru 0x0396d3 (0x5692 bytes) 000813 1782481764.953916 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3c180): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c180): Mirror 4: 0x5edf0473 (stale) Mirror 5: 0x5edf0473 (stale) Mirror 7: 0x5edf0473 (stale) Mirror 9: 0x5edf0473 (stale) Mirror 12: 0x5edf0473 (stale) Mirror 13: 0x5edf0473 (stale) CRC-32 checksum value for chunk [0, 0x3c180): Mirror 1: 0xbffe7087 Mirror 14: 0xbffe7087 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c180: skipped 000815 1782481765.787386 trunc from 0x03c180 to 0x01abdc (0x215a3 bytes) 000818 1782481765.874726 trunc from 0x01abdd to 0x02980f (0xec33 bytes) 000821 1782481765.897653 mapwrite 0x001cdc thru 0x0092fd (0x7622 bytes) 000825 1782481765.985200 mapread 0x0145c5 thru 0x020c43 (0xc67f bytes) 000827 1782481766.024238 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x29810): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29810): Mirror 4: 0x13b21126 (stale) Mirror 5: 0x13b21126 (stale) Mirror 7: 0x13b21126 (stale) Mirror 9: 0x13b21126 (stale) Mirror 12: 0x13b21126 (stale) Mirror 13: 0x13b21126 (stale) Mirror 14: 0xbf99a3fc (stale) CRC-32 checksum value for chunk [0, 0x29810): Mirror 1: 0xaea737e2 Mirror 15: 0xaea737e2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29810: skipped 000828 1782481766.993424 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29810): Mirror 4: 0x13b21126 (stale) Mirror 7: 0x13b21126 (stale) Mirror 9: 0x13b21126 (stale) Mirror 12: 0x13b21126 (stale) Mirror 13: 0x13b21126 (stale) Mirror 14: 0xbf99a3fc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29810): Mirror 4: 0x13b21126 (stale) Mirror 7: 0x13b21126 (stale) Mirror 9: 0x13b21126 (stale) Mirror 12: 0x13b21126 (stale) Mirror 13: 0x13b21126 (stale) Mirror 14: 0xbf99a3fc (stale) CRC-32 checksum value for chunk [0, 0x29810): Mirror 1: 0xaea737e2 Mirror 15: 0xaea737e2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29810: skipped 000829 1782481767.822202 mapwrite 0x0326cd thru 0x037287 (0x4bbb bytes) 000834 1782481767.979739 trunc from 0x037288 to 0x00a8e6 (0x2c9a1 bytes) 000835 1782481768.000919 trunc from 0x00a8e7 to 0x02eb42 (0x2425c bytes) 000836 1782481768.020284 mapread 0x01c671 thru 0x022f1f (0x68af bytes) 000839 1782481768.034345 mapread 0x01164e thru 0x01f598 (0xdf4b bytes) 000840 1782481768.045960 MIRROR_RESYNC 8 mirrors 000841 1782481769.066462 read 0x00adba thru 0x01a86e (0xfab5 bytes) 000843 1782481769.081952 write 0x031f1f thru 0x03ff06 (0xdfe8 bytes) 000844 1782481769.182452 read 0x0322d4 thru 0x03e3c6 (0xc0f3 bytes) 000848 1782481769.184784 read 0x015f0a thru 0x02258a (0xc681 bytes) 000851 1782481769.202500 trunc from 0x03ff07 to 0x01796f (0x28597 bytes) 000852 1782481769.241733 mapwrite 0x016c9d thru 0x023bf5 (0xcf59 bytes) 000854 1782481769.370616 mapwrite 0x0394d9 thru 0x03ba38 (0x2560 bytes) 000855 1782481769.471086 mapwrite 0x021413 thru 0x024a76 (0x3664 bytes) 000857 1782481769.527211 mapwrite 0x02ea9b thru 0x03c5db (0xdb41 bytes) 000858 1782481769.684808 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c5dc): Mirror 4: 0x9fe3edbb (stale) Mirror 9: 0x9fe3edbb (stale) Mirror 12: 0x9fe3edbb (stale) Mirror 13: 0x9fe3edbb (stale) Mirror 14: 0x9fe3edbb (stale) Mirror 15: 0x9fe3edbb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c5dc): Mirror 4: 0x9fe3edbb (stale) Mirror 9: 0x9fe3edbb (stale) Mirror 12: 0x9fe3edbb (stale) Mirror 13: 0x9fe3edbb (stale) Mirror 14: 0x9fe3edbb (stale) Mirror 15: 0x9fe3edbb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c5dc: skipped 000860 1782481770.113189 write 0x00f5e1 thru 0x01d2ae (0xdcce bytes) 000861 1782481770.133147 MIRROR_RESYNC 7 mirrors 000863 1782481771.179314 read 0x024d4a thru 0x02c870 (0x7b27 bytes) 000864 1782481771.192771 MIRROR_EXTEND to 8 mirrors 000865 1782481771.773820 read 0x01c775 thru 0x029ae7 (0xd373 bytes) 000866 1782481771.782813 trunc from 0x03c5dc to 0x024711 (0x17eca bytes) 000867 1782481771.909319 mapread 0x0113b7 thru 0x01d95c (0xc5a6 bytes) 000868 1782481771.936895 mapread 0x0181d1 thru 0x0234c5 (0xb2f5 bytes) 000869 1782481771.939485 read 0x01716a thru 0x021401 (0xa298 bytes) 000870 1782481771.948395 MIRROR_RESYNC 8 mirrors 000872 1782481773.060926 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) 000873 1782481773.389449 write 0x024f7e thru 0x02719e (0x2221 bytes) 000874 1782481773.458410 mapread 0x0137a1 thru 0x01c0fd (0x895d bytes) 000875 1782481773.490022 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2719f): Mirror 9: 0x8c6a66cf (stale) Mirror 12: 0x8c6a66cf (stale) Mirror 13: 0x8c6a66cf (stale) Mirror 15: 0x8c6a66cf (stale) Mirror 16: 0x8c6a66cf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2719f): Mirror 9: 0x8c6a66cf (stale) Mirror 12: 0x8c6a66cf (stale) Mirror 13: 0x8c6a66cf (stale) Mirror 15: 0x8c6a66cf (stale) Mirror 16: 0x8c6a66cf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2719f: skipped truncating to largest ever: 0x3fd05 000876 1782481773.953548 trunc from 0x02719f to 0x03fd04 (0x18b66 bytes) 000877 1782481773.975747 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fd05): Mirror 12: 0xc5173090 (stale) Mirror 13: 0xc5173090 (stale) Mirror 15: 0xc5173090 (stale) Mirror 16: 0xc5173090 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd05): Mirror 12: 0xc5173090 (stale) Mirror 13: 0xc5173090 (stale) Mirror 15: 0xc5173090 (stale) Mirror 16: 0xc5173090 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd05: skipped 000878 1782481774.432402 read 0x02ea2e thru 0x032bc1 (0x4194 bytes) 000879 1782481774.439041 mapread 0x0083ff thru 0x0109ce (0x85d0 bytes) 000880 1782481774.468025 trunc from 0x03fd05 to 0x007a94 (0x38270 bytes) 000882 1782481774.492089 read 0x002256 thru 0x004c53 (0x29fe bytes) 000883 1782481774.498124 mapread 0x0037c5 thru 0x004637 (0xe73 bytes) 000884 1782481774.499495 read 0x005a12 thru 0x007a94 (0x2083 bytes) 000887 1782481774.502606 trunc from 0x007a95 to 0x024274 (0x1c7e0 bytes) 000890 1782481774.524689 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24275): Mirror 13: 0x9d1ab2ba (stale) Mirror 15: 0x9d1ab2ba (stale) Mirror 16: 0x9d1ab2ba (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24275): Mirror 13: 0x9d1ab2ba (stale) Mirror 15: 0x9d1ab2ba (stale) Mirror 16: 0x9d1ab2ba (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24275: skipped 000894 1782481774.865136 read 0x0136eb thru 0x0212d1 (0xdbe7 bytes) 000895 1782481774.876539 trunc from 0x024275 to 0x01eebb (0x53b9 bytes) 000896 1782481774.902349 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1eebc): Mirror 13: 0xb5e57557 (stale) Mirror 16: 0xb5e57557 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1eebc): Mirror 13: 0xb5e57557 (stale) Mirror 16: 0xb5e57557 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1eebc: skipped 000897 1782481775.225739 read 0x018301 thru 0x01eebb (0x6bbb bytes) 000900 1782481775.235811 read 0x016169 thru 0x01eebb (0x8d53 bytes) 000901 1782481775.244887 mapwrite 0x02f960 thru 0x0382f7 (0x8998 bytes) 000904 1782481775.344304 mapwrite 0x037d16 thru 0x03ffff (0x82ea bytes) 000905 1782481775.447621 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xe97fd2a8 (stale) Mirror 16: 0xe97fd2a8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8db3c8cc Mirror 17: 0x8db3c8cc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000906 1782481776.138631 write 0x00e51e thru 0x00e6a5 (0x188 bytes) 000907 1782481776.236208 mapwrite 0x017535 thru 0x01b699 (0x4165 bytes) 000908 1782481776.321733 write 0x02008a thru 0x020b29 (0xaa0 bytes) 000909 1782481776.328859 MIRROR_RESYNC 4 mirrors 000911 1782481777.029161 trunc from 0x040000 to 0x006309 (0x39cf6 bytes) 000913 1782481777.128105 mapwrite 0x00431c thru 0x00b019 (0x6cfe bytes) 000915 1782481777.211711 trunc from 0x00b01a to 0x037a66 (0x2ca4d bytes) 000916 1782481777.228525 mapread 0x0029f9 thru 0x0079f9 (0x5001 bytes) 000917 1782481777.231081 read 0x00dec6 thru 0x00f3d9 (0x1514 bytes) 000918 1782481777.242657 MIRROR_RESYNC 4 mirrors 000919 1782481778.024345 mapread 0x02e2e3 thru 0x02efb1 (0xccf bytes) 000920 1782481778.030835 mapwrite 0x02d629 thru 0x032748 (0x5120 bytes) 000921 1782481778.228713 write 0x0167fa thru 0x020484 (0x9c8b bytes) 000922 1782481778.254255 mapwrite 0x000906 thru 0x00cb90 (0xc28b bytes) 000925 1782481778.358617 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x37a67): Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a67): Mirror 13: 0x8693dbb (stale) Mirror 16: 0x8693dbb (stale) Mirror 17: 0x8693dbb (stale) CRC-32 checksum value for chunk [0, 0x37a67): Mirror 1: 0xa25defca Mirror 18: 0xa25defca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a67: skipped 000927 1782481779.034892 trunc from 0x037a67 to 0x0331e3 (0x4883 bytes) 000930 1782481779.150335 mapwrite 0x03f7b9 thru 0x03ffff (0x847 bytes) 000935 1782481779.249244 mapread 0x0090b4 thru 0x016358 (0xd2a5 bytes) 000936 1782481779.288792 trunc from 0x040000 to 0x024f00 (0x1b0ff bytes) 000937 1782481779.331968 MIRROR_RESYNC 5 mirrors 000938 1782481780.132326 read 0x00cf8d thru 0x010e15 (0x3e89 bytes) 000940 1782481780.145960 write 0x006ed5 thru 0x011690 (0xa7bc bytes) 000941 1782481780.223330 read 0x00a990 thru 0x00b55d (0xbce bytes) 000943 1782481780.228407 read 0x00f2e0 thru 0x0129fd (0x371e bytes) 000946 1782481780.250219 trunc from 0x024f01 to 0x01da01 (0x74ff bytes) 000947 1782481780.278353 trunc from 0x01da02 to 0x00c94b (0x110b6 bytes) 000949 1782481780.305262 mapread 0x00c1b0 thru 0x00c94b (0x79c bytes) 000950 1782481780.307901 read 0x00b852 thru 0x00c94b (0x10fa bytes) 000951 1782481780.308889 trunc from 0x00c94c to 0x03691c (0x29fd1 bytes) 000956 1782481780.337012 read 0x025dd2 thru 0x02a215 (0x4444 bytes) 000957 1782481780.360466 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3691d): Mirror 13: 0x2308d5e0 (stale) Mirror 16: 0x2308d5e0 (stale) Mirror 17: 0x2308d5e0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3691d): Mirror 13: 0x2308d5e0 (stale) Mirror 16: 0x2308d5e0 (stale) Mirror 17: 0x2308d5e0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3691d: skipped 000959 1782481780.690167 MIRROR_RESYNC 4 mirrors 000960 1782481781.429358 mapread 0x020d8f thru 0x02e907 (0xdb79 bytes) 000961 1782481781.470757 write 0x01b495 thru 0x020491 (0x4ffd bytes) 000962 1782481781.561350 mapread 0x005616 thru 0x00fbb2 (0xa59d bytes) 000964 1782481781.577653 write 0x021969 thru 0x02263a (0xcd2 bytes) 000966 1782481781.598865 MIRROR_RESYNC 4 mirrors 000968 1782481782.413108 MIRROR_RESYNC 4 mirrors 000969 1782481782.781273 mapread 0x00df82 thru 0x011e8c (0x3f0b bytes) 000970 1782481782.804778 read 0x0306b2 thru 0x03691c (0x626b bytes) 000972 1782481782.818849 mapwrite 0x022c2b thru 0x026bf9 (0x3fcf bytes) 000973 1782481782.983428 mapread 0x009959 thru 0x009fbf (0x667 bytes) 000974 1782481783.009076 write 0x02a560 thru 0x030e5c (0x68fd bytes) 000975 1782481783.022471 mapwrite 0x02c559 thru 0x039543 (0xcfeb bytes) 000976 1782481783.120956 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39544): Mirror 13: 0x9ab3a19e (stale) Mirror 16: 0x9ab3a19e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39544): Mirror 13: 0x9ab3a19e (stale) Mirror 16: 0x9ab3a19e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39544: skipped 000977 1782481783.418815 trunc from 0x039544 to 0x0179a5 (0x21b9e bytes) 000979 1782481783.437135 mapwrite 0x03354e thru 0x03f07e (0xbb31 bytes) 000981 1782481783.552076 mapread 0x021835 thru 0x02f708 (0xded4 bytes) 000982 1782481783.567270 mapwrite 0x0047a1 thru 0x0081a9 (0x3a09 bytes) 000983 1782481783.678646 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f07f): Mirror 16: 0x7bc8037f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f07f): Mirror 16: 0x7bc8037f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f07f: skipped 000985 1782481783.944659 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) 000986 1782481784.063882 read 0x0231f3 thru 0x02aed6 (0x7ce4 bytes) 000987 1782481784.075658 write 0x0287e5 thru 0x02eee9 (0x6705 bytes) 000994 1782481784.090305 write 0x0260a4 thru 0x02f6c7 (0x9624 bytes) 000995 1782481784.099567 read 0x03c1c6 thru 0x03cbf4 (0xa2f bytes) 000996 1782481784.110599 write 0x03e747 thru 0x03ffff (0x18b9 bytes) 000997 1782481784.112228 mapread 0x032b1e thru 0x03dd34 (0xb217 bytes) 000998 1782481784.114099 read 0x031dd7 thru 0x03ffff (0xe229 bytes) 001000 1782481784.116056 read 0x028680 thru 0x035f47 (0xd8c8 bytes) All operations completed A-OK! PASS 70a (82s) == sanity-flr test 71: check mirror extend parent fid ==== 09:49:47 (1782481787) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 09:49:50 (1782481790) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x332:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x336:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.210534 s, 10.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x332:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x333:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x336:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178398 s, 5.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x332:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x333:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x337:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x336:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x332:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x333:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x337:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x336:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x334:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (7s) == sanity-flr test 200a: stress test ===================== 09:49:57 (1782481797) Starting client: oleg409-client.virtnet: -o user_xattr,flock 192.168.204.109@tcp:/lustre /mnt/lustre2 Starting client: oleg409-client.virtnet: -o user_xattr,flock 192.168.204.109@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2288704 .. Extending file size to 2215520 .. done Extending file size to 4325856 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7019680 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6528064 .. Extending file size to 8637184 .. Extending file size to 8783776 .. Extending file size to 8671904 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8911392 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57084 57085 57086 57089 57090 done 192.168.204.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg409-client.virtnet /mnt/lustre2 (opts:) 192.168.204.109@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg409-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (75s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:51:12 (1782481872) Starting client: oleg409-client.virtnet: -o user_xattr,flock 192.168.204.109@tcp:/lustre /mnt/lustre2 Starting client: oleg409-client.virtnet: -o user_xattr,flock 192.168.204.109@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7787904 .. Extending file size to 6449056 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7675520 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8163584 .. Extending file size to 8927488 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8260704 .. failed Extending file size to 8326528 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8818144 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9431584 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9338432 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 58663 58664 58666 58668 58670 192.168.204.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg409-client.virtnet /mnt/lustre2 (opts:) 192.168.204.109@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg409-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (71s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:52:23 (1782481943) Starting client: oleg409-client.virtnet: -o user_xattr,flock 192.168.204.109@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0688249 s, 30.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.209702 s, 10.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x6e3413bf (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg409-client.virtnet /mnt/lustre2 (opts:) PASS 200c (11s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:52:35 (1782481955) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.067233 s, 31.2 MB/s PASS 202 (4s) == sanity-flr test 203: mirror file preserve mirror ID === 09:52:39 (1782481959) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22f:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34d:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34d:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34e:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x349:0x0] } PASS 203 (4s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:52:43 (1782481963) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.142231 s, 14.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (5s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:52:48 (1782481968) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.11106 s, 18.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x350:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x351:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x350:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x351:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34d:0x0] } PASS 204b (5s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:52:53 (1782481973) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.113908 s, 18.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x352:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x352:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34f:0x0] } PASS 204c (28s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:53:21 (1782482001) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg409-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480623 s, 21.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x353:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x351:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x350:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x353:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x351:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x350:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x354:0x0] } PASS 204d (53s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:54:14 (1782482054) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg409-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x352:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x353:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.159725 s, 13.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x352:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x353:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x355:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x352:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x353:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x355:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (52s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:55:06 (1782482106) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg409-server mds-ost sync done. Creating new pool test_204f oleg409-server: Pool lustre.test_204f created Adding targets to pool oleg409-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg409-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.153905 s, 13.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x354:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x356:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x354:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x356:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg409-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg409-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg409-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (67s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:56:13 (1782482173) PASS 205a (4s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:56:17 (1782482177) PASS 205b (5s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:56:22 (1782482182) oleg409-server: Pool lustre.test_206 created Adding targets to pool oleg409-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg409-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg409-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg409-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg409-server: Pool lustre.test_206 destroyed PASS 206 (17s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:56:40 (1782482200) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053112 s, 19.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334622 s, 31.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x688116ce Mirror 3: 0x688116ce PASS 207 (7s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:56:47 (1782482207) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:56:49 (1782482209) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:56:52 (1782482212) Starting client: oleg409-client.virtnet: -o user_xattr,flock 192.168.204.109@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0523282 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2260000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x22a0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x22a0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x22a0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x25e0000 Read this (4 bytes): BBBB 192.168.204.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg409-client.virtnet /mnt/lustre2 (opts:) PASS 209a (27s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:57:19 (1782482239) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0216573 s, 378 kB/s the very first read read with warm cache PASS 209b (4s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:57:23 (1782482243) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287725 s, 36.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x254 lmm_fid: [0x200000404:0x254:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x254 lmm_fid: [0x200000404:0x254:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x361:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834516 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-26 09:57:24.000000000 -0400 Modify: 2026-06-26 09:57:24.000000000 -0400 Change: 2026-06-26 09:57:25.000000000 -0400 Birth: 2026-06-26 09:57:23.000000000 -0400 PASS 210a (5s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:57:28 (1782482248) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:57:30 (1782482250) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.491158 s, 83.4 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0439819 s, 93.1 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 26 09:57 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x256:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 26 09:57 /mnt/lustre/f211.sanity-flr PASS 211 (6s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 09:57:36 (1782482256) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (5s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 09:57:42 (1782482262) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (6s) == sanity-flr test complete, duration 1571 sec =========== 09:57:47 (1782482267) === sanity-flr: start cleanup 09:57:48 (1782482268) === === sanity-flr: finish cleanup 09:57:53 (1782482273) ===