-----============= acceptance-small: sanity-flr ============----- Mon Mar 2 12:11:07 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg213-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 12:11:32 (1772471492) === oleg213-client.virtnet: executing check_config_client /mnt/lustre oleg213-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg213-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8d1c19489000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d1c19489000.idle_timeout=debug disable quota as required oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 12:12:16 (1772471536) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7588] == sanity-flr test 0a: lfs mirror create with -N option == 12:12:19 (1772471539) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 12:12:38 (1772471558) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 12:12:42 (1772471562) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 12:12:46 (1772471566) PASS 0d (33s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 12:13:19 (1772471599) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 12:13:23 (1772471603) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 12:13:28 (1772471608) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8d1c19489000.stats=clear osc.lustre-OST0001-osc-ffff8d1c19489000.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 12:13:38 (1772471618) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 12:13:51 (1772471631) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 12:14:02 (1772471642) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 12:14:07 (1772471647) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (13s) == sanity-flr test 2: create components from existing files ========================================================== 12:14:19 (1772471659) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 12:14:28 (1772471668) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 12:14:32 (1772471672) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 12:14:41 (1772471681) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (9s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 12:14:54 (1772471694) PASS 7 (9s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 12:15:03 (1772471703) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 1.56531 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190719 s, 5.5 MB/s PASS 21 (14s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 12:15:17 (1772471717) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.484933 s, 23.8 MB/s PASS 22 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 12:15:28 (1772471728) Stopping /mnt/lustre-ost1 (opts:) on oleg213-server oleg213-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d1c19489000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d1c19489000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg213-server: oleg213-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg213-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg213-server oleg213-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d1c19489000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d1c19489000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg213-server: oleg213-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg213-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (77s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 12:16:46 (1772471806) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.702914 s, 14.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg213-server oleg213-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d1c19489000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d1c19489000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg213-server: oleg213-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg213-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (49s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 12:17:34 (1772471854) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg213-server oleg213-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d1c19489000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d1c19489000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg213-server: oleg213-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg213-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg213-server oleg213-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d1c19489000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d1c19489000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg213-server: oleg213-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg213-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (88s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 12:19:02 (1772471942) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.62267 s, 9.3 MB/s -rw-r--r-- 1 root root 52428800 Mar 2 12:19 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.119s user 0m0.002s sys 0m0.154s Stopping /mnt/lustre-ost1 (opts:) on oleg213-server oleg213-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d1c19489000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d1c19489000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.967s user 0m0.001s sys 0m0.160s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg213-server: oleg213-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg213-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg213-server oleg213-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d1c19489000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d1c19489000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.592s user 0m0.001s sys 0m0.189s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg213-server: oleg213-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg213-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg213-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d1c19489000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d1c19489000.ost_server_uuid in FULL state after 0 sec PASS 33b (98s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 12:20:40 (1772472040) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 12:20:45 (1772472045) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 12:20:49 (1772472049) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 12:20:54 (1772472054) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 12:21:02 (1772472062) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.693391 s, 18.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 12:21:16 (1772472076) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203313 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431325 s, 24.3 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 12:21:25 (1772472085) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 12:21:35 (1772472095) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000977992 s, 1.0 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00204996 s, 500 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00155015 s, 661 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000883765 s, 1.2 MB/s flv=10 olv=10 PASS 36d (13s) == sanity-flr test 37: mirror I/O API verification ======= 12:21:48 (1772472108) create test files with size 15756570 .. c9777a64443a7109a8ee52b54ed81ec2 - c661c8e5ada8cf24586d11a02fa3cdd8 - 959a6dcefeda5d5593bb57f6db584bd1 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre PASS 37 (32s) == sanity-flr test 38: resync ============================ 12:22:20 (1772472140) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (18s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 12:22:39 (1772472159) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 12:22:46 (1772472166) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130676 s, 8.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755338 s, 13.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 12:22:57 (1772472177) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.242189 s, 8.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.502973 s, 8.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.283655 s, 7.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (22s) == sanity-flr test 42: lfs mirror verify ================= 12:23:20 (1772472200) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (5s) == sanity-flr test 43a: mirror pick on write ============= 12:23:24 (1772472204) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 12:23:28 (1772472208) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578982 s, 18.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (10s) == sanity-flr test 44a: lfs mirror split check =========== 12:23:38 (1772472218) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 12:23:42 (1772472222) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 12:23:53 (1772472233) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.36434 s, 28.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (16s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 12:24:09 (1772472249) llite.lustre-ffff8d1c030d5800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.384296 s, 27.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8d1c02cc0800.xattr_cache=1 PASS 44e (18s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 12:24:27 (1772472267) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (47s) == sanity-flr test 46: Verify setstripe --copy option ==== 12:25:14 (1772472314) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (45s) == sanity-flr test 47: Verify mirror obj alloc =========== 12:25:59 (1772472359) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 12:26:03 (1772472363) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.303133 s, 10.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.233055 s, 13.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x30e172d2 (stale) Mirror 2: 0x30e172d2 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: ee9ac07a8a017c9a1b16586775aa8404 - mirror 1 checksum: c1fb746f130c8b1fa1b6f05195590975 - mirror 2 checksum: ee9ac07a8a017c9a1b16586775aa8404 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Mar 2 12:26:06 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Mar 2 12:26:06 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: ee9ac07a8a017c9a1b16586775aa8404 - mirror 1 checksum: c1fb746f130c8b1fa1b6f05195590975 - mirror 2 checksum: c1fb746f130c8b1fa1b6f05195590975 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Mar 2 12:26:06 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Mar 2 12:26:06 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 12:26:16 (1772472376) Failing mds1 on oleg213-server Stopping /mnt/lustre-mds1 (opts:) on oleg213-server 12:26:22 (1772472382) shut down facet: mds1 facet_host: oleg213-server facet_failover_host: oleg213-server Failover mds1 to oleg213-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-MDT0000 12:26:54 (1772472414) targets are mounted 12:26:54 (1772472414) facet_failover done oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383925 s, 27.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b:0x0] } PASS 50A (56s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 12:27:12 (1772472432) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 12:27:16 (1772472436) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 12:27:20 (1772472440) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0143231 s, 1.1 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 12:27:29 (1772472449) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 12:27:33 (1772472453) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (5s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 12:27:38 (1772472458) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (30s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 12:28:09 (1772472489) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (34s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 12:28:42 (1772472522) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 12:29:06 (1772472546) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (27s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 12:29:33 (1772472573) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (10s) == sanity-flr test 70: mirror create and split race ====== 12:29:43 (1772472583) mirror create pid 52348 mirror split pid 52349 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52348 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52349 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 12:30:51 (1772472651) Seed set to 2654 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1772472654.303638 mapwrite 0x002911 thru 0x008d8a (0x647a bytes) 000002 1772472654.466989 mapwrite 0x038056 thru 0x03ffff (0x7faa bytes) 000004 1772472654.614169 read 0x00c12b thru 0x017230 (0xb106 bytes) 000005 1772472654.627532 read 0x0151ab thru 0x020f5d (0xbdb3 bytes) 000007 1772472654.657035 mapread 0x036e53 thru 0x03ffff (0x91ad bytes) 000009 1772472654.664671 mapread 0x0277aa thru 0x02ae0e (0x3665 bytes) 000011 1772472654.668210 mapread 0x01ea44 thru 0x02a01b (0xb5d8 bytes) 000013 1772472654.675013 mapwrite 0x029893 thru 0x02b93c (0x20aa bytes) truncating to largest ever: 0x35af6 000014 1772472654.796427 trunc from 0x040000 to 0x035af5 (0xa50a bytes) 000015 1772472654.862206 mapwrite 0x03f391 thru 0x03ffff (0xc6f bytes) 000017 1772472655.078201 mapread 0x01b1f0 thru 0x01b3ec (0x1fd bytes) 000018 1772472655.085694 mapwrite 0x01fc3b thru 0x02e90e (0xecd4 bytes) 000019 1772472655.240978 read 0x0065d9 thru 0x015576 (0xef9e bytes) 000020 1772472655.247060 mapwrite 0x0369db thru 0x03dbbe (0x71e4 bytes) 000021 1772472655.417512 write 0x03a4cc thru 0x03ffff (0x5b34 bytes) 000027 1772472655.435052 mapwrite 0x00bfed thru 0x018885 (0xc899 bytes) 000032 1772472655.549247 mapread 0x019ae3 thru 0x028778 (0xec96 bytes) 000034 1772472655.553144 write 0x039f2f thru 0x03a0ed (0x1bf bytes) 000035 1772472655.560451 mapwrite 0x0199e9 thru 0x0239a4 (0x9fbc bytes) 000036 1772472655.760006 mapwrite 0x006bd5 thru 0x0071c6 (0x5f2 bytes) 000038 1772472655.865055 read 0x039c37 thru 0x03b1a4 (0x156e bytes) 000041 1772472655.869259 trunc from 0x040000 to 0x030200 (0xfdff bytes) 000043 1772472655.905874 mapwrite 0x017ca4 thru 0x027a26 (0xfd83 bytes) 000045 1772472655.962896 mapwrite 0x033d7b thru 0x034383 (0x609 bytes) 000046 1772472656.046418 write 0x0007a4 thru 0x00f9df (0xf23c bytes) 000048 1772472656.060172 mapwrite 0x037797 thru 0x03ffff (0x8869 bytes) 000053 1772472656.217385 read 0x00ef42 thru 0x010483 (0x1542 bytes) 000054 1772472656.218962 mapread 0x0210fb thru 0x02f5f1 (0xe4f7 bytes) 000055 1772472656.225893 MIRROR_EXTEND to 2 mirrors 000056 1772472657.114904 mapwrite 0x03e36c thru 0x03ffff (0x1c94 bytes) 000059 1772472657.640287 trunc from 0x040000 to 0x00ebc8 (0x31437 bytes) 000060 1772472657.739829 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xebc9): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xebc9): Mirror 2: 0xd5e3025 (stale) CRC-32 checksum value for chunk [0, 0xebc9): Mirror 1: 0xd5e3025 Mirror 3: 0xd5e3025 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xebc9: skipped 000061 1772472659.249219 write 0x009317 thru 0x00c1d3 (0x2ebd bytes) 000064 1772472659.345106 read 0x00577c thru 0x0088e4 (0x3169 bytes) 000065 1772472659.353585 read 0x004667 thru 0x009204 (0x4b9e bytes) 000066 1772472659.371277 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xebc9): Mirror 3: 0xd5e3025 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xebc9): Mirror 3: 0xd5e3025 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xebc9: skipped 000069 1772472659.897045 mapread 0x006f9d thru 0x00ebc8 (0x7c2c bytes) 000070 1772472659.960428 mapwrite 0x023a6c thru 0x0278e4 (0x3e79 bytes) 000072 1772472660.156065 trunc from 0x0278e5 to 0x016db3 (0x10b31 bytes) 000079 1772472660.189202 read 0x007a36 thru 0x00fc77 (0x8242 bytes) 000080 1772472660.197435 mapwrite 0x013276 thru 0x0176f7 (0x4482 bytes) 000081 1772472660.448496 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) 000082 1772472660.781395 read 0x016529 thru 0x0176f7 (0x11cf bytes) 000086 1772472660.808317 trunc from 0x0176f8 to 0x02db3a (0x16443 bytes) 000087 1772472660.874213 mapread 0x020ee7 thru 0x020eea (0x4 bytes) 000093 1772472660.896172 mapread 0x0142ef thru 0x01f3f4 (0xb106 bytes) 000094 1772472660.948242 MIRROR_EXTEND to 2 mirrors 000096 1772472662.595849 write 0x0194d4 thru 0x019f3f (0xa6c bytes) truncating to largest ever: 0x3b4ec 000098 1772472662.841095 trunc from 0x02db3b to 0x03b4eb (0xd9b1 bytes) 000099 1772472662.931644 mapread 0x01477b thru 0x01bbaf (0x7435 bytes) 000100 1772472662.982468 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) 000102 1772472663.345285 mapread 0x01dbe1 thru 0x0216a3 (0x3ac3 bytes) 000107 1772472663.401082 mapwrite 0x01d5ec thru 0x0235da (0x5fef bytes) 000108 1772472663.519868 trunc from 0x03b4ec to 0x02bea7 (0xf644 bytes) 000110 1772472663.572427 trunc from 0x02bea8 to 0x03196c (0x5ac5 bytes) 000111 1772472663.620534 mapread 0x0191f2 thru 0x02368e (0xa49d bytes) 000112 1772472663.626820 trunc from 0x03196d to 0x0231a1 (0xe7cb bytes) 000113 1772472663.675991 MIRROR_EXTEND to 2 mirrors 000114 1772472665.274763 mapwrite 0x017e33 thru 0x0233e9 (0xb5b7 bytes) 000115 1772472665.744987 write 0x03eec1 thru 0x03ffff (0x113f bytes) 000116 1772472665.757861 mapwrite 0x003e23 thru 0x010784 (0xc962 bytes) 000117 1772472666.000158 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) 000121 1772472666.471430 read 0x005720 thru 0x010592 (0xae73 bytes) 000123 1772472666.496325 mapread 0x020e91 thru 0x028695 (0x7805 bytes) 000124 1772472666.573825 mapread 0x00e434 thru 0x0180b7 (0x9c84 bytes) 000126 1772472666.632423 trunc from 0x040000 to 0x021171 (0x1ee8e bytes) 000129 1772472666.687877 mapwrite 0x029372 thru 0x02f384 (0x6013 bytes) 000130 1772472666.935519 write 0x006951 thru 0x016239 (0xf8e9 bytes) 000132 1772472666.954255 mapwrite 0x0243fe thru 0x02e04f (0x9c52 bytes) 000133 1772472667.118562 trunc from 0x02f385 to 0x00a64f (0x24d35 bytes) 000136 1772472667.206301 mapread 0x0055f7 thru 0x00948b (0x3e95 bytes) 000138 1772472667.214494 mapwrite 0x024e4c thru 0x030a86 (0xbc3b bytes) 000139 1772472667.480270 mapread 0x01e88e thru 0x02103b (0x27ae bytes) 000140 1772472667.481614 trunc from 0x030a87 to 0x00415e (0x2c928 bytes) 000141 1772472667.547037 mapwrite 0x017aa2 thru 0x018368 (0x8c7 bytes) 000144 1772472667.816159 trunc from 0x018369 to 0x03472c (0x1c3c4 bytes) 000146 1772472667.855473 trunc from 0x03472d to 0x03210b (0x2621 bytes) 000147 1772472667.918792 write 0x028d8e thru 0x033c73 (0xaee6 bytes) 000148 1772472668.038929 write 0x018c8e thru 0x01f0b6 (0x6429 bytes) 000149 1772472668.076449 mapwrite 0x03c9bf thru 0x03ed63 (0x23a5 bytes) 000152 1772472668.412287 write 0x0322f8 thru 0x03ffff (0xdd08 bytes) 000154 1772472668.424683 mapread 0x008818 thru 0x00aa7a (0x2263 bytes) 000157 1772472668.489092 mapread 0x0135dd thru 0x01d991 (0xa3b5 bytes) 000161 1772472668.507091 mapwrite 0x0143eb thru 0x01e4e7 (0xa0fd bytes) 000162 1772472668.619891 read 0x03852e thru 0x0393eb (0xebe bytes) 000164 1772472668.644204 MIRROR_EXTEND to 2 mirrors 000165 1772472670.561715 read 0x028f12 thru 0x02e328 (0x5417 bytes) 000166 1772472670.623733 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) 000169 1772472670.976484 trunc from 0x040000 to 0x02d4cb (0x12b34 bytes) 000171 1772472671.078265 trunc from 0x02d4cc to 0x02a6b6 (0x2e15 bytes) 000172 1772472671.109115 read 0x00d2ca thru 0x014dd5 (0x7b0c bytes) 000175 1772472671.123986 write 0x00c364 thru 0x014a51 (0x86ee bytes) 000177 1772472671.152096 mapread 0x002828 thru 0x00e100 (0xb8d9 bytes) 000179 1772472671.244297 mapwrite 0x0325b3 thru 0x038ca0 (0x66ee bytes) 000180 1772472671.576461 mapread 0x016816 thru 0x01963a (0x2e25 bytes) 000181 1772472671.613236 read 0x027269 thru 0x036427 (0xf1bf bytes) 000182 1772472671.643844 write 0x02e45e thru 0x03d3d4 (0xef77 bytes) 000183 1772472671.645828 mapread 0x03509f thru 0x03d3d4 (0x8336 bytes) 000186 1772472671.667198 write 0x03b311 thru 0x03ffff (0x4cef bytes) 000189 1772472671.713793 read 0x006129 thru 0x010fe4 (0xaebc bytes) 000191 1772472671.715683 trunc from 0x040000 to 0x03569b (0xa964 bytes) 000192 1772472671.795383 read 0x02af43 thru 0x03569b (0xa759 bytes) 000194 1772472671.802589 write 0x032b01 thru 0x03ffff (0xd4ff bytes) truncating to largest ever: 0x3d17c 000195 1772472671.812227 trunc from 0x040000 to 0x03d17b (0x2e84 bytes) 000197 1772472671.873502 mapread 0x0392dc thru 0x03d17b (0x3ea0 bytes) 000202 1772472671.884201 trunc from 0x03d17c to 0x023887 (0x198f4 bytes) 000206 1772472671.944990 mapread 0x00da73 thru 0x013eca (0x6458 bytes) 000210 1772472671.954545 write 0x0336ce thru 0x03eba4 (0xb4d7 bytes) 000211 1772472671.978421 read 0x00fc3f thru 0x01c9b5 (0xcd77 bytes) 000213 1772472671.989880 mapwrite 0x029c5b thru 0x038eba (0xf260 bytes) 000215 1772472672.200186 trunc from 0x03eba5 to 0x02271a (0x1c48a bytes) 000216 1772472672.282681 MIRROR_EXTEND to 2 mirrors 000217 1772472674.057518 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) 000221 1772472674.441748 MIRROR_EXTEND to 2 mirrors 000223 1772472675.827101 mapread 0x00c0a3 thru 0x00e01f (0x1f7d bytes) 000224 1772472675.894644 read 0x01dbf6 thru 0x02271a (0x4b25 bytes) 000226 1772472675.944725 trunc from 0x02271b to 0x00512e (0x1d5ec bytes) 000227 1772472676.065978 read 0x0021a3 thru 0x0037d5 (0x1633 bytes) 000228 1772472676.079816 read 0x0027ef thru 0x00512e (0x2940 bytes) 000229 1772472676.080733 mapwrite 0x013c2a thru 0x020845 (0xcc1c bytes) 000231 1772472676.191255 read 0x011c8a thru 0x018ff3 (0x736a bytes) 000236 1772472676.197213 write 0x03d93c thru 0x03ffff (0x26c4 bytes) 000237 1772472676.204004 trunc from 0x040000 to 0x01d229 (0x22dd6 bytes) 000238 1772472676.225141 write 0x01dbad thru 0x02c2eb (0xe73f bytes) 000240 1772472676.233336 trunc from 0x02c2ec to 0x002e8d (0x2945e bytes) 000241 1772472676.263954 mapwrite 0x01eed6 thru 0x02d41b (0xe546 bytes) 000246 1772472676.369638 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2d41c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d41c): Mirror 2: 0xf7ac019b (stale) CRC-32 checksum value for chunk [0, 0x2d41c): Mirror 1: 0x10ad60db Mirror 3: 0x10ad60db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d41c: skipped 000247 1772472677.540164 write 0x010946 thru 0x0119ea (0x10a5 bytes) 000252 1772472677.666068 mapread 0x024073 thru 0x028432 (0x43c0 bytes) 000256 1772472677.686245 mapwrite 0x01adb9 thru 0x02255b (0x77a3 bytes) 000258 1772472677.765021 read 0x014923 thru 0x016a30 (0x210e bytes) 000259 1772472677.770821 mapread 0x02a0be thru 0x02ce4b (0x2d8e bytes) 000262 1772472677.776457 mapwrite 0x038e8c thru 0x03ffff (0x7174 bytes) 000264 1772472677.950756 mapread 0x02270b thru 0x02e4db (0xbdd1 bytes) truncating to largest ever: 0x3eb02 000267 1772472677.964388 trunc from 0x040000 to 0x03eb01 (0x14fe bytes) 000268 1772472678.000719 write 0x016202 thru 0x0216c4 (0xb4c3 bytes) 000269 1772472678.001910 mapread 0x006eef thru 0x00842c (0x153e bytes) 000272 1772472678.022808 mapread 0x0191ac thru 0x020d8c (0x7be1 bytes) 000274 1772472678.029820 MIRROR_RESYNC 3 mirrors 000275 1772472678.972181 write 0x00cde2 thru 0x01a103 (0xd322 bytes) 000278 1772472679.047772 mapwrite 0x0277d4 thru 0x0332bf (0xbaec bytes) 000279 1772472679.115065 read 0x00b7a6 thru 0x00c4be (0xd19 bytes) 000280 1772472679.120856 trunc from 0x03eb02 to 0x00c17f (0x32982 bytes) 000281 1772472679.170987 mapread 0x0075c1 thru 0x00c17f (0x4bbf bytes) 000282 1772472679.198905 mapwrite 0x025a52 thru 0x029cc6 (0x4275 bytes) 000283 1772472679.305486 write 0x015f9d thru 0x01e0be (0x8122 bytes) 000284 1772472679.316596 read 0x024506 thru 0x029cc6 (0x57c1 bytes) 000285 1772472679.317395 read 0x00b0df thru 0x0111d1 (0x60f3 bytes) 000286 1772472679.339848 MIRROR_RESYNC 3 mirrors 000287 1772472680.211610 mapwrite 0x008aa6 thru 0x00c79d (0x3cf8 bytes) 000289 1772472680.499038 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29cc7): Mirror 2: 0x72f34c84 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29cc7): Mirror 2: 0x72f34c84 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29cc7: skipped truncating to largest ever: 0x3ef5a 000290 1772472680.969132 trunc from 0x029cc7 to 0x03ef59 (0x15293 bytes) 000291 1772472681.042705 mapread 0x008a98 thru 0x00d03c (0x45a5 bytes) 000293 1772472681.068998 mapread 0x000a7b thru 0x00a734 (0x9cba bytes) 000294 1772472681.076850 mapread 0x017fe7 thru 0x027625 (0xf63f bytes) 000296 1772472681.101196 mapwrite 0x00e5b0 thru 0x01e1cf (0xfc20 bytes) 000300 1772472681.197647 read 0x03c317 thru 0x03ef59 (0x2c43 bytes) 000302 1772472681.206713 mapread 0x02f252 thru 0x036657 (0x7406 bytes) 000303 1772472681.222028 trunc from 0x03ef5a to 0x022374 (0x1cbe5 bytes) 000305 1772472681.243220 write 0x010f69 thru 0x020e5c (0xfef4 bytes) 000306 1772472681.246863 write 0x012628 thru 0x0183ac (0x5d85 bytes) 000308 1772472681.249691 write 0x036696 thru 0x038960 (0x22cb bytes) 000309 1772472681.253511 read 0x01cb62 thru 0x02ab08 (0xdfa7 bytes) 000310 1772472681.266927 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) 000311 1772472681.440044 mapwrite 0x0300f3 thru 0x039840 (0x974e bytes) 000313 1772472681.536748 trunc from 0x039841 to 0x025911 (0x13f2f bytes) 000314 1772472681.578274 mapwrite 0x03967b thru 0x03ffff (0x6985 bytes) 000315 1772472681.704992 read 0x02a411 thru 0x031f7b (0x7b6b bytes) 000318 1772472681.746856 write 0x01fd16 thru 0x02c6a8 (0xc993 bytes) 000320 1772472681.771182 mapread 0x03b988 thru 0x03ffff (0x4678 bytes) 000322 1772472681.777724 mapread 0x01ca78 thru 0x02bc32 (0xf1bb bytes) 000324 1772472681.781618 read 0x03d18d thru 0x03ffff (0x2e73 bytes) 000327 1772472681.783874 write 0x01336c thru 0x021b9a (0xe82f bytes) 000329 1772472681.799093 write 0x006460 thru 0x00c660 (0x6201 bytes) 000330 1772472681.800164 trunc from 0x040000 to 0x005788 (0x3a877 bytes) 000331 1772472681.812903 trunc from 0x005789 to 0x02484a (0x1f0c2 bytes) 000333 1772472681.828221 mapread 0x00f6ce thru 0x016dd5 (0x7708 bytes) 000334 1772472681.854148 mapread 0x021873 thru 0x02484a (0x2fd8 bytes) 000336 1772472681.867377 trunc from 0x02484b to 0x01e8c5 (0x5f85 bytes) 000338 1772472681.879169 read 0x004ebe thru 0x007d7e (0x2ec1 bytes) 000339 1772472681.885807 trunc from 0x01e8c6 to 0x03b53e (0x1cc79 bytes) 000341 1772472681.902629 trunc from 0x03b53f to 0x02391d (0x17c21 bytes) 000342 1772472681.927342 mapwrite 0x00d81d thru 0x012bf6 (0x53da bytes) 000343 1772472681.981100 MIRROR_EXTEND to 2 mirrors 000344 1772472682.621098 read 0x00b9f2 thru 0x017ff0 (0xc5ff bytes) 000345 1772472682.631505 mapwrite 0x02305e thru 0x029162 (0x6105 bytes) 000347 1772472682.776828 trunc from 0x029163 to 0x004a9b (0x246c7 bytes) 000348 1772472682.800056 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x4a9c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4a9c): Mirror 2: 0x1e3a9f32 (stale) CRC-32 checksum value for chunk [0, 0x4a9c): Mirror 1: 0x1e3a9f32 Mirror 3: 0x1e3a9f32 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4a9c: skipped 000351 1772472683.537968 write 0x02712b thru 0x034d7b (0xdc51 bytes) 000353 1772472683.594389 mapwrite 0x03131b thru 0x035f86 (0x4c6c bytes) 000354 1772472683.646188 trunc from 0x035f87 to 0x02baee (0xa498 bytes) 000357 1772472683.663437 mapwrite 0x017841 thru 0x0268d5 (0xf095 bytes) 000361 1772472683.735665 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2baef): Mirror 3: 0x68641d50 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2baef): Mirror 3: 0x68641d50 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2baef: skipped 000362 1772472683.919208 read 0x01c242 thru 0x0204f7 (0x42b6 bytes) 000363 1772472683.925709 trunc from 0x02baef to 0x03c471 (0x10983 bytes) 000364 1772472683.936086 mapread 0x030630 thru 0x03c2c2 (0xbc93 bytes) 000366 1772472683.958783 trunc from 0x03c472 to 0x01a4c6 (0x21fab bytes) 000367 1772472683.975767 mapread 0x00c742 thru 0x013e72 (0x7731 bytes) 000369 1772472683.987670 trunc from 0x01a4c7 to 0x02f944 (0x1547e bytes) 000370 1772472683.999608 trunc from 0x02f945 to 0x00557e (0x2a3c6 bytes) 000372 1772472684.017267 trunc from 0x00557f to 0x008378 (0x2dfa bytes) 000374 1772472684.025763 read 0x000e11 thru 0x002461 (0x1651 bytes) 000375 1772472684.030728 read 0x002124 thru 0x008378 (0x6255 bytes) 000376 1772472684.033668 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x8379): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8379): Mirror 3: 0x8eba2419 (stale) CRC-32 checksum value for chunk [0, 0x8379): Mirror 1: 0x8eba2419 Mirror 4: 0x8eba2419 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8379: skipped 000377 1772472684.521264 mapread 0x000503 thru 0x0025ff (0x20fd bytes) 000381 1772472684.546822 read 0x002156 thru 0x008378 (0x6223 bytes) 000383 1772472684.560472 mapwrite 0x03d551 thru 0x03ffff (0x2aaf bytes) 000384 1772472684.662484 read 0x00adf8 thru 0x013498 (0x86a1 bytes) 000385 1772472684.671322 mapread 0x0296ea thru 0x02a099 (0x9b0 bytes) 000386 1772472684.672843 mapwrite 0x0309ca thru 0x036732 (0x5d69 bytes) 000388 1772472684.713946 mapwrite 0x004128 thru 0x009ffb (0x5ed4 bytes) 000389 1772472684.770233 read 0x03508d thru 0x036f90 (0x1f04 bytes) 000390 1772472684.772277 write 0x01636b thru 0x017c5f (0x18f5 bytes) 000391 1772472684.784053 trunc from 0x040000 to 0x037c36 (0x83c9 bytes) 000393 1772472684.818194 mapread 0x00e739 thru 0x017d22 (0x95ea bytes) 000395 1772472684.819799 mapread 0x0287d2 thru 0x029473 (0xca2 bytes) 000396 1772472684.820860 write 0x00a8e3 thru 0x0189b8 (0xe0d6 bytes) 000397 1772472684.822336 read 0x02ceff thru 0x02fb4b (0x2c4d bytes) 000399 1772472684.823190 read 0x02c891 thru 0x037c36 (0xb3a6 bytes) 000400 1772472684.827017 MIRROR_RESYNC 3 mirrors 000401 1772472685.772117 mapread 0x00c22e thru 0x01bea6 (0xfc79 bytes) 000402 1772472685.810956 write 0x02fa81 thru 0x03c3a0 (0xc920 bytes) 000404 1772472685.933741 MIRROR_RESYNC 3 mirrors 000405 1772472687.180950 mapwrite 0x01704b thru 0x01cf65 (0x5f1b bytes) 000406 1772472687.427405 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3c3a1): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c3a1): Mirror 3: 0x4eacea77 (stale) Mirror 4: 0x4eacea77 (stale) CRC-32 checksum value for chunk [0, 0x3c3a1): Mirror 1: 0x96e82bc9 Mirror 5: 0x96e82bc9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c3a1: skipped 000407 1772472688.821765 mapread 0x000fd9 thru 0x006505 (0x552d bytes) 000408 1772472688.945541 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c3a1): Mirror 4: 0x4eacea77 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c3a1): Mirror 4: 0x4eacea77 (stale) CRC-32 checksum value for chunk [0, 0x3c3a1): Mirror 1: 0x96e82bc9 Mirror 5: 0x96e82bc9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c3a1: skipped 000409 1772472689.988830 read 0x02dca3 thru 0x03c3a0 (0xe6fe bytes) 000410 1772472690.025769 trunc from 0x03c3a1 to 0x00ba31 (0x3096f bytes) 000412 1772472690.114763 trunc from 0x00ba32 to 0x02aa3d (0x1f00c bytes) 000418 1772472690.129233 read 0x00f830 thru 0x019d75 (0xa546 bytes) 000419 1772472690.136403 read 0x016ffb thru 0x019876 (0x287c bytes) 000420 1772472690.137002 mapread 0x004b1f thru 0x00ff9c (0xb47e bytes) 000421 1772472690.138806 write 0x02e4fc thru 0x036800 (0x8305 bytes) 000422 1772472690.141774 read 0x01b880 thru 0x027429 (0xbbaa bytes) 000423 1772472690.146786 mapwrite 0x03da88 thru 0x03ffff (0x2578 bytes) 000425 1772472690.208832 MIRROR_RESYNC 3 mirrors 000426 1772472690.823167 mapwrite 0x01b471 thru 0x027311 (0xbea1 bytes) 000428 1772472690.972576 trunc from 0x040000 to 0x023066 (0x1cf99 bytes) 000429 1772472691.046501 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23067): Mirror 5: 0x29dd5fa1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23067): Mirror 5: 0x29dd5fa1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23067: skipped 000431 1772472691.305951 mapread 0x013ccc thru 0x01ede2 (0xb117 bytes) 000432 1772472691.335077 MIRROR_RESYNC 2 mirrors 000433 1772472691.968645 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) 000434 1772472692.120839 mapwrite 0x0336e0 thru 0x03ec3e (0xb55f bytes) 000435 1772472692.218244 mapread 0x020dd4 thru 0x022a96 (0x1cc3 bytes) 000438 1772472692.235629 mapwrite 0x038f13 thru 0x03ffff (0x70ed bytes) 000439 1772472692.377781 trunc from 0x040000 to 0x018318 (0x27ce7 bytes) 000443 1772472692.425138 trunc from 0x018319 to 0x00ddad (0xa56b bytes) 000448 1772472692.488781 read 0x00b07a thru 0x00ddad (0x2d34 bytes) 000450 1772472692.505370 write 0x032601 thru 0x03332e (0xd2e bytes) 000452 1772472692.518314 mapwrite 0x00459c thru 0x0049af (0x414 bytes) 000453 1772472692.650546 mapread 0x017db6 thru 0x01b723 (0x396e bytes) 000454 1772472692.685279 write 0x02952b thru 0x031aff (0x85d5 bytes) 000455 1772472692.706112 mapread 0x00c023 thru 0x011133 (0x5111 bytes) 000456 1772472692.730065 MIRROR_EXTEND to 2 mirrors 000459 1772472693.931612 trunc from 0x03332f to 0x03ad88 (0x7a5a bytes) 000460 1772472694.103084 trunc from 0x03ad89 to 0x03009f (0xace9 bytes) 000461 1772472694.184173 trunc from 0x0300a0 to 0x037600 (0x7561 bytes) 000462 1772472694.239226 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37601): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37601): Mirror 2: 0x4e541a5a (stale) CRC-32 checksum value for chunk [0, 0x37601): Mirror 1: 0xe22a354d Mirror 3: 0xe22a354d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37601: skipped 000464 1772472695.355134 read 0x036d7e thru 0x037600 (0x883 bytes) 000465 1772472695.388047 mapwrite 0x0221a2 thru 0x022ee4 (0xd43 bytes) 000466 1772472695.698074 mapread 0x027edf thru 0x0312ed (0x940f bytes) 000468 1772472695.751976 mapread 0x00c684 thru 0x015eb6 (0x9833 bytes) 000472 1772472695.823227 mapread 0x00717d thru 0x0099ad (0x2831 bytes) 000473 1772472695.824615 read 0x026348 thru 0x02d149 (0x6e02 bytes) 000474 1772472695.843517 write 0x01b698 thru 0x027659 (0xbfc2 bytes) 000475 1772472695.853172 MIRROR_RESYNC 3 mirrors 000481 1772472696.747287 write 0x02b58f thru 0x031894 (0x6306 bytes) 000482 1772472696.834320 trunc from 0x037601 to 0x0281c7 (0xf439 bytes) 000483 1772472696.867003 mapwrite 0x0026b3 thru 0x00b8ee (0x923c bytes) 000484 1772472696.954220 trunc from 0x0281c8 to 0x016e28 (0x1139f bytes) 000485 1772472696.980002 mapwrite 0x03eff9 thru 0x03ffff (0x1007 bytes) 000486 1772472697.077086 read 0x013f78 thru 0x01cb3d (0x8bc6 bytes) 000487 1772472697.100462 MIRROR_RESYNC 3 mirrors 000488 1772472698.540089 read 0x030813 thru 0x035d7b (0x5569 bytes) 000491 1772472698.561963 mapread 0x00d1f4 thru 0x01a7f9 (0xd606 bytes) 000492 1772472698.613117 trunc from 0x040000 to 0x00d212 (0x32ded bytes) 000493 1772472698.755449 trunc from 0x00d213 to 0x023de3 (0x16bd1 bytes) 000494 1772472698.784216 write 0x035ab7 thru 0x03ffff (0xa549 bytes) 000495 1772472698.785915 trunc from 0x040000 to 0x00bcb2 (0x3434d bytes) 000499 1772472698.809717 read 0x0008cb thru 0x005d2c (0x5462 bytes) 000500 1772472698.821962 mapwrite 0x023c6e thru 0x02de6c (0xa1ff bytes) 000501 1772472698.904609 mapread 0x0108f6 thru 0x01c1f0 (0xb8fb bytes) 000502 1772472698.912062 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2de6d): Mirror 3: 0x2f5c9f04 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2de6d): Mirror 3: 0x2f5c9f04 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2de6d: skipped 000504 1772472699.175801 trunc from 0x02de6d to 0x013cb8 (0x1a1b4 bytes) 000505 1772472699.210058 mapread 0x00084d thru 0x0086ef (0x7ea3 bytes) 000506 1772472699.258716 mapwrite 0x01b9d9 thru 0x027330 (0xb958 bytes) 000507 1772472699.412948 write 0x001772 thru 0x00c069 (0xa8f8 bytes) 000509 1772472699.414955 write 0x031018 thru 0x038775 (0x775e bytes) 000510 1772472699.421057 MIRROR_RESYNC 2 mirrors 000512 1772472700.075088 mapread 0x0239ee thru 0x02b148 (0x775b bytes) 000514 1772472700.099125 read 0x01ed91 thru 0x027d5f (0x8fcf bytes) 000516 1772472700.101037 write 0x007fbc thru 0x00eb1c (0x6b61 bytes) 000520 1772472700.206776 write 0x03e98d thru 0x03ffff (0x1673 bytes) 000521 1772472700.209837 write 0x033ad4 thru 0x0395b9 (0x5ae6 bytes) 000522 1772472700.222226 mapread 0x01b63e thru 0x028e5e (0xd821 bytes) 000523 1772472700.246732 write 0x012eaf thru 0x017b06 (0x4c58 bytes) 000524 1772472700.256181 trunc from 0x040000 to 0x02599e (0x1a661 bytes) 000525 1772472700.320504 mapread 0x00bb36 thru 0x00eb6f (0x303a bytes) 000527 1772472700.322221 mapwrite 0x029944 thru 0x02df44 (0x4601 bytes) 000528 1772472700.500961 read 0x0268e1 thru 0x02df44 (0x7664 bytes) 000529 1772472700.502984 trunc from 0x02df45 to 0x0122c4 (0x1bc80 bytes) 000531 1772472700.534681 mapwrite 0x01fdc8 thru 0x0226e3 (0x291c bytes) 000533 1772472700.636258 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x226e4): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x226e4): Mirror 3: 0xc256c131 (stale) CRC-32 checksum value for chunk [0, 0x226e4): Mirror 1: 0xab2487ed Mirror 4: 0xab2487ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x226e4: skipped 000534 1772472701.691021 write 0x031188 thru 0x03aee8 (0x9d61 bytes) 000535 1772472701.808299 trunc from 0x03aee9 to 0x006cc5 (0x34223 bytes) 000538 1772472701.840105 mapwrite 0x01fab7 thru 0x02d632 (0xdb7c bytes) 000540 1772472702.045195 write 0x022951 thru 0x0282b6 (0x5966 bytes) 000541 1772472702.055764 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d633): Mirror 4: 0xbec6685f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d633): Mirror 4: 0xbec6685f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d633: skipped 000542 1772472702.460312 write 0x00daa1 thru 0x00edd0 (0x1330 bytes) 000543 1772472702.492977 read 0x00bca6 thru 0x018ee2 (0xd23d bytes) 000544 1772472702.519980 read 0x026d25 thru 0x02d632 (0x690e bytes) 000545 1772472702.553813 mapread 0x0196b0 thru 0x0242dc (0xac2d bytes) 000547 1772472702.563694 trunc from 0x02d633 to 0x02c6b8 (0xf7a bytes) 000550 1772472702.618209 read 0x0050e4 thru 0x0114a4 (0xc3c1 bytes) 000551 1772472702.621271 mapread 0x009f06 thru 0x014cc6 (0xadc1 bytes) 000553 1772472702.627207 trunc from 0x02c6b9 to 0x00d35c (0x1f35c bytes) 000554 1772472702.656465 write 0x022231 thru 0x02b4e7 (0x92b7 bytes) 000555 1772472702.660010 mapread 0x01894e thru 0x027fd3 (0xf686 bytes) 000556 1772472702.681989 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) 000562 1772472702.970926 MIRROR_EXTEND to 2 mirrors 000565 1772472703.876122 MIRROR_EXTEND to 3 mirrors 000566 1772472704.717640 trunc from 0x02b4e8 to 0x001020 (0x2a4c7 bytes) 000568 1772472704.919416 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1021): Mirror 3: 0xd915df45 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1021): Mirror 3: 0xd915df45 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1021: skipped 000569 1772472705.403219 trunc from 0x001021 to 0x013889 (0x12869 bytes) 000570 1772472705.426097 mapread 0x002e34 thru 0x004ee4 (0x20b1 bytes) 000572 1772472705.462114 mapread 0x0132f5 thru 0x013889 (0x595 bytes) 000574 1772472705.489080 MIRROR_RESYNC 2 mirrors 000576 1772472706.521787 mapread 0x003930 thru 0x004a38 (0x1109 bytes) 000578 1772472706.581763 MIRROR_EXTEND to 3 mirrors 000579 1772472707.564894 write 0x013075 thru 0x021761 (0xe6ed bytes) 000580 1772472707.663843 read 0x007ee7 thru 0x00dd38 (0x5e52 bytes) 000582 1772472707.687162 MIRROR_RESYNC 3 mirrors 000584 1772472708.567187 mapwrite 0x030f0b thru 0x0323dc (0x14d2 bytes) 000585 1772472708.748632 read 0x02e59a thru 0x0323dc (0x3e43 bytes) 000587 1772472708.750273 mapwrite 0x03feec thru 0x03ffff (0x114 bytes) 000588 1772472708.890092 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe6ccde01 (stale) Mirror 4: 0xe6ccde01 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x25ce91b4 Mirror 5: 0x25ce91b4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000589 1772472710.552079 write 0x02be33 thru 0x037d24 (0xbef2 bytes) 000591 1772472710.808429 mapread 0x022945 thru 0x024015 (0x16d1 bytes) 000593 1772472710.842404 write 0x0362ea thru 0x03ffff (0x9d16 bytes) 000595 1772472710.848165 read 0x01c60b thru 0x02a020 (0xda16 bytes) 000596 1772472710.849114 write 0x0355cb thru 0x03e2df (0x8d15 bytes) 000597 1772472710.861555 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe6ccde01 (stale) Mirror 4: 0xe6ccde01 (stale) Mirror 5: 0x25ce91b4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x68a1c80 Mirror 6: 0x68a1c80 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000602 1772472712.081999 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe6ccde01 (stale) Mirror 4: 0xe6ccde01 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe6ccde01 (stale) Mirror 4: 0xe6ccde01 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x68a1c80 Mirror 6: 0x68a1c80 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000603 1772472712.949497 write 0x00685b thru 0x00981a (0x2fc0 bytes) 000604 1772472713.093774 read 0x03c046 thru 0x03ffff (0x3fba bytes) 000605 1772472713.102592 mapread 0x031459 thru 0x03b2fd (0x9ea5 bytes) 000607 1772472713.124413 mapread 0x0084cb thru 0x011afe (0x9634 bytes) 000609 1772472713.144266 mapwrite 0x02cad5 thru 0x03244c (0x5978 bytes) 000611 1772472713.254946 write 0x031580 thru 0x03ffff (0xea80 bytes) 000612 1772472713.258616 mapwrite 0x007aa0 thru 0x014511 (0xca72 bytes) 000613 1772472713.364037 trunc from 0x040000 to 0x025798 (0x1a867 bytes) 000616 1772472713.394545 mapread 0x0196df thru 0x025798 (0xc0ba bytes) 000617 1772472713.417455 mapread 0x022465 thru 0x025798 (0x3334 bytes) 000619 1772472713.420113 write 0x035b1f thru 0x03c58b (0x6a6d bytes) 000620 1772472713.425940 read 0x006640 thru 0x013192 (0xcb53 bytes) 000621 1772472713.432904 MIRROR_RESYNC 4 mirrors 000622 1772472714.878978 write 0x00b019 thru 0x00efb0 (0x3f98 bytes) 000623 1772472714.975078 read 0x0054f9 thru 0x00f3da (0x9ee2 bytes) 000624 1772472714.995136 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3c58c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c58c): Mirror 3: 0x321c462b (stale) Mirror 4: 0x321c462b (stale) Mirror 6: 0x321c462b (stale) CRC-32 checksum value for chunk [0, 0x3c58c): Mirror 1: 0x7237f0bb Mirror 7: 0x7237f0bb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c58c: skipped 000626 1772472716.190058 write 0x02391f thru 0x02ed0f (0xb3f1 bytes) 000630 1772472716.284265 MIRROR_RESYNC 5 mirrors 000632 1772472717.495889 read 0x01b2f0 thru 0x0241a5 (0x8eb6 bytes) 000636 1772472717.507555 mapwrite 0x020ff0 thru 0x030bb6 (0xfbc7 bytes) 000639 1772472717.736887 mapread 0x01a451 thru 0x02a2ea (0xfe9a bytes) 000641 1772472717.757823 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3c58c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c58c): Mirror 3: 0xbaaf48e0 (stale) Mirror 4: 0xbaaf48e0 (stale) Mirror 6: 0xbaaf48e0 (stale) Mirror 7: 0xbaaf48e0 (stale) CRC-32 checksum value for chunk [0, 0x3c58c): Mirror 1: 0x8314c220 Mirror 8: 0x8314c220 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c58c: skipped 000642 1772472719.058585 mapread 0x033b83 thru 0x039217 (0x5695 bytes) 000643 1772472719.112514 write 0x027bd3 thru 0x02cedc (0x530a bytes) 000644 1772472719.218019 read 0x00021d thru 0x00fcf6 (0xfada bytes) 000645 1772472719.238676 read 0x03c080 thru 0x03c58b (0x50c bytes) 000647 1772472719.254675 read 0x01a7c5 thru 0x01d99d (0x31d9 bytes) 000649 1772472719.258869 trunc from 0x03c58c to 0x00544d (0x3713e bytes) 000650 1772472719.298612 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x544e): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x544e): Mirror 3: 0x9b67736f (stale) Mirror 4: 0x9b67736f (stale) Mirror 6: 0x9b67736f (stale) Mirror 7: 0x9b67736f (stale) Mirror 8: 0x9b67736f (stale) CRC-32 checksum value for chunk [0, 0x544e): Mirror 1: 0x9b67736f Mirror 9: 0x9b67736f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x544e: skipped 000651 1772472720.264377 trunc from 0x00544e to 0x033434 (0x2dfe7 bytes) 000652 1772472720.459579 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x33435): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33435): Mirror 3: 0xd816277 (stale) Mirror 4: 0xd816277 (stale) Mirror 6: 0xd816277 (stale) Mirror 7: 0xd816277 (stale) Mirror 8: 0xb004b59a (stale) Mirror 9: 0xf32a9c00 (stale) CRC-32 checksum value for chunk [0, 0x33435): Mirror 1: 0xf32a9c00 Mirror 10: 0xf32a9c00 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33435: skipped 000653 1772472721.602498 trunc from 0x033435 to 0x03e7b4 (0xb380 bytes) 000654 1772472721.696950 trunc from 0x03e7b5 to 0x021505 (0x1d2af bytes) 000656 1772472721.718555 read 0x009022 thru 0x009386 (0x365 bytes) 000657 1772472721.724653 mapread 0x00a083 thru 0x0169bf (0xc93d bytes) 000658 1772472721.754760 read 0x008e15 thru 0x01471b (0xb907 bytes) 000659 1772472721.755482 write 0x02ba77 thru 0x02c8cd (0xe57 bytes) 000661 1772472721.757068 mapread 0x0098f0 thru 0x00d389 (0x3a9a bytes) 000665 1772472721.761553 write 0x028d08 thru 0x030baa (0x7ea3 bytes) 000667 1772472721.774492 mapwrite 0x0046b1 thru 0x005146 (0xa96 bytes) 000669 1772472721.928093 MIRROR_RESYNC 8 mirrors 000671 1772472723.560172 write 0x0261e5 thru 0x0326d8 (0xc4f4 bytes) 000673 1772472723.843143 trunc from 0x0326d9 to 0x01736e (0x1b36a bytes) 000674 1772472723.917501 mapwrite 0x01af19 thru 0x01efc3 (0x40ab bytes) 000678 1772472724.074696 mapwrite 0x0127e5 thru 0x016472 (0x3c8e bytes) 000680 1772472724.133097 mapread 0x002b1f thru 0x008b0d (0x5fef bytes) 000682 1772472724.145780 mapwrite 0x037dd7 thru 0x03ffff (0x8229 bytes) 000683 1772472724.228245 trunc from 0x040000 to 0x025a83 (0x1a57c bytes) 000685 1772472724.254018 write 0x0045cb thru 0x00f0b8 (0xaaee bytes) 000686 1772472724.256814 mapread 0x01fd54 thru 0x025a83 (0x5d30 bytes) 000689 1772472724.276915 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25a84): Mirror 3: 0x62bb38d5 (stale) Mirror 4: 0x62bb38d5 (stale) Mirror 6: 0x62bb38d5 (stale) Mirror 7: 0x62bb38d5 (stale) Mirror 8: 0x62bb38d5 (stale) Mirror 10: 0x62bb38d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25a84): Mirror 3: 0x62bb38d5 (stale) Mirror 4: 0x62bb38d5 (stale) Mirror 6: 0x62bb38d5 (stale) Mirror 7: 0x62bb38d5 (stale) Mirror 8: 0x62bb38d5 (stale) Mirror 10: 0x62bb38d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25a84: skipped 000691 1772472724.825548 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25a84): Mirror 3: 0x62bb38d5 (stale) Mirror 4: 0x62bb38d5 (stale) Mirror 7: 0x62bb38d5 (stale) Mirror 8: 0x62bb38d5 (stale) Mirror 10: 0x62bb38d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25a84): Mirror 3: 0x62bb38d5 (stale) Mirror 4: 0x62bb38d5 (stale) Mirror 7: 0x62bb38d5 (stale) Mirror 8: 0x62bb38d5 (stale) Mirror 10: 0x62bb38d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25a84: skipped 000692 1772472725.306154 mapwrite 0x01e28e thru 0x02315a (0x4ecd bytes) 000694 1772472725.387063 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25a84): Mirror 3: 0x62bb38d5 (stale) Mirror 4: 0x62bb38d5 (stale) Mirror 8: 0x62bb38d5 (stale) Mirror 10: 0x62bb38d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25a84): Mirror 3: 0x62bb38d5 (stale) Mirror 4: 0x62bb38d5 (stale) Mirror 8: 0x62bb38d5 (stale) Mirror 10: 0x62bb38d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25a84: skipped 000695 1772472725.807906 trunc from 0x025a84 to 0x0196f5 (0xc38e bytes) 000696 1772472725.834391 mapread 0x0118ef thru 0x011f0d (0x61f bytes) 000697 1772472725.843045 mapread 0x00e098 thru 0x010a2d (0x2996 bytes) 000698 1772472725.859261 write 0x03591b thru 0x03c456 (0x6b3c bytes) 000699 1772472725.866484 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3c457): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c457): Mirror 3: 0xaf2c5b12 (stale) Mirror 4: 0xaf2c5b12 (stale) Mirror 8: 0xaf2c5b12 (stale) Mirror 10: 0xaf2c5b12 (stale) CRC-32 checksum value for chunk [0, 0x3c457): Mirror 1: 0x78076657 Mirror 11: 0x78076657 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c457: skipped 000701 1772472727.083979 trunc from 0x03c457 to 0x030e4d (0xb609 bytes) 000702 1772472727.207452 mapread 0x0139b3 thru 0x020ed1 (0xd51f bytes) 000704 1772472727.249079 read 0x005e7e thru 0x00fd2e (0x9eb1 bytes) 000705 1772472727.260058 mapwrite 0x012207 thru 0x020fee (0xede8 bytes) 000706 1772472727.343991 mapread 0x01ce58 thru 0x026cc9 (0x9e72 bytes) 000707 1772472727.345322 read 0x01934a thru 0x01c768 (0x341f bytes) 000708 1772472727.346004 trunc from 0x030e4e to 0x01b472 (0x159db bytes) 000709 1772472727.398803 write 0x023ee4 thru 0x02ab8d (0x6caa bytes) 000710 1772472727.401914 trunc from 0x02ab8e to 0x008780 (0x2240d bytes) 000712 1772472727.422925 read 0x001e6e thru 0x004a0e (0x2ba1 bytes) 000713 1772472727.432862 mapwrite 0x000758 thru 0x005f87 (0x5830 bytes) 000714 1772472727.478461 write 0x00d39e thru 0x013ddd (0x6a40 bytes) 000716 1772472727.483379 trunc from 0x013dde to 0x006c7e (0xd15f bytes) 000718 1772472727.507811 mapwrite 0x0376cd thru 0x03fe6d (0x87a1 bytes) 000719 1772472727.669694 write 0x0352ae thru 0x03ffff (0xad52 bytes) 000720 1772472727.678701 mapwrite 0x0192fe thru 0x02366b (0xa36e bytes) 000721 1772472727.759836 trunc from 0x040000 to 0x004668 (0x3b997 bytes) 000722 1772472727.781309 trunc from 0x004669 to 0x030123 (0x2babb bytes) 000724 1772472727.822716 trunc from 0x030124 to 0x0116c4 (0x1ea5f bytes) 000725 1772472727.859603 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x116c5): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x116c5): Mirror 3: 0x127eb434 (stale) Mirror 4: 0x127eb434 (stale) Mirror 8: 0x127eb434 (stale) Mirror 10: 0x127eb434 (stale) Mirror 11: 0x451f9e3a (stale) CRC-32 checksum value for chunk [0, 0x116c5): Mirror 1: 0x7427268c Mirror 12: 0x7427268c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x116c5: skipped 000726 1772472728.897040 trunc from 0x0116c5 to 0x0381f2 (0x26b2e bytes) 000729 1772472729.067841 MIRROR_RESYNC 7 mirrors 000731 1772472730.392820 mapwrite 0x0113f6 thru 0x016876 (0x5481 bytes) 000733 1772472730.596469 trunc from 0x0381f3 to 0x03d45d (0x526b bytes) 000736 1772472730.636490 mapwrite 0x03601c thru 0x03bf1c (0x5f01 bytes) 000739 1772472730.711485 write 0x027110 thru 0x0283e2 (0x12d3 bytes) 000740 1772472730.728572 write 0x03585d thru 0x03b127 (0x58cb bytes) 000741 1772472730.729889 mapwrite 0x01f2e0 thru 0x028b4e (0x986f bytes) 000742 1772472730.802303 trunc from 0x03d45e to 0x02f1d8 (0xe285 bytes) 000743 1772472730.828654 write 0x005e13 thru 0x007542 (0x1730 bytes) 000744 1772472730.840904 read 0x0273c0 thru 0x02f1d8 (0x7e19 bytes) 000745 1772472730.842438 read 0x00b0b0 thru 0x00f23c (0x418d bytes) 000747 1772472730.850941 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f1d9): Mirror 4: 0x8187e675 (stale) Mirror 8: 0x8187e675 (stale) Mirror 10: 0x8187e675 (stale) Mirror 11: 0x8187e675 (stale) Mirror 12: 0x8187e675 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f1d9): Mirror 4: 0x8187e675 (stale) Mirror 8: 0x8187e675 (stale) Mirror 10: 0x8187e675 (stale) Mirror 11: 0x8187e675 (stale) Mirror 12: 0x8187e675 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f1d9: skipped 000748 1772472731.256515 mapread 0x01ee81 thru 0x029fb9 (0xb139 bytes) 000749 1772472731.288073 mapread 0x0024c2 thru 0x00a0d0 (0x7c0f bytes) 000751 1772472731.299657 read 0x01207c thru 0x01777e (0x5703 bytes) 000753 1772472731.310945 read 0x018355 thru 0x025b55 (0xd801 bytes) 000754 1772472731.319037 mapread 0x0068a1 thru 0x0101f6 (0x9956 bytes) 000755 1772472731.321266 read 0x023dc3 thru 0x02f1d8 (0xb416 bytes) 000756 1772472731.326023 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2f1d9): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f1d9): Mirror 4: 0x8187e675 (stale) Mirror 8: 0x8187e675 (stale) Mirror 10: 0x8187e675 (stale) Mirror 11: 0x8187e675 (stale) Mirror 12: 0x8187e675 (stale) CRC-32 checksum value for chunk [0, 0x2f1d9): Mirror 1: 0x1546a100 Mirror 13: 0x1546a100 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f1d9: skipped 000759 1772472732.354721 mapwrite 0x00973f thru 0x00e1c4 (0x4a86 bytes) 000760 1772472732.585197 mapread 0x01c9da thru 0x01dcdd (0x1304 bytes) 000761 1772472732.599289 read 0x0185ba thru 0x023ce6 (0xb72d bytes) 000762 1772472732.600279 trunc from 0x02f1d9 to 0x017fec (0x171ec bytes) 000763 1772472732.630483 mapwrite 0x0149ec thru 0x01e98e (0x9fa3 bytes) 000764 1772472732.755003 MIRROR_RESYNC 7 mirrors 000765 1772472733.939396 write 0x00d667 thru 0x010590 (0x2f2a bytes) 000766 1772472734.043036 read 0x010aec thru 0x014c0a (0x411f bytes) 000767 1772472734.054243 mapwrite 0x028d8c thru 0x02b3b8 (0x262d bytes) 000769 1772472734.179989 write 0x01b443 thru 0x01c296 (0xe54 bytes) 000770 1772472734.199765 write 0x015e34 thru 0x01cabe (0x6c8b bytes) 000771 1772472734.207919 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2b3b9): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b3b9): Mirror 4: 0x6224458a (stale) Mirror 8: 0x6224458a (stale) Mirror 10: 0x6224458a (stale) Mirror 11: 0x6224458a (stale) Mirror 12: 0x6224458a (stale) Mirror 13: 0x6224458a (stale) CRC-32 checksum value for chunk [0, 0x2b3b9): Mirror 1: 0x44891101 Mirror 14: 0x44891101 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b3b9: skipped 000772 1772472735.482864 trunc from 0x02b3b9 to 0x015e54 (0x15564 bytes) 000774 1772472735.713688 MIRROR_RESYNC 8 mirrors 000777 1772472737.454638 read 0x0006c2 thru 0x006e18 (0x6757 bytes) 000779 1772472737.468923 mapread 0x011bee thru 0x015e54 (0x4267 bytes) 000780 1772472737.490633 mapread 0x00602b thru 0x010045 (0xa01b bytes) 000783 1772472737.502528 read 0x000ec2 thru 0x00fecf (0xf00e bytes) 000787 1772472737.504652 write 0x0204f7 thru 0x02e16a (0xdc74 bytes) 000788 1772472737.580159 mapread 0x00da67 thru 0x01b1dc (0xd776 bytes) 000791 1772472737.600175 mapwrite 0x005921 thru 0x012bb3 (0xd293 bytes) 000792 1772472737.707275 trunc from 0x02e16b to 0x02ddac (0x3be bytes) 000794 1772472737.733269 trunc from 0x02ddad to 0x02179d (0xc60f bytes) 000795 1772472737.757745 mapwrite 0x02ae38 thru 0x02fd34 (0x4efd bytes) 000796 1772472737.855836 trunc from 0x02fd35 to 0x014ca9 (0x1b08b bytes) 000797 1772472737.880965 mapread 0x000074 thru 0x006637 (0x65c4 bytes) 000798 1772472737.885007 read 0x00b89a thru 0x014ca9 (0x9410 bytes) 000799 1772472737.888709 mapwrite 0x02201d thru 0x029bbe (0x7ba2 bytes) 000800 1772472737.985664 write 0x01cb2f thru 0x0265d2 (0x9aa4 bytes) 000801 1772472737.987817 read 0x01e83f thru 0x029bbe (0xb380 bytes) 000802 1772472737.989698 mapwrite 0x00b364 thru 0x00ff62 (0x4bff bytes) 000807 1772472738.047404 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x29bbf): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29bbf): Mirror 4: 0xb44ebf7 (stale) Mirror 8: 0xb44ebf7 (stale) Mirror 10: 0xb44ebf7 (stale) Mirror 11: 0xb44ebf7 (stale) Mirror 12: 0xb44ebf7 (stale) Mirror 13: 0xb44ebf7 (stale) Mirror 14: 0xb44ebf7 (stale) CRC-32 checksum value for chunk [0, 0x29bbf): Mirror 1: 0x5b22921 Mirror 15: 0x5b22921 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29bbf: skipped 000808 1772472739.548916 mapread 0x029664 thru 0x029afc (0x499 bytes) 000809 1772472739.594872 read 0x018c04 thru 0x019b5e (0xf5b bytes) 000812 1772472739.658731 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29bbf): Mirror 4: 0xb44ebf7 (stale) Mirror 8: 0xb44ebf7 (stale) Mirror 10: 0xb44ebf7 (stale) Mirror 11: 0xb44ebf7 (stale) Mirror 13: 0xb44ebf7 (stale) Mirror 14: 0xb44ebf7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29bbf): Mirror 4: 0xb44ebf7 (stale) Mirror 8: 0xb44ebf7 (stale) Mirror 10: 0xb44ebf7 (stale) Mirror 11: 0xb44ebf7 (stale) Mirror 13: 0xb44ebf7 (stale) Mirror 14: 0xb44ebf7 (stale) CRC-32 checksum value for chunk [0, 0x29bbf): Mirror 1: 0x5b22921 Mirror 15: 0x5b22921 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29bbf: skipped 000813 1772472740.941756 trunc from 0x029bbf to 0x0064ee (0x236d0 bytes) 000814 1772472741.163086 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x64ef): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x64ef): Mirror 4: 0xc45a7f57 (stale) Mirror 8: 0xc45a7f57 (stale) Mirror 10: 0xc45a7f57 (stale) Mirror 11: 0xc45a7f57 (stale) Mirror 13: 0xc45a7f57 (stale) Mirror 14: 0xc45a7f57 (stale) Mirror 15: 0xcb02f5df (stale) CRC-32 checksum value for chunk [0, 0x64ef): Mirror 1: 0xcb02f5df Mirror 16: 0xcb02f5df lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x64ef: skipped 000815 1772472742.961309 mapwrite 0x034667 thru 0x03fe76 (0xb810 bytes) 000816 1772472743.533577 read 0x003b8d thru 0x0108b1 (0xcd25 bytes) 000817 1772472743.575140 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3fe77): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe77): Mirror 4: 0x3c170a82 (stale) Mirror 8: 0x3c170a82 (stale) Mirror 10: 0x3c170a82 (stale) Mirror 11: 0x3c170a82 (stale) Mirror 13: 0x3c170a82 (stale) Mirror 14: 0x3c170a82 (stale) Mirror 15: 0x218c3a8e (stale) Mirror 16: 0x890199f2 (stale) CRC-32 checksum value for chunk [0, 0x3fe77): Mirror 1: 0xcdc53900 Mirror 17: 0xcdc53900 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe77: skipped 000818 1772472744.993630 trunc from 0x03fe77 to 0x03de6d (0x2009 bytes) 000819 1772472745.133055 write 0x01479d thru 0x0183f7 (0x3c5b bytes) 000820 1772472745.141326 write 0x00642d thru 0x013bea (0xd7be bytes) 000822 1772472745.157194 write 0x01a0d0 thru 0x028015 (0xdf46 bytes) 000823 1772472745.175386 mapwrite 0x006334 thru 0x006f33 (0xc00 bytes) 000824 1772472745.290920 read 0x03c8a3 thru 0x03de6d (0x15cb bytes) 000825 1772472745.301138 write 0x02f5ff thru 0x03d4d0 (0xded2 bytes) 000827 1772472745.314209 mapread 0x0135cf thru 0x0143df (0xe11 bytes) 000828 1772472745.315652 read 0x011ed1 thru 0x01c7a3 (0xa8d3 bytes) 000830 1772472745.317883 mapwrite 0x0006f7 thru 0x0007a2 (0xac bytes) 000832 1772472745.421324 MIRROR_RESYNC 10 mirrors 000834 1772472747.898535 mapread 0x028771 thru 0x02d5ed (0x4e7d bytes) 000836 1772472747.931815 write 0x00250c thru 0x00e738 (0xc22d bytes) 000838 1772472748.063505 trunc from 0x03de6e to 0x02ac82 (0x131eb bytes) 000840 1772472748.092601 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ac83): Mirror 4: 0xe5a9b891 (stale) Mirror 8: 0xe5a9b891 (stale) Mirror 10: 0xe5a9b891 (stale) Mirror 11: 0xe5a9b891 (stale) Mirror 13: 0xe5a9b891 (stale) Mirror 14: 0xe5a9b891 (stale) Mirror 15: 0xe5a9b891 (stale) Mirror 17: 0xe5a9b891 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ac83): Mirror 4: 0xe5a9b891 (stale) Mirror 8: 0xe5a9b891 (stale) Mirror 10: 0xe5a9b891 (stale) Mirror 11: 0xe5a9b891 (stale) Mirror 13: 0xe5a9b891 (stale) Mirror 14: 0xe5a9b891 (stale) Mirror 15: 0xe5a9b891 (stale) Mirror 17: 0xe5a9b891 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ac83: skipped 000841 1772472748.722109 mapread 0x018515 thru 0x020afb (0x85e7 bytes) 000843 1772472748.760047 mapread 0x01f3db thru 0x02ac82 (0xb8a8 bytes) 000844 1772472748.775110 mapread 0x02a862 thru 0x02ac82 (0x421 bytes) 000845 1772472748.776838 mapwrite 0x014ac0 thru 0x01d40e (0x894f bytes) 000846 1772472748.877053 trunc from 0x02ac83 to 0x02dec0 (0x323e bytes) 000849 1772472748.910217 mapread 0x017aa1 thru 0x022686 (0xabe6 bytes) 000851 1772472748.920500 MIRROR_RESYNC 9 mirrors 000853 1772472750.870579 read 0x01a332 thru 0x025538 (0xb207 bytes) 000854 1772472750.882525 trunc from 0x02dec1 to 0x026879 (0x7647 bytes) 000855 1772472751.007038 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2687a): Mirror 4: 0x344d76d4 (stale) Mirror 8: 0x344d76d4 (stale) Mirror 10: 0x344d76d4 (stale) Mirror 13: 0x344d76d4 (stale) Mirror 14: 0x344d76d4 (stale) Mirror 15: 0x344d76d4 (stale) Mirror 17: 0x344d76d4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2687a): Mirror 4: 0x344d76d4 (stale) Mirror 8: 0x344d76d4 (stale) Mirror 10: 0x344d76d4 (stale) Mirror 13: 0x344d76d4 (stale) Mirror 14: 0x344d76d4 (stale) Mirror 15: 0x344d76d4 (stale) Mirror 17: 0x344d76d4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2687a: skipped 000856 1772472751.646199 write 0x0045d5 thru 0x007868 (0x3294 bytes) 000857 1772472751.659541 mapread 0x01bf44 thru 0x01f88e (0x394b bytes) 000860 1772472751.683173 write 0x036d1c thru 0x037a29 (0xd0e bytes) 000861 1772472751.691894 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37a2a): Mirror 4: 0xf3ce254b (stale) Mirror 8: 0xf3ce254b (stale) Mirror 10: 0xf3ce254b (stale) Mirror 13: 0xf3ce254b (stale) Mirror 15: 0xf3ce254b (stale) Mirror 17: 0xf3ce254b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a2a): Mirror 4: 0xf3ce254b (stale) Mirror 8: 0xf3ce254b (stale) Mirror 10: 0xf3ce254b (stale) Mirror 13: 0xf3ce254b (stale) Mirror 15: 0xf3ce254b (stale) Mirror 17: 0xf3ce254b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a2a: skipped 000862 1772472752.478116 write 0x0354f0 thru 0x03c818 (0x7329 bytes) 000863 1772472752.486613 read 0x009702 thru 0x017b8c (0xe48b bytes) 000865 1772472752.506741 MIRROR_RESYNC 7 mirrors 000866 1772472753.995481 mapread 0x0350ed thru 0x03c818 (0x772c bytes) 000867 1772472754.024320 write 0x02ce9f thru 0x03299e (0x5b00 bytes) 000872 1772472754.113542 mapread 0x00671b thru 0x00dae8 (0x73ce bytes) 000873 1772472754.132780 mapread 0x021c9d thru 0x0266cf (0x4a33 bytes) 000874 1772472754.161426 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c819): Mirror 4: 0x1567529a (stale) Mirror 10: 0x1567529a (stale) Mirror 13: 0x1567529a (stale) Mirror 15: 0x1567529a (stale) Mirror 17: 0x1567529a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c819): Mirror 4: 0x1567529a (stale) Mirror 10: 0x1567529a (stale) Mirror 13: 0x1567529a (stale) Mirror 15: 0x1567529a (stale) Mirror 17: 0x1567529a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c819: skipped 000877 1772472754.722959 mapread 0x023392 thru 0x02a991 (0x7600 bytes) 000878 1772472754.749203 trunc from 0x03c819 to 0x03cf8c (0x774 bytes) 000882 1772472754.766959 trunc from 0x03cf8d to 0x02de58 (0xf134 bytes) 000883 1772472754.793239 MIRROR_RESYNC 6 mirrors 000885 1772472755.976284 mapwrite 0x031ed6 thru 0x039a81 (0x7bac bytes) 000887 1772472756.155018 trunc from 0x039a82 to 0x0031e1 (0x368a0 bytes) 000889 1772472756.185747 write 0x03cec9 thru 0x03ffff (0x3137 bytes) 000890 1772472756.188087 mapread 0x008718 thru 0x00c449 (0x3d32 bytes) 000891 1772472756.198224 read 0x0322ad thru 0x03ed00 (0xca54 bytes) 000899 1772472756.213109 read 0x02bfbd thru 0x0396c7 (0xd70b bytes) 000900 1772472756.223717 read 0x0139b7 thru 0x0211f0 (0xd83a bytes) 000901 1772472756.229733 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1c5301d7 (stale) Mirror 10: 0x1c5301d7 (stale) Mirror 13: 0x1c5301d7 (stale) Mirror 15: 0x1c5301d7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1c5301d7 (stale) Mirror 10: 0x1c5301d7 (stale) Mirror 13: 0x1c5301d7 (stale) Mirror 15: 0x1c5301d7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000904 1772472757.004842 read 0x000b14 thru 0x00902d (0x851a bytes) 000905 1772472757.043884 MIRROR_RESYNC 5 mirrors 000906 1772472758.693400 mapread 0x036ef9 thru 0x037cfb (0xe03 bytes) 000908 1772472758.733502 MIRROR_EXTEND to 6 mirrors 000915 1772472760.145678 mapread 0x030308 thru 0x031092 (0xd8b bytes) 000916 1772472760.215060 trunc from 0x040000 to 0x03dfba (0x2045 bytes) 000920 1772472760.431591 mapread 0x014be8 thru 0x01dea1 (0x92ba bytes) 000921 1772472760.488802 write 0x00a6d6 thru 0x0145d8 (0x9f03 bytes) 000922 1772472760.503119 trunc from 0x03dfbb to 0x0197f7 (0x247c3 bytes) 000923 1772472760.538735 mapread 0x00ef53 thru 0x0197f7 (0xa8a5 bytes) 000924 1772472760.542152 write 0x02d0d7 thru 0x03715c (0xa086 bytes) 000929 1772472760.569265 write 0x019af3 thru 0x01e385 (0x4893 bytes) 000930 1772472760.611210 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3715d): Mirror 4: 0x3c66fb1f (stale) Mirror 13: 0x3c66fb1f (stale) Mirror 15: 0x3c66fb1f (stale) Mirror 16: 0x3c66fb1f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3715d): Mirror 4: 0x3c66fb1f (stale) Mirror 13: 0x3c66fb1f (stale) Mirror 15: 0x3c66fb1f (stale) Mirror 16: 0x3c66fb1f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3715d: skipped 000932 1772472761.255665 mapread 0x00a47e thru 0x00e845 (0x43c8 bytes) 000933 1772472761.282022 mapread 0x012dc3 thru 0x01660d (0x384b bytes) 000937 1772472761.297472 trunc from 0x03715d to 0x0345ef (0x2b6d bytes) 000939 1772472761.320683 trunc from 0x0345f0 to 0x035eaa (0x18bb bytes) 000941 1772472761.345847 trunc from 0x035eab to 0x0383d0 (0x2526 bytes) 000944 1772472761.381817 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x383d1): Mirror 4: 0x1f7833b (stale) Mirror 15: 0x1f7833b (stale) Mirror 16: 0x1f7833b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x383d1): Mirror 4: 0x1f7833b (stale) Mirror 15: 0x1f7833b (stale) Mirror 16: 0x1f7833b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x383d1: skipped 000945 1772472761.913330 trunc from 0x0383d1 to 0x03613d (0x2293 bytes) 000947 1772472761.940081 mapwrite 0x01ac0a thru 0x01d968 (0x2d5f bytes) 000949 1772472762.110801 mapread 0x022098 thru 0x0264a4 (0x440d bytes) 000951 1772472762.132549 write 0x038a09 thru 0x038e8d (0x485 bytes) 000953 1772472762.137935 mapread 0x0321af thru 0x038e8d (0x6cdf bytes) 000954 1772472762.160424 read 0x01f41d thru 0x01f63d (0x221 bytes) 000955 1772472762.162108 mapwrite 0x0298aa thru 0x02c536 (0x2c8d bytes) 000957 1772472762.227707 MIRROR_RESYNC 4 mirrors 000958 1772472763.264956 read 0x02f04c thru 0x03822c (0x91e1 bytes) 000961 1772472763.279033 mapwrite 0x034af7 thru 0x039ab7 (0x4fc1 bytes) 000962 1772472763.540017 mapwrite 0x01f09d thru 0x020a35 (0x1999 bytes) 000963 1772472763.633691 read 0x02e000 thru 0x032bb2 (0x4bb3 bytes) 000967 1772472763.661119 mapread 0x0253e2 thru 0x033795 (0xe3b4 bytes) 000968 1772472763.664193 write 0x011eff thru 0x01d54a (0xb64c bytes) 000969 1772472763.667708 trunc from 0x039ab8 to 0x037b3d (0x1f7a bytes) 000971 1772472763.699940 read 0x01dba0 thru 0x022424 (0x4885 bytes) 000972 1772472763.700781 trunc from 0x037b3e to 0x01c729 (0x1b414 bytes) 000973 1772472763.726756 trunc from 0x01c72a to 0x03dc70 (0x21547 bytes) 000974 1772472763.753801 trunc from 0x03dc71 to 0x00aaa4 (0x331cc bytes) 000975 1772472763.782607 mapread 0x0053d9 thru 0x00aaa4 (0x56cc bytes) 000976 1772472763.784254 write 0x00844a thru 0x016c5a (0xe811 bytes) 000977 1772472763.789138 mapread 0x01470d thru 0x016c5a (0x254e bytes) 000979 1772472763.793127 mapread 0x009acf thru 0x0103aa (0x68dc bytes) 000980 1772472763.796093 trunc from 0x016c5b to 0x020165 (0x950b bytes) 000981 1772472763.826805 write 0x02cbc0 thru 0x034c3c (0x807d bytes) 000982 1772472763.835801 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34c3d): Mirror 15: 0x992fdaab (stale) Mirror 16: 0x992fdaab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34c3d): Mirror 15: 0x992fdaab (stale) Mirror 16: 0x992fdaab (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34c3d: skipped 000983 1772472764.366086 trunc from 0x034c3d to 0x01f4ad (0x1578f bytes) 000984 1772472764.402632 mapwrite 0x017c0d thru 0x022155 (0xa549 bytes) 000987 1772472764.669208 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x22156): Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22156): Mirror 15: 0x4bb5ba93 (stale) Mirror 16: 0x4bb5ba93 (stale) CRC-32 checksum value for chunk [0, 0x22156): Mirror 1: 0x9222ec26 Mirror 17: 0x9222ec26 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22156: skipped 000988 1772472765.808951 mapread 0x0131ea thru 0x01e7b8 (0xb5cf bytes) 000992 1772472766.020051 write 0x00078a thru 0x00a389 (0x9c00 bytes) 000993 1772472766.183986 read 0x0165fb thru 0x022155 (0xbb5b bytes) 000994 1772472766.201032 read 0x00fc54 thru 0x010682 (0xa2f bytes) 000996 1772472766.222831 read 0x010e74 thru 0x01563c (0x47c9 bytes) 000999 1772472766.228214 mapwrite 0x03bc35 thru 0x03ffff (0x43cb bytes) All operations completed A-OK! PASS 70a (121s) == sanity-flr test 71: check mirror extend parent fid ==== 12:32:52 (1772472772) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 12:32:57 (1772472777) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x204:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x208:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.363387 s, 5.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x204:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x205:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x208:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291017 s, 3.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x204:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x205:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x209:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x208:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x204:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x205:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x209:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x208:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x206:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (13s) == sanity-flr test 200a: stress test ===================== 12:33:09 (1772472789) Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 671296 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3214048 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4742688 .. Extending file size to 5453984 .. Extending file size to 8023520 .. Extending file size to 6294496 .. Extending file size to 7294336 .. Extending file size to 8451328 .. Extending file size to 7459424 .. Extending file size to 9059072 .. Extending file size to 8774240 .. done Extending file size to 8846528 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8996832 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 55374 55375 55376 55379 55380 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) 192.168.202.113@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg213-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (85s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 12:34:34 (1772472874) Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7476224 .. Extending file size to 7537216 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8550688 .. Extending file size to 8740416 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8819200 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8806240 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9036576 .. Extending file size to 8826592 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 57017 57018 57020 57022 57024 done 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) 192.168.202.113@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg213-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (98s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 12:36:12 (1772472972) Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3968 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 25600 3743744 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 18432 3750912 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3968 2204544 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 25600 3743744 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 18432 3750912 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.129742 s, 16.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.383025 s, 5.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x5a5a5a04 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 12:36:35 (1772472995) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.160969 s, 13.0 MB/s PASS 202 (10s) == sanity-flr test 203: mirror file preserve mirror ID === 12:36:45 (1772473005) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x21f:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x21f:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x220:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x21b:0x0] } PASS 203 (10s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 12:36:55 (1772473015) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.427142 s, 4.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x221:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x21c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x21d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (11s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 12:37:06 (1772473026) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.321093 s, 6.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x222:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x223:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x21e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x222:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x223:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x21e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x21f:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 12:37:16 (1772473036) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.292105 s, 7.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x224:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x220:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x224:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x220:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x221:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 12:37:52 (1772473072) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg213-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151865 s, 6.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x225:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x223:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x222:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x225:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x223:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x222:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x226:0x0] } PASS 204d (66s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 12:38:58 (1772473138) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg213-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x224:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x225:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.397824 s, 5.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x224:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x225:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x227:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x224:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x225:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x227:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (67s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 12:40:05 (1772473205) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg213-server mds-ost sync done. Creating new pool test_204f oleg213-server: Pool lustre.test_204f created Adding targets to pool oleg213-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg213-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.36048 s, 5.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x226:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x228:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x226:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x228:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg213-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg213-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg213-server: Pool lustre.test_204f destroyed PASS 204f (88s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 12:41:33 (1772473293) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 12:41:43 (1772473303) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 12:41:52 (1772473312) oleg213-server: Pool lustre.test_206 created Adding targets to pool oleg213-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg213-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg213-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg213-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg213-server: Pool lustre.test_206 destroyed PASS 206 (35s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 12:42:27 (1772473347) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980878 s, 10.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362947 s, 28.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xbe94a86d Mirror 3: 0xbe94a86d PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 12:42:41 (1772473361) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 12:42:45 (1772473365) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 12:42:50 (1772473370) Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.135483 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xe80000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2380000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2380000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2380000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1cb0000 Read this (4 bytes): BBBB 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) PASS 209a (45s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 12:43:35 (1772473415) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0420277 s, 195 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 12:43:43 (1772473423) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372634 s, 28.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1db lmm_fid: [0x200000404:0x1db:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1db lmm_fid: [0x200000404:0x1db:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x232:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834395 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-02 12:43:45.000000000 -0500 Modify: 2026-03-02 12:43:45.000000000 -0500 Change: 2026-03-02 12:43:46.000000000 -0500 Birth: 2026-03-02 12:43:44.000000000 -0500 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 12:43:53 (1772473433) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 12:43:57 (1772473437) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.856238 s, 47.8 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0781771 s, 52.4 kB/s size after second write -rw-r--r-- 1 root root 4096 Mar 2 12:44 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1dd:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Mar 2 12:44 /mnt/lustre/f211.sanity-flr PASS 211 (11s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 12:44:08 (1772473448) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 12:44:17 (1772473457) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1999 sec =========== 12:44:28 (1772473468) === sanity-flr: start cleanup 12:44:29 (1772473469) === === sanity-flr: finish cleanup 12:44:36 (1772473476) ===