-----============= acceptance-small: sanity-flr ============----- Tue Sep 29 22:46:32 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky use SANITY_FLR_ONLY='' SANITY_FLR_EXCEPT='' oleg436-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 22:46:51 (1790736411) === oleg436-client.virtnet: executing check_config_client /mnt/lustre oleg436-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg436-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9fb105448000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9fb105448000.idle_timeout=debug disable quota as required oleg436-server: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 22:47:29 (1790736449) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7612] == sanity-flr test 0a: lfs mirror create with -N option == 22:47:31 (1790736451) PASS 0a (20s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 22:47:51 (1790736471) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 22:47:55 (1790736475) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 22:47:59 (1790736479) PASS 0d (29s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 22:48:28 (1790736508) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 22:48:31 (1790736511) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 22:48:35 (1790736515) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9fb105448000.stats=clear osc.lustre-OST0001-osc-ffff9fb105448000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 22:48:42 (1790736522) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--ec-map] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 22:48:54 (1790736534) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 22:49:03 (1790736543) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 22:49:07 (1790736547) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 22:49:15 (1790736555) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 22:49:25 (1790736565) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 22:49:33 (1790736573) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 22:49:37 (1790736577) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 22:49:45 (1790736585) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 22:49:56 (1790736596) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 22:50:04 (1790736604) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 2.43097 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190286 s, 5.5 MB/s PASS 21 (13s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 22:50:17 (1790736617) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.351286 s, 47.8 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 22:50:25 (1790736625) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 22:50:36 (1790736636) Stopping /mnt/lustre-ost1 (opts:) on oleg436-server oleg436-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9fb105448000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fb105448000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg436-server: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg436-server: oleg436-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg436-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg436-server oleg436-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9fb105448000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fb105448000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg436-server: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg436-server: oleg436-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg436-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (75s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 22:51:51 (1790736711) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.655679 s, 16.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg436-server oleg436-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9fb105448000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fb105448000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg436-server: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg436-server: oleg436-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg436-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (52s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 22:52:43 (1790736763) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg436-server oleg436-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9fb105448000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fb105448000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg436-server: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg436-server: oleg436-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg436-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg436-server oleg436-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9fb105448000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fb105448000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg436-server: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg436-server: oleg436-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg436-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (89s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 22:54:13 (1790736853) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.51547 s, 9.5 MB/s -rw-r--r-- 1 root root 52428800 Sep 29 22:54 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.397s user 0m0.001s sys 0m0.182s Stopping /mnt/lustre-ost1 (opts:) on oleg436-server oleg436-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9fb105448000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fb105448000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.303s user 0m0.001s sys 0m0.187s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg436-server: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg436-server: oleg436-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg436-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg436-server oleg436-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9fb105448000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fb105448000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.402s user 0m0.000s sys 0m0.191s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg436-server: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg436-server: oleg436-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg436-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg436-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fb105448000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fb105448000.ost_server_uuid in FULL state after 0 sec PASS 33b (101s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 22:55:55 (1790736955) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 22:55:58 (1790736958) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 22:56:02 (1790736962) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 22:56:06 (1790736966) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 22:56:15 (1790736975) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.743623 s, 16.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 22:56:30 (1790736990) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345091 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275721 s, 38.0 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 22:56:38 (1790736998) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 22:56:47 (1790737007) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00137496 s, 745 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00334914 s, 306 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00395669 s, 259 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00200471 s, 511 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 22:56:58 (1790737018) create test files with size 15758509 .. cdea347cf56ea525d963fdbe3cd5098f - 18fe0eb219b500a80b782adbddbde55c - 75b8c48bc6f1d264434fbbffac84a0fd - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg436-client.virtnet /mnt/lustre (opts:) Starting client: oleg436-client.virtnet: -o user_xattr,flock 192.168.204.136@tcp:/lustre /mnt/lustre PASS 37 (29s) == sanity-flr test 38: resync ============================ 22:57:28 (1790737048) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (19s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 22:57:47 (1790737067) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 22:57:54 (1790737074) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120633 s, 8.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037805 s, 27.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xe0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 22:58:03 (1790737083) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.181417 s, 11.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.382768 s, 11.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.346018 s, 6.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (22s) == sanity-flr test 42: lfs mirror verify ================= 22:58:25 (1790737105) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 22:58:29 (1790737109) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 22:58:33 (1790737113) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550612 s, 19.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 22:58:42 (1790737122) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 22:58:46 (1790737126) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe7:0x0] } split mirror_id 2 id 131073 error lfs split: file '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' has 1 component, cannot split make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 22:58:58 (1790737138) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.308178 s, 34.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg436-client.virtnet /mnt/lustre (opts:) Starting client: oleg436-client.virtnet: -o user_xattr,flock 192.168.204.136@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 22:59:12 (1790737152) llite.lustre-ffff9fb1099e5000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.387381 s, 27.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg436-client.virtnet /mnt/lustre (opts:) Starting client: oleg436-client.virtnet: -o user_xattr,flock 192.168.204.136@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9fb1096cb000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 44f: lfs mirror split does not leak the victim file descriptor ========================================================== 22:59:27 (1790737167) error lfs split: create victim file failed: File exists PASS 44f (19s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 22:59:46 (1790737186) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (42s) == sanity-flr test 46: Verify setstripe --copy option ==== 23:00:28 (1790737228) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 23:01:10 (1790737270) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 23:01:13 (1790737273) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.25289 s, 12.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.135328 s, 23.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x80687992 (nosync,stale) Mirror 2: 0x80687992 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 0903d6baea44d8ecf9db6a77f10bb3ce - mirror 1 checksum: bc8bd1e9335df66ccc98fb3051de9066 - mirror 2 checksum: 0903d6baea44d8ecf9db6a77f10bb3ce - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-29T23:01:18-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x65 lmm_fid: [0x200000404:0x65:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x105:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-29T23:01:18-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x65 lmm_fid: [0x200000404:0x65:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x106:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 0903d6baea44d8ecf9db6a77f10bb3ce - mirror 1 checksum: bc8bd1e9335df66ccc98fb3051de9066 - mirror 2 checksum: bc8bd1e9335df66ccc98fb3051de9066 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-29T23:01:18-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x65 lmm_fid: [0x200000404:0x65:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x105:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-29T23:01:18-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x65 lmm_fid: [0x200000404:0x65:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x106:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 23:01:27 (1790737287) Failing mds1 on oleg436-server Stopping /mnt/lustre-mds1 (opts:) on oleg436-server 23:01:35 (1790737295) shut down facet: mds1 facet_host: oleg436-server facet_failover_host: oleg436-server Failover mds1 to oleg436-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg436-server: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-MDT0000 23:02:09 (1790737329) targets are mounted 23:02:09 (1790737329) facet_failover done oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387582 s, 27.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x69 lmm_fid: [0x200000404:0x69:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x107:0x0] } PASS 50A (59s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 23:02:26 (1790737346) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 23:02:30 (1790737350) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 23:02:34 (1790737354) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00973159 s, 1.7 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 23:02:43 (1790737363) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 23:02:46 (1790737366) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00943556 s, 434 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.107898 s, 38.0 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 23:02:56 (1790737376) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) PASS 60b (26s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 23:03:21 (1790737401) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 23:03:49 (1790737429) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa7:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 23:04:14 (1790737454) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 23:04:38 (1790737478) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 23:04:46 (1790737486) mirror create pid 54997 mirror split pid 54998 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5095: 54997 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5095: 54998 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 23:05:53 (1790737553) Seed set to 7555 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x1ca3c 000002 1790737555.867371 trunc from 00000000 to 0x01ca3b (0x1ca3c bytes) 000003 1790737555.884075 write 0x01c0d8 thru 0x0264fa (0xa423 bytes) 000004 1790737555.892454 punch from 0x0127b0 to 0x01d77e (0xafcf bytes) 000005 1790737555.916241 punch from 0x0036e0 to 0x00bbdc (0x84fd bytes) 000006 1790737555.921371 mapread 0x017a89 thru 0x01a26a (0x27e2 bytes) 000007 1790737555.956022 MIRROR_EXTEND to 2 mirrors 000008 1790737556.820752 read 0x00a609 thru 0x013e27 (0x981f bytes) 000009 1790737556.830057 punch from 0x01ec3a to 0x0264fa (0x78c1 bytes) 000010 1790737556.899920 read 0x00b5a0 thru 0x010516 (0x4f77 bytes) 000011 1790737556.914868 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x264fb): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x264fb): Mirror 2: 0xdc9617fa (stale) CRC-32 checksum value for chunk [0, 0x264fb): Mirror 1: 0x622e4b9b Mirror 3: 0x622e4b9b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x264fb: skipped 000012 1790737558.286159 mapread 0x00a09f thru 0x01059e (0x6500 bytes) 000013 1790737558.317966 trunc from 0x0264fb to 0x012868 (0x13c92 bytes) 000014 1790737558.429799 mapwrite 0x035731 thru 0x035acf (0x39f bytes) 000015 1790737558.563173 trunc from 0x035ad0 to 0x00d3da (0x286f5 bytes) 000016 1790737558.628342 read 0x00c824 thru 0x00d3da (0xbb7 bytes) truncating to largest ever: 0x2132c 000019 1790737558.647313 trunc from 0x00d3db to 0x02132b (0x13f51 bytes) 000020 1790737558.672401 trunc from 0x02132c to 0x006898 (0x1aa93 bytes) 000021 1790737558.699938 mapread 0x000532 thru 0x001105 (0xbd4 bytes) 000022 1790737558.715862 mapread 0x006376 thru 0x006898 (0x523 bytes) 000023 1790737558.716937 mapwrite 0x02b7ce thru 0x031cec (0x651f bytes) 000024 1790737558.836295 punch from 0x02863a to 0x031cec (0x96b3 bytes) 000025 1790737558.844419 read 0x012ad0 thru 0x0150b8 (0x25e9 bytes) 000027 1790737558.865416 mapread 0x01e796 thru 0x026bce (0x8439 bytes) 000028 1790737558.868457 read 0x00db7c thru 0x0192ba (0xb73f bytes) 000029 1790737558.869852 write 0x00b1a3 thru 0x00cdb5 (0x1c13 bytes) truncating to largest ever: 0x262cf 000030 1790737558.871782 trunc from 0x031ced to 0x0262ce (0xba1e bytes) 000031 1790737558.909101 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x262cf): Mirror 3: 0xcdcc2108 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x262cf): Mirror 3: 0xcdcc2108 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x262cf: skipped 000032 1790737559.393319 write 0x01029f thru 0x01d911 (0xd673 bytes) 000033 1790737559.408329 mapwrite 0x01213f thru 0x014e57 (0x2d19 bytes) 000034 1790737559.469027 read 0x019b24 thru 0x01ab30 (0x100d bytes) 000035 1790737559.475800 MIRROR_RESYNC 2 mirrors 000036 1790737560.286565 write 0x001839 thru 0x00e529 (0xccf1 bytes) 000039 1790737560.392006 mapwrite 0x03bb31 thru 0x03be4b (0x31b bytes) 000040 1790737560.554145 read 0x01e231 thru 0x0283b6 (0xa186 bytes) 000041 1790737560.571057 trunc from 0x03be4c to 0x00663d (0x3580e bytes) 000042 1790737560.621093 write 0x00ebf9 thru 0x018ce8 (0xa0f0 bytes) 000043 1790737560.643740 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) 000044 1790737560.850394 read 0x008737 thru 0x00f91d (0x71e7 bytes) 000048 1790737560.886516 trunc from 0x018ce9 to 0x013381 (0x5967 bytes) 000050 1790737560.920916 mapread 0x004996 thru 0x00dd6c (0x93d7 bytes) 000051 1790737560.964412 mapread 0x00fb66 thru 0x011b83 (0x201e bytes) 000052 1790737560.970901 punch from 0x011f9a to 0x013381 (0x13e8 bytes) 000053 1790737560.992703 MIRROR_EXTEND to 2 mirrors 000054 1790737561.978064 punch from 0x011002 to 0x013381 (0x2380 bytes) 000055 1790737562.106011 mapread 0x005c6d thru 0x00b2c0 (0x5654 bytes) 000056 1790737562.141021 punch from 0x00f33c to 0x010deb (0x1ab0 bytes) 000058 1790737562.157123 MIRROR_RESYNC 2 mirrors 000060 1790737563.093692 read 0x012c9f thru 0x013381 (0x6e3 bytes) 000062 1790737563.114504 mapread 0x012f2c thru 0x013381 (0x456 bytes) 000063 1790737563.123867 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) 000064 1790737563.318203 punch from 0x002756 to 0x009dfe (0x76a9 bytes) 000068 1790737563.338975 mapread 0x009584 thru 0x013381 (0x9dfe bytes) 000070 1790737563.363916 write 0x00433e thru 0x00bb14 (0x77d7 bytes) truncating to largest ever: 0x34402 000071 1790737563.366012 trunc from 0x013382 to 0x034401 (0x21080 bytes) 000072 1790737563.383553 mapread 0x02e600 thru 0x02e656 (0x57 bytes) 000073 1790737563.395146 mapread 0x0228fd thru 0x03168d (0xed91 bytes) 000074 1790737563.406535 read 0x002f41 thru 0x006ea5 (0x3f65 bytes) 000075 1790737563.416130 mapwrite 0x0199d6 thru 0x027421 (0xda4c bytes) 000076 1790737563.569701 punch from 0x0244c5 to 0x02b6c8 (0x7204 bytes) 000077 1790737563.579143 trunc from 0x034402 to 0x00388f (0x30b72 bytes) 000078 1790737563.634936 mapwrite 0x00c793 thru 0x016764 (0x9fd2 bytes) 000079 1790737563.779023 punch from 0x003bfe to 0x0110ba (0xd4bd bytes) 000081 1790737563.787574 punch from 0x014922 to 0x016764 (0x1e43 bytes) 000083 1790737563.794644 write 0x006904 thru 0x00e57a (0x7c77 bytes) 000084 1790737563.806172 trunc from 0x016765 to 0x009760 (0xd004 bytes) 000085 1790737563.830583 punch from 0x0046c3 to 0x009105 (0x4a43 bytes) 000086 1790737563.852606 punch from 0x00917a to 0x009760 (0x5e7 bytes) 000089 1790737563.860614 mapwrite 0x034ad2 thru 0x03f816 (0xad45 bytes) 000090 1790737563.953845 write 0x029485 thru 0x02e08e (0x4c0a bytes) 000091 1790737563.962094 mapread 0x0014f7 thru 0x00650a (0x5014 bytes) 000093 1790737563.976218 MIRROR_EXTEND to 2 mirrors 000094 1790737565.003875 MIRROR_RESYNC 2 mirrors 000095 1790737565.533352 write 0x01c2fd thru 0x028103 (0xbe07 bytes) 000098 1790737565.663805 mapwrite 0x008802 thru 0x017c3e (0xf43d bytes) 000100 1790737565.821455 mapwrite 0x035e4d thru 0x03a7c3 (0x4977 bytes) 000101 1790737565.907819 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f817): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f817): Mirror 2: 0x465efc5c (stale) CRC-32 checksum value for chunk [0, 0x3f817): Mirror 1: 0x878bf587 Mirror 3: 0x878bf587 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f817: skipped 000103 1790737567.355390 mapwrite 0x0207cd thru 0x024743 (0x3f77 bytes) 000104 1790737567.739687 read 0x01c38e thru 0x01f746 (0x33b9 bytes) 000105 1790737567.740639 mapwrite 0x02e6a9 thru 0x03dfca (0xf922 bytes) 000107 1790737567.913819 mapwrite 0x005209 thru 0x0133e2 (0xe1da bytes) 000108 1790737568.101233 mapwrite 0x027760 thru 0x02cb6c (0x540d bytes) 000109 1790737568.146146 trunc from 0x03f817 to 0x0234d0 (0x1c346 bytes) 000110 1790737568.170501 trunc from 0x0234d1 to 0x01343a (0x10096 bytes) 000111 1790737568.192453 punch from 0x004891 to 0x01240f (0xdb7f bytes) 000112 1790737568.198923 trunc from 0x01343b to 0x022229 (0xedef bytes) 000116 1790737568.236158 punch from 0x018d58 to 0x01da5e (0x4d07 bytes) 000117 1790737568.241771 trunc from 0x02222a to 0x033bd1 (0x119a8 bytes) 000120 1790737568.262165 write 0x0317a6 thru 0x036579 (0x4dd4 bytes) 000123 1790737568.274997 punch from 0x028562 to 0x02c4c1 (0x3f60 bytes) 000125 1790737568.288954 read 0x00c8d7 thru 0x011293 (0x49bd bytes) 000127 1790737568.306491 punch from 0x023517 to 0x024c31 (0x171b bytes) 000128 1790737568.311705 trunc from 0x03657a to 0x030b00 (0x5a79 bytes) 000133 1790737568.335895 read 0x02d219 thru 0x030b00 (0x38e8 bytes) 000134 1790737568.336930 mapwrite 0x00c10f thru 0x01337d (0x726f bytes) 000135 1790737568.399761 punch from 0x01b135 to 0x0243a3 (0x926f bytes) 000136 1790737568.405517 trunc from 0x030b01 to 0x004d70 (0x2bd90 bytes) 000137 1790737568.427827 mapwrite 0x016821 thru 0x023486 (0xcc66 bytes) 000138 1790737568.538622 trunc from 0x023487 to 0x0071aa (0x1c2dc bytes) truncating to largest ever: 0x3fcbd 000139 1790737568.561008 trunc from 0x0071ab to 0x03fcbc (0x38b12 bytes) 000140 1790737568.589274 mapwrite 0x037e40 thru 0x03abca (0x2d8b bytes) 000141 1790737568.714397 mapwrite 0x000cbe thru 0x004212 (0x3555 bytes) 000142 1790737568.820420 mapread 0x03f9d8 thru 0x03fcbc (0x2e5 bytes) 000143 1790737568.821985 write 0x010217 thru 0x0141ed (0x3fd7 bytes) 000144 1790737568.841870 MIRROR_RESYNC 3 mirrors 000145 1790737570.157811 write 0x030e1e thru 0x0371b9 (0x639c bytes) 000148 1790737570.254740 mapwrite 0x02dc8a thru 0x034a8d (0x6e04 bytes) 000149 1790737570.334235 mapwrite 0x00db23 thru 0x01cad9 (0xefb7 bytes) 000150 1790737570.427452 trunc from 0x03fcbd to 0x02c693 (0x13629 bytes) 000151 1790737570.459905 punch from 0x02c148 to 0x02c693 (0x54c bytes) 000152 1790737570.485440 mapread 0x00ff02 thru 0x01c0b6 (0xc1b5 bytes) 000155 1790737570.489218 read 0x029fc5 thru 0x02b65c (0x1698 bytes) 000156 1790737570.490693 write 0x03313a thru 0x03be0f (0x8cd6 bytes) 000157 1790737570.492063 mapread 0x01f1de thru 0x02384d (0x4670 bytes) 000160 1790737570.511614 punch from 0x029f02 to 0x039439 (0xf538 bytes) 000161 1790737570.529418 mapwrite 0x016184 thru 0x019e8c (0x3d09 bytes) 000162 1790737570.586417 punch from 0x00a36e to 0x019e67 (0xfafa bytes) 000163 1790737570.592463 mapread 0x03ba61 thru 0x03be0f (0x3af bytes) 000164 1790737570.593708 mapread 0x036181 thru 0x03be0f (0x5c8f bytes) 000165 1790737570.610744 write 0x01b1d0 thru 0x021d8f (0x6bc0 bytes) 000167 1790737570.612213 trunc from 0x03be10 to 0x00a05d (0x31db2 bytes) 000168 1790737570.629667 read 0x003021 thru 0x00a05d (0x703d bytes) 000170 1790737570.645160 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xa05e): Mirror 2: 0x5663ef0d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa05e): Mirror 2: 0x5663ef0d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa05e: skipped 000171 1790737570.987389 mapread 0x0078a8 thru 0x007b8c (0x2e5 bytes) 000173 1790737570.995937 mapwrite 0x01a3ba thru 0x02639d (0xbfe4 bytes) 000175 1790737571.110361 read 0x002017 thru 0x00b115 (0x90ff bytes) 000176 1790737571.142071 read 0x0027ca thru 0x0065ca (0x3e01 bytes) 000177 1790737571.142930 punch from 0x01e63c to 0x0221a4 (0x3b69 bytes) 000178 1790737571.154986 mapwrite 0x02dcd1 thru 0x0309bc (0x2cec bytes) 000179 1790737571.304595 write 0x0300fb thru 0x03c188 (0xc08e bytes) 000180 1790737571.309876 write 0x0063dd thru 0x0105a0 (0xa1c4 bytes) 000182 1790737571.317696 trunc from 0x03c189 to 0x00edcd (0x2d3bb bytes) 000183 1790737571.349079 punch from 0x00adff to 0x00edcd (0x3fcf bytes) 000184 1790737571.387963 read 0x009ceb thru 0x00edcd (0x50e3 bytes) 000185 1790737571.419940 read 0x008016 thru 0x00edcd (0x6db8 bytes) 000186 1790737571.422780 read 0x00ae6e thru 0x00edcd (0x3f60 bytes) 000187 1790737571.430886 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) 000188 1790737571.612127 read 0x004451 thru 0x005cc0 (0x1870 bytes) 000192 1790737571.625181 read 0x004fa6 thru 0x0058e6 (0x941 bytes) 000194 1790737571.627072 mapread 0x00ab7c thru 0x00c2a1 (0x1726 bytes) 000195 1790737571.638772 mapwrite 0x038cf8 thru 0x03a95f (0x1c68 bytes) 000196 1790737571.718566 read 0x02d4f5 thru 0x03a95f (0xd46b bytes) 000197 1790737571.719706 read 0x0370a6 thru 0x03a95f (0x38ba bytes) 000199 1790737571.721716 mapread 0x0248b4 thru 0x026a2b (0x2178 bytes) 000200 1790737571.723481 read 0x002cea thru 0x00c2c9 (0x95e0 bytes) 000201 1790737571.724750 trunc from 0x03a960 to 0x03b58b (0xc2c bytes) 000202 1790737571.743500 mapread 0x01c1b0 thru 0x020520 (0x4371 bytes) 000205 1790737571.746439 punch from 0x0312fe to 0x0351da (0x3edd bytes) 000207 1790737571.752048 trunc from 0x03b58c to 0x01bd2c (0x1f85f bytes) 000208 1790737571.769096 punch from 0x000d8b to 0x007da7 (0x701d bytes) 000209 1790737571.784402 trunc from 0x01bd2d to 0x01f3d9 (0x36ad bytes) 000210 1790737571.829637 MIRROR_EXTEND to 2 mirrors 000211 1790737573.274865 mapwrite 0x0349fd thru 0x037ef0 (0x34f4 bytes) 000212 1790737573.685024 write 0x0377b7 thru 0x03806f (0x8b9 bytes) 000214 1790737573.693663 punch from 0x025334 to 0x033f06 (0xebd3 bytes) 000215 1790737573.717154 trunc from 0x038070 to 0x012246 (0x25e29 bytes) 000216 1790737573.768852 punch from 0x00f93f to 0x012246 (0x2908 bytes) 000217 1790737573.776370 mapwrite 0x02e4b7 thru 0x03cf9d (0xeae7 bytes) 000218 1790737573.925881 write 0x01d838 thru 0x01e4c0 (0xc89 bytes) 000219 1790737573.944554 mapwrite 0x026469 thru 0x02ce13 (0x69ab bytes) 000220 1790737574.021939 write 0x03b6d7 thru 0x03ffff (0x4929 bytes) 000223 1790737574.026931 read 0x03d705 thru 0x03eba8 (0x14a4 bytes) 000224 1790737574.028093 trunc from 0x040000 to 0x0113b0 (0x2ec4f bytes) 000225 1790737574.062072 mapwrite 0x00254c thru 0x008805 (0x62ba bytes) 000227 1790737574.132710 read 0x003c56 thru 0x008e44 (0x51ef bytes) 000228 1790737574.138694 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) 000229 1790737574.305042 punch from 0x001b3d to 0x00db59 (0xc01d bytes) 000232 1790737574.312926 punch from 0x00803f to 0x008ce6 (0xca8 bytes) 000233 1790737574.318456 punch from 0x000917 to 0x008b0f (0x81f9 bytes) 000234 1790737574.328955 trunc from 0x0113b1 to 0x038ee6 (0x27b36 bytes) 000235 1790737574.357941 trunc from 0x038ee7 to 0x03baa5 (0x2bbf bytes) 000236 1790737574.384890 mapwrite 0x035543 thru 0x038dc8 (0x3886 bytes) 000238 1790737574.486760 MIRROR_EXTEND to 2 mirrors 000239 1790737575.566823 MIRROR_RESYNC 2 mirrors 000240 1790737576.200528 mapread 0x00bfb8 thru 0x00cc48 (0xc91 bytes) 000242 1790737576.224916 mapwrite 0x0119e6 thru 0x01dde6 (0xc401 bytes) 000243 1790737576.459769 mapread 0x00dcbc thru 0x015941 (0x7c86 bytes) 000244 1790737576.461866 trunc from 0x03baa6 to 0x033945 (0x8160 bytes) 000245 1790737576.493608 mapread 0x0223bc thru 0x030866 (0xe4ab bytes) 000246 1790737576.579222 mapwrite 0x03c476 thru 0x03ffff (0x3b8a bytes) 000247 1790737576.712433 MIRROR_RESYNC 2 mirrors 000251 1790737578.108308 read 0x0143f1 thru 0x020c6f (0xc87f bytes) 000252 1790737578.119021 punch from 0x01757b to 0x01d4e3 (0x5f69 bytes) 000253 1790737578.276673 mapread 0x022b25 thru 0x026fd7 (0x44b3 bytes) 000254 1790737578.315167 read 0x007889 thru 0x0115df (0x9d57 bytes) 000255 1790737578.348892 punch from 0x020e61 to 0x02241b (0x15bb bytes) 000256 1790737578.357898 punch from 0x034d3f to 0x03f1bb (0xa47d bytes) 000257 1790737578.367601 write 0x03fdd3 thru 0x03ffff (0x22d bytes) 000258 1790737578.386212 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8a05bdc7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcfe2d304 Mirror 3: 0xcfe2d304 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000259 1790737580.002767 trunc from 0x040000 to 0x00750a (0x38af5 bytes) 000260 1790737580.101431 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x750b): Mirror 3: 0x2041eef (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x750b): Mirror 3: 0x2041eef (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x750b: skipped 000263 1790737580.575962 write 0x0161b7 thru 0x01c96b (0x67b5 bytes) 000264 1790737580.582011 punch from 0x00d66a to 0x014e72 (0x7809 bytes) 000265 1790737580.599030 mapread 0x001411 thru 0x005765 (0x4355 bytes) 000266 1790737580.623347 mapwrite 0x0120a4 thru 0x015db1 (0x3d0e bytes) 000267 1790737580.698016 write 0x02e31f thru 0x038cd1 (0xa9b3 bytes) 000268 1790737580.700599 mapwrite 0x01164d thru 0x01ebe5 (0xd599 bytes) 000269 1790737580.789487 mapwrite 0x010642 thru 0x01cbc1 (0xc580 bytes) 000271 1790737580.873791 mapwrite 0x030bf9 thru 0x03275c (0x1b64 bytes) 000272 1790737580.930101 write 0x00889b thru 0x013973 (0xb0d9 bytes) 000273 1790737580.931534 mapread 0x0360c4 thru 0x038cd1 (0x2c0e bytes) 000274 1790737580.933703 trunc from 0x038cd2 to 0x031ee6 (0x6deb bytes) 000275 1790737580.953975 mapread 0x007934 thru 0x00f1ac (0x7879 bytes) 000277 1790737580.957792 mapwrite 0x00c819 thru 0x016f42 (0xa72a bytes) 000280 1790737581.007339 write 0x029790 thru 0x02d484 (0x3cf5 bytes) 000281 1790737581.024186 read 0x009383 thru 0x0131e8 (0x9e66 bytes) 000283 1790737581.025710 write 0x00ee34 thru 0x0102b7 (0x1484 bytes) 000284 1790737581.032908 MIRROR_RESYNC 2 mirrors 000285 1790737581.843825 write 0x029fee thru 0x030437 (0x644a bytes) 000287 1790737581.975442 mapwrite 0x00bf61 thru 0x011449 (0x54e9 bytes) 000290 1790737582.101388 write 0x0363f9 thru 0x03ffff (0x9c07 bytes) 000292 1790737582.105719 write 0x03caa0 thru 0x03dcfe (0x125f bytes) 000293 1790737582.120202 mapwrite 0x000b77 thru 0x006f40 (0x63ca bytes) 000295 1790737582.262920 mapwrite 0x00ffc8 thru 0x0171c6 (0x71ff bytes) 000296 1790737582.327957 read 0x03272f thru 0x034383 (0x1c55 bytes) 000297 1790737582.335434 trunc from 0x040000 to 0x02866b (0x17994 bytes) 000298 1790737582.389440 mapread 0x01fb26 thru 0x02866b (0x8b46 bytes) 000299 1790737582.407872 mapwrite 0x0235ae thru 0x026660 (0x30b3 bytes) 000300 1790737582.469806 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2866c): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2866c): Mirror 3: 0x8ec70065 (stale) CRC-32 checksum value for chunk [0, 0x2866c): Mirror 1: 0x58ca3c8e Mirror 4: 0x58ca3c8e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2866c: skipped 000301 1790737583.591618 mapwrite 0x0016e3 thru 0x007ea4 (0x67c2 bytes) 000306 1790737583.844667 read 0x007f9b thru 0x011070 (0x90d6 bytes) 000309 1790737583.855297 mapread 0x019ea2 thru 0x024928 (0xaa87 bytes) 000310 1790737583.928439 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2866c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2866c): Mirror 3: 0x8ec70065 (stale) Mirror 4: 0x58ca3c8e (stale) CRC-32 checksum value for chunk [0, 0x2866c): Mirror 1: 0x1d91ec55 Mirror 5: 0x1d91ec55 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2866c: skipped 000311 1790737584.991127 write 0x032306 thru 0x03ffff (0xdcfa bytes) 000313 1790737585.094549 punch from 0x026672 to 0x027c6a (0x15f9 bytes) 000315 1790737585.120866 trunc from 0x040000 to 0x022af6 (0x1d509 bytes) 000316 1790737585.161671 mapwrite 0x00bbd7 thru 0x01b061 (0xf48b bytes) 000319 1790737585.279877 trunc from 0x022af7 to 0x03673b (0x13c45 bytes) 000320 1790737585.303578 read 0x031fe8 thru 0x03673b (0x4754 bytes) 000322 1790737585.316655 punch from 0x01c245 to 0x02a8a5 (0xe661 bytes) 000324 1790737585.324072 mapread 0x03235d thru 0x03673b (0x43df bytes) 000325 1790737585.326061 mapread 0x02f60b thru 0x03673b (0x7131 bytes) 000326 1790737585.343865 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3673c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3673c): Mirror 3: 0x572f71f1 (stale) Mirror 4: 0xd6667b7f (stale) Mirror 5: 0xf5203963 (stale) CRC-32 checksum value for chunk [0, 0x3673c): Mirror 1: 0x35ca8ecb Mirror 6: 0x35ca8ecb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3673c: skipped 000328 1790737587.319073 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3673c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3673c): Mirror 3: 0x572f71f1 (stale) Mirror 4: 0xd6667b7f (stale) Mirror 5: 0xf5203963 (stale) CRC-32 checksum value for chunk [0, 0x3673c): Mirror 1: 0x35ca8ecb Mirror 6: 0x35ca8ecb Mirror 7: 0x35ca8ecb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3673c: skipped 000332 1790737588.542852 punch from 0x00b2c2 to 0x01aed8 (0xfc17 bytes) 000333 1790737588.682055 mapread 0x00ce0c thru 0x01466a (0x785f bytes) 000336 1790737588.740926 read 0x02bd30 thru 0x033eff (0x81d0 bytes) 000337 1790737588.766905 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3673c): Mirror 3: 0x572f71f1 (stale) Mirror 4: 0xd6667b7f (stale) Mirror 5: 0xf5203963 (stale) Mirror 7: 0x35ca8ecb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3673c): Mirror 3: 0x572f71f1 (stale) Mirror 4: 0xd6667b7f (stale) Mirror 5: 0xf5203963 (stale) Mirror 7: 0x35ca8ecb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3673c: skipped 000340 1790737589.316962 trunc from 0x03673c to 0x02e9ea (0x7d51 bytes) 000341 1790737589.356972 write 0x00e203 thru 0x0184e5 (0xa2e3 bytes) 000342 1790737589.377575 mapread 0x0123fa thru 0x01c248 (0x9e4f bytes) 000343 1790737589.409960 write 0x026939 thru 0x035d0b (0xf3d3 bytes) 000344 1790737589.418613 punch from 0x02cb92 to 0x031a23 (0x4e92 bytes) 000345 1790737589.453810 write 0x03ee08 thru 0x03ffff (0x11f8 bytes) 000346 1790737589.459786 punch from 0x034205 to 0x03ef2a (0xad26 bytes) 000347 1790737589.483796 read 0x010698 thru 0x01d05d (0xc9c6 bytes) 000348 1790737589.492410 trunc from 0x040000 to 0x035b02 (0xa4fd bytes) 000349 1790737589.539515 write 0x02d783 thru 0x034a1c (0x729a bytes) 000350 1790737589.550353 punch from 0x00a752 to 0x00fe20 (0x56cf bytes) 000353 1790737589.579589 punch from 0x00115f to 0x009b47 (0x89e9 bytes) 000354 1790737589.591487 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x35b03): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35b03): Mirror 3: 0xfd8c5d7c (stale) Mirror 4: 0x718b7e24 (stale) Mirror 5: 0xb62440f2 (stale) Mirror 7: 0xe14236a9 (stale) CRC-32 checksum value for chunk [0, 0x35b03): Mirror 1: 0x2a081636 Mirror 8: 0x2a081636 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35b03: skipped 000356 1790737591.165132 mapwrite 0x03bb0c thru 0x03c7cd (0xcc2 bytes) 000357 1790737591.393796 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c7ce): Mirror 3: 0xb7763ffa (stale) Mirror 4: 0xd6095e2e (stale) Mirror 7: 0xade51575 (stale) Mirror 8: 0x1f93864 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c7ce): Mirror 3: 0xb7763ffa (stale) Mirror 4: 0xd6095e2e (stale) Mirror 7: 0xade51575 (stale) Mirror 8: 0x1f93864 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c7ce: skipped 000358 1790737591.816146 read 0x03b8f2 thru 0x03c7cd (0xedc bytes) 000359 1790737591.822807 read 0x03427e thru 0x03c7cd (0x8550 bytes) 000360 1790737591.833811 write 0x02eada thru 0x03cb60 (0xe087 bytes) 000361 1790737591.835550 mapread 0x016ee1 thru 0x024a8a (0xdbaa bytes) 000362 1790737591.841004 MIRROR_RESYNC 5 mirrors 000363 1790737593.664289 mapread 0x00b9bf thru 0x013318 (0x795a bytes) 000364 1790737593.700475 punch from 0x01c804 to 0x022a1a (0x6217 bytes) 000365 1790737593.783782 punch from 0x0184e7 to 0x026278 (0xdd92 bytes) 000366 1790737593.789173 mapread 0x02dac9 thru 0x033861 (0x5d99 bytes) 000367 1790737593.804591 mapwrite 0x00607b thru 0x015c5a (0xfbe0 bytes) 000369 1790737593.918236 trunc from 0x03cb61 to 0x006ef7 (0x35c69 bytes) 000371 1790737593.949746 mapwrite 0x031799 thru 0x03d0aa (0xb912 bytes) 000372 1790737594.056137 trunc from 0x03d0ab to 0x017bff (0x254ab bytes) 000374 1790737594.079439 punch from 0x009b42 to 0x00bc07 (0x20c6 bytes) 000375 1790737594.086046 punch from 0x000209 to 0x0056e2 (0x54da bytes) 000376 1790737594.093023 read 0x00f92f thru 0x017bff (0x82d1 bytes) 000377 1790737594.109797 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17c00): Mirror 3: 0x7ccf77b9 (stale) Mirror 4: 0x7ccf77b9 (stale) Mirror 8: 0x7ccf77b9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17c00): Mirror 3: 0x7ccf77b9 (stale) Mirror 4: 0x7ccf77b9 (stale) Mirror 8: 0x7ccf77b9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17c00: skipped 000378 1790737594.589293 mapread 0x00275a thru 0x006d9d (0x4644 bytes) 000381 1790737594.621552 mapread 0x00fd86 thru 0x017bff (0x7e7a bytes) 000382 1790737594.640859 read 0x00697d thru 0x00a6de (0x3d62 bytes) 000383 1790737594.641843 write 0x0215be thru 0x021da1 (0x7e4 bytes) 000384 1790737594.648403 write 0x03b59c thru 0x03ffff (0x4a64 bytes) 000385 1790737594.653193 punch from 0x0114e4 to 0x01ec95 (0xd7b2 bytes) 000388 1790737594.685469 punch from 0x035988 to 0x03d28b (0x7904 bytes) 000390 1790737594.695379 mapwrite 0x03ecf6 thru 0x03ffff (0x130a bytes) 000391 1790737594.758459 mapwrite 0x000ecd thru 0x008793 (0x78c7 bytes) 000393 1790737594.869844 write 0x026cb1 thru 0x027cb3 (0x1003 bytes) 000394 1790737594.888223 read 0x017b68 thru 0x01fb1b (0x7fb4 bytes) 000395 1790737594.905792 mapread 0x019659 thru 0x028cc1 (0xf669 bytes) 000396 1790737594.907034 read 0x01b677 thru 0x024250 (0x8bda bytes) 000397 1790737594.910449 punch from 0x022b2b to 0x028f62 (0x6438 bytes) 000398 1790737594.923087 punch from 0x012d4e to 0x01b350 (0x8603 bytes) 000399 1790737594.928726 punch from 0x021dc6 to 0x026966 (0x4ba1 bytes) 000400 1790737594.936415 read 0x0055e5 thru 0x00f4e0 (0x9efc bytes) 000402 1790737594.937717 mapwrite 0x00cd7d thru 0x016204 (0x9488 bytes) 000403 1790737594.993214 read 0x039880 thru 0x03ffff (0x6780 bytes) 000404 1790737594.998665 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x5b70f748 (stale) Mirror 8: 0x5b70f748 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x5b70f748 (stale) Mirror 8: 0x5b70f748 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000406 1790737595.337546 mapwrite 0x0000ea thru 0x005599 (0x54b0 bytes) 000407 1790737595.434242 read 0x026f40 thru 0x0337d3 (0xc894 bytes) 000408 1790737595.450420 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x5b70f748 (stale) Mirror 8: 0x5b70f748 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x96b03af6 Mirror 9: 0x96b03af6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000409 1790737596.517166 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x5b70f748 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x5b70f748 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x96b03af6 Mirror 9: 0x96b03af6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000410 1790737597.256695 write 0x021589 thru 0x02f1c1 (0xdc39 bytes) 000412 1790737597.395001 write 0x039275 thru 0x03ffff (0x6d8b bytes) 000413 1790737597.404095 trunc from 0x040000 to 0x03e7bb (0x1844 bytes) 000414 1790737597.483497 mapread 0x032485 thru 0x038b1e (0x669a bytes) 000416 1790737597.516180 trunc from 0x03e7bc to 0x028b06 (0x15cb5 bytes) 000418 1790737597.539199 mapwrite 0x03c586 thru 0x03ffff (0x3a7a bytes) 000420 1790737597.650176 punch from 0x03f4f5 to 0x03ffff (0xb0b bytes) 000421 1790737597.660900 MIRROR_RESYNC 3 mirrors 000423 1790737599.062740 punch from 0x039b08 to 0x03ffff (0x64f8 bytes) 000424 1790737599.128242 read 0x00e138 thru 0x01624e (0x8117 bytes) 000426 1790737599.143281 mapread 0x00b8a2 thru 0x0127bb (0x6f1a bytes) 000428 1790737599.165492 mapread 0x03b19f thru 0x03f35b (0x41bd bytes) 000429 1790737599.176382 read 0x014c89 thru 0x01d010 (0x8388 bytes) 000431 1790737599.188393 trunc from 0x040000 to 0x03da68 (0x2597 bytes) 000432 1790737599.215412 trunc from 0x03da69 to 0x006e34 (0x36c34 bytes) 000433 1790737599.235623 read 0x002804 thru 0x006e34 (0x4631 bytes) 000435 1790737599.236853 mapread 0x0022bb thru 0x006e34 (0x4b7a bytes) 000436 1790737599.239227 punch from 0x0026be to 0x006977 (0x42ba bytes) 000437 1790737599.246566 mapwrite 0x03eff3 thru 0x03ffff (0x100d bytes) 000438 1790737599.341201 MIRROR_RESYNC 3 mirrors 000442 1790737601.001007 mapwrite 0x00ab7c thru 0x018091 (0xd516 bytes) 000443 1790737601.400081 punch from 0x01671a to 0x01cae2 (0x63c9 bytes) 000444 1790737601.412554 read 0x015bc8 thru 0x025aa5 (0xfede bytes) 000446 1790737601.426484 trunc from 0x040000 to 0x013833 (0x2c7cc bytes) 000447 1790737601.450327 write 0x016343 thru 0x0262a5 (0xff63 bytes) 000448 1790737601.453787 write 0x03d532 thru 0x03ffff (0x2ace bytes) 000449 1790737601.455785 write 0x03a7c3 thru 0x03ffff (0x583d bytes) 000450 1790737601.468363 MIRROR_RESYNC 3 mirrors 000453 1790737602.529799 mapwrite 0x03a7a9 thru 0x03ffff (0x5857 bytes) 000456 1790737602.737228 mapwrite 0x00c4fa thru 0x01bd49 (0xf850 bytes) 000460 1790737602.844871 mapread 0x01716f thru 0x025d34 (0xebc6 bytes) 000461 1790737602.868027 MIRROR_RESYNC 3 mirrors 000462 1790737603.953173 write 0x02f7f3 thru 0x035a70 (0x627e bytes) 000463 1790737604.079518 punch from 0x000cef to 0x00400a (0x331c bytes) 000464 1790737604.097070 trunc from 0x040000 to 0x027105 (0x18efa bytes) 000465 1790737604.119107 trunc from 0x027106 to 0x01c514 (0xabf1 bytes) 000466 1790737604.145225 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c515): Mirror 9: 0x1ced5b10 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c515): Mirror 9: 0x1ced5b10 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c515: skipped 000469 1790737604.466968 write 0x00dc42 thru 0x00f04f (0x140e bytes) 000471 1790737604.480471 mapread 0x019805 thru 0x01c514 (0x2d10 bytes) 000473 1790737604.509720 trunc from 0x01c515 to 0x00b680 (0x10e94 bytes) 000474 1790737604.559815 mapwrite 0x019b03 thru 0x020351 (0x684f bytes) 000475 1790737604.700832 mapwrite 0x0143e5 thru 0x016f8f (0x2bab bytes) 000476 1790737604.778380 trunc from 0x020352 to 0x03108b (0x10d3a bytes) 000478 1790737604.835113 mapwrite 0x03e12a thru 0x03ffff (0x1ed6 bytes) 000479 1790737604.952436 mapread 0x020a65 thru 0x028cd1 (0x826d bytes) 000480 1790737604.991448 read 0x0066cd thru 0x00d47d (0x6db1 bytes) 000483 1790737605.024450 read 0x033e7c thru 0x03d530 (0x96b5 bytes) 000484 1790737605.046591 write 0x027591 thru 0x027ea7 (0x917 bytes) 000485 1790737605.058006 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) 000486 1790737605.283136 mapread 0x00acae thru 0x017e7c (0xd1cf bytes) 000488 1790737605.350927 mapwrite 0x0004a5 thru 0x00ad34 (0xa890 bytes) 000489 1790737605.420313 punch from 0x03d1d2 to 0x03d2c7 (0xf6 bytes) 000490 1790737605.431089 write 0x008477 thru 0x013f2a (0xbab4 bytes) 000493 1790737605.448291 punch from 0x006a53 to 0x013597 (0xcb45 bytes) 000494 1790737605.473873 read 0x002242 thru 0x00c126 (0x9ee5 bytes) 000496 1790737605.486742 mapwrite 0x00700f thru 0x00eced (0x7cdf bytes) 000498 1790737605.559121 mapwrite 0x005f5e thru 0x009431 (0x34d4 bytes) 000499 1790737605.635295 MIRROR_EXTEND to 2 mirrors 000503 1790737606.596524 punch from 0x0031de to 0x00b0f5 (0x7f18 bytes) 000504 1790737606.687792 punch from 0x006f2b to 0x010a20 (0x9af6 bytes) 000505 1790737606.699476 MIRROR_RESYNC 2 mirrors 000506 1790737607.528686 mapwrite 0x02a608 thru 0x02c2d2 (0x1ccb bytes) 000507 1790737607.767866 punch from 0x0128fe to 0x013d95 (0x1498 bytes) 000508 1790737607.781895 trunc from 0x040000 to 0x016bed (0x29412 bytes) 000510 1790737607.815651 punch from 0x0155a1 to 0x016bed (0x164d bytes) 000511 1790737607.823681 trunc from 0x016bee to 0x01e77d (0x7b90 bytes) 000513 1790737607.846557 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) 000514 1790737608.101914 punch from 0x019a64 to 0x01d250 (0x37ed bytes) 000515 1790737608.109372 punch from 0x01b62d to 0x01e77d (0x3151 bytes) 000517 1790737608.117882 write 0x034a4b thru 0x03d797 (0x8d4d bytes) 000519 1790737608.125228 read 0x03bd93 thru 0x03d797 (0x1a05 bytes) 000520 1790737608.126438 punch from 0x0214f2 to 0x030135 (0xec44 bytes) 000521 1790737608.143635 mapread 0x0096b2 thru 0x0152e7 (0xbc36 bytes) 000522 1790737608.185488 trunc from 0x03d798 to 0x03a5d7 (0x31c0 bytes) 000523 1790737608.212344 mapread 0x019096 thru 0x024f14 (0xbe7f bytes) 000524 1790737608.224228 mapwrite 0x0216d0 thru 0x029718 (0x8049 bytes) 000525 1790737608.302683 read 0x00ce40 thru 0x01cad2 (0xfc93 bytes) 000526 1790737608.303328 mapwrite 0x03c896 thru 0x03ffff (0x376a bytes) 000527 1790737608.389760 punch from 0x009f36 to 0x013315 (0x93e0 bytes) 000528 1790737608.397399 trunc from 0x040000 to 0x02143d (0x1ebc2 bytes) 000530 1790737608.424279 write 0x03bef3 thru 0x03ffff (0x410d bytes) 000532 1790737608.426700 write 0x03d623 thru 0x03ffff (0x29dd bytes) 000534 1790737608.433169 write 0x0322cc thru 0x0374a8 (0x51dd bytes) 000536 1790737608.444398 trunc from 0x040000 to 0x02d3da (0x12c25 bytes) 000539 1790737608.466378 mapwrite 0x007c38 thru 0x017825 (0xfbee bytes) 000540 1790737608.595356 trunc from 0x02d3db to 0x01d237 (0x101a3 bytes) 000541 1790737608.618672 trunc from 0x01d238 to 0x00d3ce (0xfe69 bytes) 000542 1790737608.635840 punch from 0x002404 to 0x00d3ce (0xafcb bytes) 000543 1790737608.641619 write 0x03f8d4 thru 0x03ffff (0x72c bytes) 000549 1790737608.652753 mapwrite 0x038186 thru 0x03ffff (0x7e7a bytes) 000550 1790737608.748987 mapwrite 0x010ce9 thru 0x01db6c (0xce84 bytes) 000552 1790737608.934680 trunc from 0x040000 to 0x029c2f (0x163d0 bytes) 000553 1790737608.962712 read 0x00bd92 thru 0x00e7e0 (0x2a4f bytes) 000555 1790737608.980730 MIRROR_EXTEND to 2 mirrors 000556 1790737609.876668 mapread 0x025c1b thru 0x029c2f (0x4015 bytes) 000557 1790737609.903658 MIRROR_RESYNC 2 mirrors 000558 1790737610.354155 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) 000559 1790737610.533272 read 0x004386 thru 0x00c367 (0x7fe2 bytes) 000560 1790737610.575725 punch from 0x014e30 to 0x022735 (0xd906 bytes) 000563 1790737610.594155 write 0x00cca9 thru 0x01aaa6 (0xddfe bytes) 000565 1790737610.607357 punch from 0x00aec0 to 0x01aa38 (0xfb79 bytes) 000566 1790737610.627932 read 0x024aa1 thru 0x029c2f (0x518f bytes) 000567 1790737610.638181 mapread 0x0072a5 thru 0x010ec9 (0x9c25 bytes) 000568 1790737610.639917 read 0x001a63 thru 0x001e5f (0x3fd bytes) 000569 1790737610.640540 mapread 0x022bbd thru 0x029c2f (0x7073 bytes) 000571 1790737610.642217 mapread 0x01eaa7 thru 0x029c2f (0xb189 bytes) 000572 1790737610.643881 trunc from 0x029c30 to 0x01a612 (0xf61d bytes) 000574 1790737610.672043 read 0x00a11b thru 0x018322 (0xe208 bytes) 000575 1790737610.672920 read 0x00fbfa thru 0x013c54 (0x405b bytes) 000576 1790737610.674510 write 0x020493 thru 0x02144a (0xfb8 bytes) 000577 1790737610.675907 mapwrite 0x0370f4 thru 0x03982a (0x2737 bytes) 000578 1790737610.779493 mapread 0x02e066 thru 0x032970 (0x490b bytes) 000584 1790737610.785119 read 0x020548 thru 0x022456 (0x1f0f bytes) 000586 1790737610.787478 punch from 0x026953 to 0x02b073 (0x4721 bytes) 000589 1790737610.793700 mapread 0x02a2ab thru 0x036d2a (0xca80 bytes) 000590 1790737610.804120 mapwrite 0x004788 thru 0x006cd2 (0x254b bytes) 000591 1790737610.850400 trunc from 0x03982b to 0x03787e (0x1fac bytes) 000592 1790737610.880070 read 0x017cf7 thru 0x021c5a (0x9f64 bytes) 000594 1790737610.883685 read 0x02f142 thru 0x031d82 (0x2c41 bytes) 000595 1790737610.885186 mapread 0x022b2f thru 0x027f1f (0x53f1 bytes) 000596 1790737610.887195 trunc from 0x03787f to 0x0336ce (0x41b0 bytes) 000597 1790737610.912669 mapwrite 0x0023ab thru 0x00b0c7 (0x8d1d bytes) 000598 1790737610.972011 mapwrite 0x0354d3 thru 0x03a419 (0x4f47 bytes) 000602 1790737611.069692 mapwrite 0x00f248 thru 0x013ef5 (0x4cae bytes) 000603 1790737611.158037 trunc from 0x03a41a to 0x01cff8 (0x1d421 bytes) 000604 1790737611.195624 mapread 0x01064e thru 0x0143a0 (0x3d53 bytes) 000605 1790737611.206658 MIRROR_EXTEND to 2 mirrors 000606 1790737612.217506 mapwrite 0x01a40b thru 0x02068e (0x6284 bytes) 000607 1790737612.570681 trunc from 0x02068f to 0x0275ab (0x6f1d bytes) 000608 1790737612.609644 mapread 0x02009d thru 0x0275ab (0x750f bytes) 000610 1790737612.630702 trunc from 0x0275ac to 0x001d07 (0x258a4 bytes) 000611 1790737612.660275 mapread 0x0009d7 thru 0x001d07 (0x1331 bytes) 000612 1790737612.670621 mapread 0x000027 thru 0x001d07 (0x1ce1 bytes) 000614 1790737612.673141 punch from 0x000873 to 0x000eae (0x63c bytes) 000615 1790737612.682105 read 0x001302 thru 0x001573 (0x272 bytes) 000616 1790737612.688525 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) 000618 1790737612.906990 trunc from 0x001d08 to 0x0236a8 (0x219a1 bytes) 000619 1790737612.932594 mapread 0x00c198 thru 0x00f1d1 (0x303a bytes) 000620 1790737612.961287 mapwrite 0x00cc02 thru 0x017b76 (0xaf75 bytes) 000621 1790737613.089360 mapwrite 0x025e27 thru 0x032ac2 (0xcc9c bytes) 000622 1790737613.268334 mapread 0x01bd71 thru 0x028440 (0xc6d0 bytes) 000625 1790737613.282467 MIRROR_EXTEND to 2 mirrors 000627 1790737614.315497 mapwrite 0x02d802 thru 0x0315a4 (0x3da3 bytes) 000628 1790737614.603513 read 0x02954f thru 0x02ecd7 (0x5789 bytes) 000629 1790737614.604167 mapread 0x0080be thru 0x010390 (0x82d3 bytes) 000631 1790737614.617566 write 0x012e9c thru 0x015ad2 (0x2c37 bytes) 000632 1790737614.634884 read 0x00c486 thru 0x01be01 (0xf97c bytes) 000634 1790737614.655693 read 0x02f81c thru 0x032ac2 (0x32a7 bytes) 000636 1790737614.656708 mapread 0x02cc01 thru 0x032ac2 (0x5ec2 bytes) 000638 1790737614.659296 write 0x02a30a thru 0x031047 (0x6d3e bytes) 000639 1790737614.661578 trunc from 0x032ac3 to 0x011fd0 (0x20af2 bytes) 000642 1790737614.714052 write 0x02013c thru 0x028cf9 (0x8bbe bytes) 000643 1790737614.717581 trunc from 0x028cfa to 0x010090 (0x18c69 bytes) 000644 1790737614.744960 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x10091): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10091): Mirror 2: 0xa27441c2 (stale) CRC-32 checksum value for chunk [0, 0x10091): Mirror 1: 0xa27441c2 Mirror 3: 0xa27441c2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10091: skipped 000645 1790737615.615547 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) 000648 1790737616.019063 trunc from 0x010091 to 0x02bf33 (0x1bea3 bytes) 000649 1790737616.157721 read 0x00c7c4 thru 0x016951 (0xa18e bytes) 000651 1790737616.175195 write 0x01f961 thru 0x02070a (0xdaa bytes) 000652 1790737616.186535 mapread 0x01d123 thru 0x0223e8 (0x52c6 bytes) 000653 1790737616.215225 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) 000655 1790737616.412027 punch from 0x027e87 to 0x02bf33 (0x40ad bytes) 000656 1790737616.418328 write 0x02e5d5 thru 0x032cc8 (0x46f4 bytes) 000658 1790737616.422229 trunc from 0x032cc9 to 0x01c005 (0x16cc3 bytes) 000663 1790737616.453573 mapread 0x01a79c thru 0x01c005 (0x186a bytes) 000664 1790737616.478561 mapread 0x009251 thru 0x010d7a (0x7b2a bytes) 000671 1790737616.509231 mapwrite 0x01fe53 thru 0x01ff6f (0x11d bytes) 000673 1790737616.639003 mapread 0x01badd thru 0x01ff6f (0x4493 bytes) 000674 1790737616.640115 mapwrite 0x038cf7 thru 0x03ffff (0x7309 bytes) 000676 1790737616.838599 trunc from 0x040000 to 0x016657 (0x299a8 bytes) 000678 1790737616.872034 mapwrite 0x03d9ac thru 0x03ffff (0x2654 bytes) 000679 1790737616.998947 trunc from 0x040000 to 0x036c04 (0x93fb bytes) 000682 1790737617.034597 trunc from 0x036c05 to 0x002486 (0x3477e bytes) 000683 1790737617.103856 punch from 0x001ebe to 0x002486 (0x5c9 bytes) 000685 1790737617.114803 punch from 0x000809 to 0x002486 (0x1c7e bytes) 000687 1790737617.131173 read 0x00071d thru 0x002486 (0x1d6a bytes) 000688 1790737617.140759 read 0x002322 thru 0x002486 (0x165 bytes) 000689 1790737617.141934 read 0x001aac thru 0x002486 (0x9db bytes) 000691 1790737617.153839 MIRROR_EXTEND to 2 mirrors 000692 1790737617.969914 mapwrite 0x00fe93 thru 0x01d446 (0xd5b4 bytes) 000693 1790737618.209731 punch from 0x01a2e2 to 0x01d446 (0x3165 bytes) 000694 1790737618.226567 MIRROR_RESYNC 2 mirrors 000695 1790737619.108188 read 0x00d37d thru 0x0141c3 (0x6e47 bytes) 000696 1790737619.125934 trunc from 0x01d447 to 0x01f74a (0x2304 bytes) 000697 1790737619.282977 punch from 0x0123a6 to 0x01f74a (0xd3a5 bytes) 000700 1790737619.290882 write 0x033642 thru 0x033794 (0x153 bytes) 000701 1790737619.292476 read 0x021a82 thru 0x02c6de (0xac5d bytes) 000702 1790737619.305192 write 0x031114 thru 0x03e7cd (0xd6ba bytes) 000703 1790737619.308173 mapread 0x0284a4 thru 0x02ba85 (0x35e2 bytes) 000704 1790737619.312820 mapwrite 0x00e1f8 thru 0x019db7 (0xbbc0 bytes) 000705 1790737619.381890 read 0x01c0ed thru 0x022cdf (0x6bf3 bytes) 000706 1790737619.391770 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e7ce): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e7ce): Mirror 2: 0x1c72c539 (stale) CRC-32 checksum value for chunk [0, 0x3e7ce): Mirror 1: 0x9e98b061 Mirror 3: 0x9e98b061 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e7ce: skipped 000709 1790737620.582933 punch from 0x0230d3 to 0x029521 (0x644f bytes) 000710 1790737620.678153 read 0x0296a6 thru 0x037852 (0xe1ad bytes) 000711 1790737620.694223 MIRROR_RESYNC 3 mirrors 000712 1790737621.651625 punch from 0x02a732 to 0x02f076 (0x4945 bytes) 000713 1790737621.727464 mapread 0x02d6df thru 0x03a82d (0xd14f bytes) 000714 1790737621.757965 write 0x017f34 thru 0x01b48c (0x3559 bytes) 000715 1790737621.769645 write 0x0284e9 thru 0x035fb6 (0xdace bytes) 000716 1790737621.771369 write 0x00809e thru 0x00ac09 (0x2b6c bytes) 000717 1790737621.786030 trunc from 0x03e7ce to 0x026174 (0x18659 bytes) 000718 1790737621.807186 punch from 0x00a0df to 0x00b195 (0x10b7 bytes) 000719 1790737621.829740 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x26175): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26175): Mirror 2: 0x7e4c3ee9 (stale) Mirror 3: 0x7e4c3ee9 (stale) CRC-32 checksum value for chunk [0, 0x26175): Mirror 1: 0x8e3db6c1 Mirror 4: 0x8e3db6c1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26175: skipped 000720 1790737623.383111 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26175): Mirror 2: 0x7e4c3ee9 (stale) Mirror 3: 0x7e4c3ee9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26175): Mirror 2: 0x7e4c3ee9 (stale) Mirror 3: 0x7e4c3ee9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26175: skipped 000721 1790737623.772805 trunc from 0x026175 to 0x03a2e5 (0x14171 bytes) 000722 1790737623.850824 trunc from 0x03a2e6 to 0x03a780 (0x49b bytes) 000724 1790737623.873670 trunc from 0x03a781 to 0x02a975 (0xfe0b bytes) 000725 1790737623.893085 mapread 0x00caf5 thru 0x018165 (0xb671 bytes) 000726 1790737623.927140 write 0x0321e0 thru 0x033827 (0x1648 bytes) 000730 1790737623.931565 trunc from 0x033828 to 0x01a710 (0x19117 bytes) 000732 1790737623.954880 mapwrite 0x00c056 thru 0x01ad7b (0xed26 bytes) 000733 1790737624.034012 trunc from 0x01ad7c to 0x007692 (0x136e9 bytes) 000734 1790737624.056586 trunc from 0x007693 to 0x006bce (0xac4 bytes) 000735 1790737624.080315 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6bcf): Mirror 3: 0x4fd5b794 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6bcf): Mirror 3: 0x4fd5b794 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6bcf: skipped 000736 1790737624.432224 read 0x0065d4 thru 0x006bce (0x5fb bytes) 000737 1790737624.440929 mapwrite 0x03637c thru 0x039e4e (0x3ad3 bytes) 000738 1790737624.676693 read 0x0225de thru 0x02fe43 (0xd866 bytes) 000739 1790737624.718447 write 0x03bdbb thru 0x03ffff (0x4245 bytes) 000740 1790737624.721526 punch from 0x03090a to 0x03f6cb (0xedc2 bytes) 000741 1790737624.748239 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa9344082 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf3fe3cb Mirror 4: 0xf3fe3cb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000744 1790737626.507072 write 0x03dd2c thru 0x03ffff (0x22d4 bytes) 000745 1790737626.619758 punch from 0x036a2e to 0x03ffff (0x95d2 bytes) 000747 1790737626.634437 mapwrite 0x01762e thru 0x0227e7 (0xb1ba bytes) 000748 1790737626.729002 write 0x005865 thru 0x012a94 (0xd230 bytes) 000751 1790737626.741523 mapwrite 0x02dc84 thru 0x0312f7 (0x3674 bytes) 000753 1790737626.796817 trunc from 0x040000 to 0x00d229 (0x32dd6 bytes) 000754 1790737626.819320 mapread 0x006363 thru 0x00d229 (0x6ec7 bytes) 000755 1790737626.820902 write 0x0022e2 thru 0x011ab4 (0xf7d3 bytes) 000757 1790737626.828688 mapread 0x0057e2 thru 0x006afa (0x1319 bytes) 000758 1790737626.830435 punch from 0x00b2b9 to 0x011ab4 (0x67fc bytes) 000759 1790737626.857646 trunc from 0x011ab5 to 0x00d418 (0x469c bytes) 000760 1790737626.881982 trunc from 0x00d419 to 0x011ae1 (0x46c9 bytes) 000761 1790737626.910132 mapwrite 0x012359 thru 0x016ce1 (0x4989 bytes) 000762 1790737627.011640 punch from 0x0129bd to 0x016ce1 (0x4325 bytes) 000763 1790737627.018696 read 0x005aae thru 0x015729 (0xfc7c bytes) 000765 1790737627.028157 trunc from 0x016ce2 to 0x00fc2a (0x70b7 bytes) 000766 1790737627.055057 write 0x0215b6 thru 0x029dfe (0x8849 bytes) 000767 1790737627.058062 mapwrite 0x022e94 thru 0x032db4 (0xff21 bytes) 000768 1790737627.194369 mapwrite 0x029a6e thru 0x036bed (0xd180 bytes) 000769 1790737627.311010 trunc from 0x036bee to 0x020b87 (0x16066 bytes) 000770 1790737627.334554 write 0x02967a thru 0x02c20e (0x2b95 bytes) 000771 1790737627.335824 write 0x02ec3f thru 0x0337f4 (0x4bb6 bytes) 000772 1790737627.341036 mapwrite 0x009c2b thru 0x00f797 (0x5b6d bytes) 000774 1790737627.461658 mapread 0x0082f8 thru 0x012e36 (0xab3f bytes) 000776 1790737627.487372 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x337f5): Mirror 4: 0x42ff36de (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x337f5): Mirror 4: 0x42ff36de (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x337f5: skipped 000777 1790737628.028133 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x337f5): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x337f5): Mirror 4: 0x42ff36de (stale) CRC-32 checksum value for chunk [0, 0x337f5): Mirror 1: 0x5f475653 Mirror 5: 0x5f475653 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x337f5: skipped 000778 1790737629.541932 punch from 0x0169ee to 0x0188b1 (0x1ec4 bytes) 000779 1790737629.717777 mapwrite 0x030791 thru 0x03aecd (0xa73d bytes) 000780 1790737629.860580 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3aece): Mirror 4: 0xbc200bcc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aece): Mirror 4: 0xbc200bcc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aece: skipped 000781 1790737630.366436 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) 000782 1790737630.538881 punch from 0x0135f4 to 0x019d7f (0x678c bytes) 000783 1790737630.545725 mapread 0x022900 thru 0x025de3 (0x34e4 bytes) 000784 1790737630.572090 read 0x031ddd thru 0x035dd3 (0x3ff7 bytes) 000785 1790737630.596535 punch from 0x001c38 to 0x0039da (0x1da3 bytes) 000786 1790737630.608229 mapread 0x02645c thru 0x02d17f (0x6d24 bytes) 000788 1790737630.621841 trunc from 0x03aece to 0x02dce0 (0xd1ed bytes) 000789 1790737630.644283 read 0x01b4de thru 0x01b55a (0x7d bytes) 000790 1790737630.655510 punch from 0x009154 to 0x0171c8 (0xe075 bytes) 000793 1790737630.666869 read 0x021648 thru 0x0264fb (0x4eb4 bytes) 000794 1790737630.667960 trunc from 0x02dce1 to 0x0296d8 (0x4608 bytes) 000795 1790737630.693820 punch from 0x003fd5 to 0x00ade0 (0x6e0c bytes) 000798 1790737630.702301 mapread 0x0293db thru 0x0296d8 (0x2fe bytes) 000799 1790737630.703690 mapwrite 0x0032e8 thru 0x00f5ce (0xc2e7 bytes) 000800 1790737630.804910 punch from 0x01f475 to 0x027617 (0x81a3 bytes) 000801 1790737630.813242 mapread 0x026ca2 thru 0x0296d8 (0x2a37 bytes) 000802 1790737630.827548 trunc from 0x0296d9 to 0x00b427 (0x1e2b1 bytes) 000803 1790737630.896218 mapwrite 0x0120e3 thru 0x0177bb (0x56d9 bytes) 000804 1790737631.054845 read 0x016cf5 thru 0x0177bb (0xac7 bytes) 000806 1790737631.057058 mapwrite 0x00ffc5 thru 0x01f0ad (0xf0e9 bytes) 000807 1790737631.147580 write 0x0070bc thru 0x00ae23 (0x3d68 bytes) 000808 1790737631.149597 trunc from 0x01f0ae to 0x018cf0 (0x63bd bytes) 000809 1790737631.167818 mapwrite 0x0338a4 thru 0x03ba7f (0x81dc bytes) 000812 1790737631.247821 write 0x02e93a thru 0x03ad75 (0xc43c bytes) 000813 1790737631.250709 mapwrite 0x0011b0 thru 0x002c9b (0x1aec bytes) 000815 1790737631.311023 punch from 0x020c0e to 0x02dba7 (0xcf9a bytes) 000816 1790737631.317183 mapread 0x00b1d1 thru 0x015559 (0xa389 bytes) 000825 1790737631.324676 trunc from 0x03ba80 to 0x021127 (0x1a958 bytes) 000827 1790737631.345738 mapwrite 0x034625 thru 0x03b836 (0x7212 bytes) 000828 1790737631.459897 read 0x031924 thru 0x033e6d (0x254a bytes) 000830 1790737631.483881 trunc from 0x03b837 to 0x00584b (0x35feb bytes) 000831 1790737631.517613 MIRROR_EXTEND to 2 mirrors 000832 1790737632.350545 mapwrite 0x01671e thru 0x01caae (0x6391 bytes) 000833 1790737632.715627 mapwrite 0x0309bb thru 0x03bbea (0xb230 bytes) 000836 1790737632.886784 read 0x004f54 thru 0x005177 (0x224 bytes) 000837 1790737632.894292 read 0x00bdea thru 0x01a8b6 (0xeacd bytes) 000838 1790737632.910546 read 0x039723 thru 0x03bbea (0x24c8 bytes) 000839 1790737632.911590 punch from 0x025fbe to 0x0340e0 (0xe123 bytes) 000840 1790737632.917726 write 0x00cc74 thru 0x01168b (0x4a18 bytes) 000841 1790737632.920740 mapread 0x008e20 thru 0x00e88c (0x5a6d bytes) 000843 1790737632.943522 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) 000844 1790737633.197554 read 0x0235f9 thru 0x0320e2 (0xeaea bytes) 000847 1790737633.210392 mapwrite 0x03746a thru 0x03ffff (0x8b96 bytes) 000848 1790737633.364534 punch from 0x005020 to 0x012d82 (0xdd63 bytes) 000850 1790737633.374828 mapwrite 0x01faa1 thru 0x029384 (0x98e4 bytes) 000851 1790737633.445370 trunc from 0x040000 to 0x01ee3b (0x211c4 bytes) 000852 1790737633.463944 read 0x0124a4 thru 0x018cc0 (0x681d bytes) 000854 1790737633.489771 mapwrite 0x00e144 thru 0x00f52e (0x13eb bytes) 000856 1790737633.538518 read 0x0153fd thru 0x01d9be (0x85c2 bytes) 000858 1790737633.542205 write 0x033579 thru 0x03f052 (0xbada bytes) 000860 1790737633.545451 trunc from 0x03f053 to 0x01d413 (0x21c3f bytes) 000861 1790737633.565892 mapwrite 0x02dc28 thru 0x03d635 (0xfa0e bytes) 000863 1790737633.675065 read 0x032f76 thru 0x0351b7 (0x2242 bytes) 000865 1790737633.677486 mapwrite 0x032790 thru 0x03f569 (0xcdda bytes) 000867 1790737633.759970 punch from 0x00730c to 0x00e268 (0x6f5d bytes) 000870 1790737633.768194 read 0x008ee6 thru 0x009cf5 (0xe10 bytes) 000871 1790737633.778725 write 0x03bc30 thru 0x03ffff (0x43d0 bytes) 000872 1790737633.781151 mapread 0x03f744 thru 0x03ffff (0x8bc bytes) 000873 1790737633.782411 write 0x020385 thru 0x024001 (0x3c7d bytes) 000874 1790737633.784620 punch from 0x0151aa to 0x01a6ed (0x5544 bytes) 000876 1790737633.798930 write 0x03d47c thru 0x03ffff (0x2b84 bytes) 000878 1790737633.801643 punch from 0x00d107 to 0x014a07 (0x7901 bytes) 000879 1790737633.810732 write 0x032649 thru 0x035b02 (0x34ba bytes) 000880 1790737633.812590 mapwrite 0x0112d0 thru 0x01be4d (0xab7e bytes) 000881 1790737633.965759 MIRROR_EXTEND to 2 mirrors 000884 1790737634.984617 punch from 0x004c9c to 0x008a1f (0x3d84 bytes) 000885 1790737635.125821 punch from 0x03ae52 to 0x03b1f8 (0x3a7 bytes) 000886 1790737635.133752 mapwrite 0x03ea89 thru 0x03ffff (0x1577 bytes) 000889 1790737635.214948 write 0x0060fb thru 0x00efd7 (0x8edd bytes) 000891 1790737635.254842 read 0x01589c thru 0x01ac71 (0x53d6 bytes) 000892 1790737635.283750 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7d906e21 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x645d8dd7 Mirror 3: 0x645d8dd7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000893 1790737636.751537 mapread 0x01a635 thru 0x023d06 (0x96d2 bytes) 000894 1790737636.776845 mapread 0x00d747 thru 0x0158f5 (0x81af bytes) 000895 1790737636.790163 mapwrite 0x01f3f1 thru 0x02d6a9 (0xe2b9 bytes) 000897 1790737636.974416 punch from 0x0375e0 to 0x03a63d (0x305e bytes) 000898 1790737636.988630 punch from 0x026908 to 0x028ec7 (0x25c0 bytes) 000900 1790737637.001546 trunc from 0x040000 to 0x01ef3e (0x210c1 bytes) 000902 1790737637.084643 write 0x00d040 thru 0x013b7a (0x6b3b bytes) 000903 1790737637.102450 trunc from 0x01ef3f to 0x00f55f (0xf9df bytes) 000904 1790737637.127871 mapwrite 0x00a153 thru 0x019c3a (0xfae8 bytes) 000906 1790737637.232061 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x19c3b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19c3b): Mirror 2: 0xb23df847 (stale) Mirror 3: 0x6e4bc137 (stale) CRC-32 checksum value for chunk [0, 0x19c3b): Mirror 1: 0xbbcd67ce Mirror 4: 0xbbcd67ce lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19c3b: skipped 000908 1790737638.177674 read 0x00be8a thru 0x0142a5 (0x841c bytes) 000909 1790737638.204466 mapwrite 0x002fa6 thru 0x007938 (0x4993 bytes) 000910 1790737638.443455 mapread 0x017774 thru 0x019c3a (0x24c7 bytes) 000911 1790737638.457127 trunc from 0x019c3b to 0x019952 (0x2e8 bytes) 000912 1790737638.537322 mapwrite 0x03db28 thru 0x03ffff (0x24d8 bytes) 000913 1790737638.674448 punch from 0x0083a5 to 0x010814 (0x8470 bytes) 000915 1790737638.692991 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7d906e21 (stale) Mirror 3: 0x645d8dd7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7d906e21 (stale) Mirror 3: 0x645d8dd7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000917 1790737639.216489 read 0x03edad thru 0x03ffff (0x1253 bytes) 000918 1790737639.223854 mapwrite 0x0128dc thru 0x014756 (0x1e7b bytes) 000919 1790737639.343370 MIRROR_RESYNC 3 mirrors 000920 1790737640.825332 mapread 0x03d42b thru 0x03f07a (0x1c50 bytes) 000921 1790737640.843811 punch from 0x0053fd to 0x011bbc (0xc7c0 bytes) 000922 1790737640.914869 punch from 0x03f26f to 0x03ffff (0xd91 bytes) 000923 1790737640.920689 read 0x000417 thru 0x009c0c (0x97f6 bytes) 000924 1790737640.933862 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xda9ca7dc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xda9ca7dc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000925 1790737641.275717 punch from 0x004c25 to 0x0111c5 (0xc5a1 bytes) 000927 1790737641.292678 read 0x00b7f1 thru 0x00d3b0 (0x1bc0 bytes) 000928 1790737641.313379 read 0x02d503 thru 0x036ee5 (0x99e3 bytes) 000931 1790737641.366189 mapread 0x035225 thru 0x03ffff (0xaddb bytes) 000933 1790737641.387366 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xda9ca7dc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb9c6e1c5 Mirror 3: 0xb9c6e1c5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000935 1790737642.804983 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xda9ca7dc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xda9ca7dc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000936 1790737643.210822 write 0x028b70 thru 0x02e982 (0x5e13 bytes) 000937 1790737643.242422 write 0x02223a thru 0x0318ed (0xf6b4 bytes) 000938 1790737643.252521 read 0x03c7b4 thru 0x03ffff (0x384c bytes) 000940 1790737643.279893 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xda9ca7dc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x77e0faed Mirror 3: 0x77e0faed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000941 1790737644.273181 punch from 0x0312cc to 0x0380c1 (0x6df6 bytes) 000942 1790737644.365059 write 0x018042 thru 0x025d09 (0xdcc8 bytes) 000945 1790737644.378953 punch from 0x02120a to 0x02c58d (0xb384 bytes) 000946 1790737644.399916 mapread 0x00c72c thru 0x011b22 (0x53f7 bytes) 000947 1790737644.435436 mapread 0x02a325 thru 0x039919 (0xf5f5 bytes) 000948 1790737644.464675 mapread 0x02b919 thru 0x031f00 (0x65e8 bytes) 000949 1790737644.465700 mapread 0x03dc42 thru 0x03ffff (0x23be bytes) 000950 1790737644.472056 punch from 0x036d7d to 0x03c8f1 (0x5b75 bytes) 000951 1790737644.486059 write 0x004c15 thru 0x013b18 (0xef04 bytes) 000952 1790737644.501036 mapwrite 0x018634 thru 0x01c0d0 (0x3a9d bytes) 000954 1790737644.684750 punch from 0x01fab1 to 0x01fda2 (0x2f2 bytes) 000955 1790737644.700218 write 0x01b472 thru 0x01d4dc (0x206b bytes) 000956 1790737644.704299 write 0x000407 thru 0x002ef6 (0x2af0 bytes) 000957 1790737644.726559 mapread 0x02e067 thru 0x03444f (0x63e9 bytes) 000958 1790737644.727872 trunc from 0x040000 to 0x012ad7 (0x2d528 bytes) 000959 1790737644.783667 mapread 0x00439a thru 0x0060e8 (0x1d4f bytes) 000960 1790737644.784836 read 0x00ec23 thru 0x012ad7 (0x3eb5 bytes) 000962 1790737644.787346 read 0x00d191 thru 0x012ad7 (0x5947 bytes) 000963 1790737644.795436 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x12ad8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12ad8): Mirror 2: 0x68094417 (stale) Mirror 3: 0xf1507d2f (stale) CRC-32 checksum value for chunk [0, 0x12ad8): Mirror 1: 0x99b46b11 Mirror 4: 0x99b46b11 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12ad8: skipped 000964 1790737645.847910 write 0x00e850 thru 0x01a1c5 (0xb976 bytes) 000965 1790737645.983004 write 0x005441 thru 0x0149be (0xf57e bytes) 000966 1790737645.989757 mapwrite 0x02cd89 thru 0x0303c7 (0x363f bytes) 000967 1790737646.123776 mapwrite 0x0126bd thru 0x01a3e7 (0x7d2b bytes) 000969 1790737646.200470 mapread 0x007b44 thru 0x00c43b (0x48f8 bytes) 000970 1790737646.201357 punch from 0x014366 to 0x02390a (0xf5a5 bytes) 000971 1790737646.213627 read 0x0291cd thru 0x0303c7 (0x71fb bytes) 000972 1790737646.214344 write 0x028714 thru 0x036cc5 (0xe5b2 bytes) 000973 1790737646.216130 punch from 0x022a5e to 0x0309cd (0xdf70 bytes) 000974 1790737646.236040 read 0x0129ba thru 0x012f6e (0x5b5 bytes) 000976 1790737646.238698 punch from 0x01cccf to 0x0218a8 (0x4bda bytes) 000977 1790737646.250964 MIRROR_RESYNC 4 mirrors 000980 1790737647.739593 read 0x013744 thru 0x017f47 (0x4804 bytes) 000981 1790737647.752140 trunc from 0x036cc6 to 0x026645 (0x10680 bytes) 000982 1790737647.860901 read 0x00c9aa thru 0x01c4e1 (0xfb38 bytes) 000983 1790737647.892609 trunc from 0x026646 to 0x03fb83 (0x1953e bytes) 000985 1790737647.937949 trunc from 0x03fb84 to 0x0210d1 (0x1eab2 bytes) 000986 1790737647.976853 read 0x0197e0 thru 0x01e856 (0x5077 bytes) 000987 1790737647.981457 mapread 0x0084a6 thru 0x00bb34 (0x368f bytes) 000988 1790737648.000702 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x210d2): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x210d2): Mirror 2: 0x5c252914 (stale) Mirror 3: 0x5c252914 (stale) Mirror 4: 0x5c252914 (stale) CRC-32 checksum value for chunk [0, 0x210d2): Mirror 1: 0x5c252914 Mirror 5: 0x5c252914 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x210d2: skipped 000989 1790737649.156592 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x210d2): Mirror 2: 0x5c252914 (stale) Mirror 3: 0x5c252914 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x210d2): Mirror 2: 0x5c252914 (stale) Mirror 3: 0x5c252914 (stale) CRC-32 checksum value for chunk [0, 0x210d2): Mirror 1: 0x5c252914 Mirror 5: 0x5c252914 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x210d2: skipped 000990 1790737650.064219 mapwrite 0x021132 thru 0x02a29f (0x916e bytes) 000991 1790737650.421727 read 0x0226c1 thru 0x02a29f (0x7bdf bytes) 000992 1790737650.430234 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2a2a0): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a2a0): Mirror 2: 0xaf6787a9 (stale) Mirror 3: 0xaf6787a9 (stale) Mirror 5: 0xaf6787a9 (stale) CRC-32 checksum value for chunk [0, 0x2a2a0): Mirror 1: 0xd81566be Mirror 6: 0xd81566be lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a2a0: skipped 000993 1790737651.528160 trunc from 0x02a2a0 to 0x026148 (0x4157 bytes) 000994 1790737651.758684 trunc from 0x026149 to 0x007fbf (0x1e189 bytes) 000995 1790737651.781490 trunc from 0x007fc0 to 0x007c33 (0x38c bytes) 000996 1790737651.798485 trunc from 0x007c34 to 0x016f56 (0xf323 bytes) 000997 1790737651.820831 mapwrite 0x01b3d8 thru 0x01efc5 (0x3bee bytes) 000998 1790737651.922471 mapread 0x007ff4 thru 0x00de15 (0x5e22 bytes) 000999 1790737651.937041 MIRROR_RESYNC 5 mirrors All operations completed A-OK! PASS 70a (105s) == sanity-flr test 71: check mirror extend parent fid ==== 23:07:38 (1790737658) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 23:07:42 (1790737662) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.462597 s, 4.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300413 s, 3.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a9:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 23:07:53 (1790737673) Starting client: oleg436-client.virtnet: -o user_xattr,flock 192.168.204.136@tcp:/lustre /mnt/lustre2 Starting client: oleg436-client.virtnet: -o user_xattr,flock 192.168.204.136@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3675424 .. Extending file size to 7459808 .. failed Extending file size to 6368160 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7323616 .. Extending file size to 7482624 .. Extending file size to 7997504 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8603456 .. Extending file size to 9115424 .. Extending file size to 8289184 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8547552 .. Extending file size to 8745792 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57979 57981 57982 57983 57986 done 192.168.204.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg436-client.virtnet /mnt/lustre2 (opts:) 192.168.204.136@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg436-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 23:09:13 (1790737753) Starting client: oleg436-client.virtnet: -o user_xattr,flock 192.168.204.136@tcp:/lustre /mnt/lustre2 Starting client: oleg436-client.virtnet: -o user_xattr,flock 192.168.204.136@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8489024 .. Extending file size to 247296 .. done Extending file size to 6398720 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8503488 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8760960 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9384576 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 59554 59555 59557 59559 59560 192.168.204.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg436-client.virtnet /mnt/lustre2 (opts:) 192.168.204.136@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg436-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 23:10:31 (1790737831) Starting client: oleg436-client.virtnet: -o user_xattr,flock 192.168.204.136@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.176063 s, 11.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.285195 s, 7.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x5cf05733 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg436-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 23:10:52 (1790737852) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123706 s, 17.0 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 23:11:01 (1790737861) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2da:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2da:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2db:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 23:11:09 (1790737869) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.318142 s, 6.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 23:11:18 (1790737878) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.255219 s, 8.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2de:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2de:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 23:11:28 (1790737888) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.374921 s, 5.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c4:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 23:12:04 (1790737924) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg436-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118366 s, 8.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e1:0x0] } PASS 204d (61s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 23:13:05 (1790737985) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg436-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.296438 s, 7.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (58s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 23:14:03 (1790738043) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg436-server mds-ost sync done. Creating new pool test_204f oleg436-server: Pool lustre.test_204f created Adding targets to pool oleg436-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg436-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.23311 s, 9.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg436-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg436-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg436-server: Pool lustre.test_204f destroyed PASS 204f (84s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 23:15:27 (1790738127) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 23:15:35 (1790738135) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 23:15:44 (1790738144) oleg436-server: Pool lustre.test_206 created Adding targets to pool oleg436-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg436-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg436-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg436-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg436-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 23:16:14 (1790738174) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949647 s, 11.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372826 s, 28.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xc557ba88 Mirror 3: 0xc557ba88 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 23:16:26 (1790738186) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 23:16:30 (1790738190) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 23:16:33 (1790738193) Starting client: oleg436-client.virtnet: -o user_xattr,flock 192.168.204.136@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.102827 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xff0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xa40000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xa40000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xa40000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x12c0000 Read this (4 bytes): BBBB 192.168.204.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg436-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 23:17:12 (1790738232) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.011171 s, 733 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 23:17:20 (1790738240) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645291 s, 16.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x22d lmm_fid: [0x200000404:0x22d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x22d lmm_fid: [0x200000404:0x22d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834477 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-29 23:17:22.000000000 -0400 Modify: 2026-09-29 23:17:22.000000000 -0400 Change: 2026-09-29 23:17:24.000000000 -0400 Birth: 2026-09-29 23:17:23.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 23:17:30 (1790738250) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 23:17:34 (1790738254) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.648319 s, 63.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0868343 s, 47.2 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 29 23:17 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22f:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 29 23:17 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 23:17:43 (1790738263) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 23:17:52 (1790738272) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1889 sec =========== 23:18:02 (1790738282) === sanity-flr: start cleanup 23:18:03 (1790738283) === === sanity-flr: finish cleanup 23:18:10 (1790738290) ===