-----============= acceptance-small: sanity-flr ============----- Tue Jun 16 11:00:57 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg123-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 11:01:11 (1781622071) === oleg123-client.virtnet: executing check_config_client /mnt/lustre oleg123-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg123-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b2b86664000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b2b86664000.idle_timeout=debug disable quota as required oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 11:01:35 (1781622095) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7571] == sanity-flr test 0a: lfs mirror create with -N option == 11:01:37 (1781622097) PASS 0a (11s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 11:01:47 (1781622107) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 11:01:49 (1781622109) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 11:01:51 (1781622111) PASS 0d (16s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 11:02:07 (1781622127) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 11:02:09 (1781622129) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 11:02:11 (1781622131) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9b2b86664000.stats=clear osc.lustre-OST0001-osc-ffff9b2b86664000.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 11:02:15 (1781622135) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (6s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 11:02:21 (1781622141) PASS 0j (5s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 11:02:26 (1781622146) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 11:02:28 (1781622148) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 11:02:33 (1781622153) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (7s) == sanity-flr test 2: create components from existing files ========================================================== 11:02:39 (1781622159) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 11:02:45 (1781622165) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 11:02:48 (1781622168) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (5s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 11:02:54 (1781622174) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (5s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 11:03:01 (1781622181) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 11:03:06 (1781622186) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.929482 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107052 s, 9.8 MB/s PASS 21 (13s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 11:03:19 (1781622199) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.854396 s, 16.0 MB/s PASS 22 (13s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 11:03:33 (1781622213) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (17s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 11:03:50 (1781622230) Stopping /mnt/lustre-ost1 (opts:) on oleg123-server oleg123-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b2b86664000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b2b86664000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg123-server: oleg123-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg123-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg123-server oleg123-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b2b86664000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b2b86664000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg123-server: oleg123-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg123-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (79s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 11:05:09 (1781622309) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.283876 s, 22.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg123-server oleg123-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b2b86664000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b2b86664000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg123-server: oleg123-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg123-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (47s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 11:05:56 (1781622356) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg123-server oleg123-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b2b86664000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b2b86664000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg123-server: oleg123-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg123-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg123-server oleg123-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b2b86664000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b2b86664000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg123-server: oleg123-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg123-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (84s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 11:07:21 (1781622441) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.60144 s, 9.4 MB/s -rw-r--r-- 1 root root 52428800 Jun 16 11:07 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.237s user 0m0.000s sys 0m0.197s Stopping /mnt/lustre-ost1 (opts:) on oleg123-server oleg123-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b2b86664000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b2b86664000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.426s user 0m0.000s sys 0m0.202s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg123-server: oleg123-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg123-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg123-server oleg123-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b2b86664000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b2b86664000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.256s user 0m0.000s sys 0m0.155s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg123-server: oleg123-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg123-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg123-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b2b86664000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b2b86664000.ost_server_uuid in FULL state after 0 sec PASS 33b (94s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 11:08:54 (1781622534) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 11:08:57 (1781622537) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 11:09:01 (1781622541) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 11:09:04 (1781622544) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 11:09:10 (1781622550) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.562253 s, 22.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 11:09:22 (1781622562) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265742 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045645 s, 23.0 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 11:09:28 (1781622568) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 11:09:36 (1781622576) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00431814 s, 237 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00148384 s, 690 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0022448 s, 456 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00134535 s, 761 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 11:09:47 (1781622587) create test files with size 15730927 .. e4ea965143be1708bbefa363ea3e5fb6 - 05c7dae3dee239a9168956a4b8cd0907 - f9cd8ea92f7e46baa272d40293aaa75f - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg123-client.virtnet /mnt/lustre (opts:) Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 11:10:12 (1781622612) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (19s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 11:10:31 (1781622631) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 11:10:39 (1781622639) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137427 s, 7.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335691 s, 31.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 11:10:49 (1781622649) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.193822 s, 10.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.524056 s, 8.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.314235 s, 6.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (20s) == sanity-flr test 42: lfs mirror verify ================= 11:11:09 (1781622669) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 11:11:13 (1781622673) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 11:11:17 (1781622677) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438404 s, 23.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 11:11:25 (1781622685) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 11:11:29 (1781622689) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc8:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 11:11:39 (1781622699) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.361737 s, 29.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg123-client.virtnet /mnt/lustre (opts:) Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 11:11:52 (1781622712) llite.lustre-ffff9b2b9156c800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.204685 s, 51.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg123-client.virtnet /mnt/lustre (opts:) Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9b2bb91e4800.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 11:12:06 (1781622726) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (34s) == sanity-flr test 46: Verify setstripe --copy option ==== 11:12:40 (1781622760) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (37s) == sanity-flr test 47: Verify mirror obj alloc =========== 11:13:18 (1781622798) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 11:13:21 (1781622801) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.187954 s, 16.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.186657 s, 16.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x25500ee0 (nosync,stale) Mirror 2: 0x25500ee0 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 9e41f4dcfac6af61aef04725b9cd9f2f - mirror 1 checksum: 4ccdf334e555646972f4bff155e873a5 - mirror 2 checksum: 9e41f4dcfac6af61aef04725b9cd9f2f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-16T11:13:24-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-16T11:13:24-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 9e41f4dcfac6af61aef04725b9cd9f2f - mirror 1 checksum: 4ccdf334e555646972f4bff155e873a5 - mirror 2 checksum: 4ccdf334e555646972f4bff155e873a5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-16T11:13:24-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-16T11:13:24-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 11:13:33 (1781622813) Failing mds1 on oleg123-server Stopping /mnt/lustre-mds1 (opts:) on oleg123-server 11:13:39 (1781622819) shut down facet: mds1 facet_host: oleg123-server facet_failover_host: oleg123-server Failover mds1 to oleg123-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-MDT0000 11:14:01 (1781622841) targets are mounted 11:14:01 (1781622841) facet_failover done oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361076 s, 29.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } PASS 50A (42s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 11:14:15 (1781622855) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 11:14:18 (1781622858) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 11:14:21 (1781622861) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0118094 s, 1.4 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 11:14:28 (1781622868) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 11:14:30 (1781622870) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 11:14:34 (1781622874) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 11:14:57 (1781622897) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 11:15:25 (1781622925) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 11:15:48 (1781622948) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 11:16:11 (1781622971) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 11:16:18 (1781622978) mirror create pid 53718 mirror split pid 53720 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5023: 53718 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5023: 53720 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 11:17:25 (1781623045) Seed set to 3046 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1781623046.860066 write 0x019802 thru 0x021eb9 (0x86b8 bytes) truncating to largest ever: 0x2df0e 000004 1781623046.868498 trunc from 0x021eba to 0x02df0d (0xc054 bytes) truncating to largest ever: 0x2e5a4 000005 1781623046.901810 trunc from 0x02df0e to 0x02e5a3 (0x696 bytes) truncating to largest ever: 0x384d6 000007 1781623046.922165 trunc from 0x02e5a4 to 0x0384d5 (0x9f32 bytes) 000008 1781623046.949145 MIRROR_EXTEND to 2 mirrors 000009 1781623047.844434 mapread 0x025ddf thru 0x0268fd (0xb1f bytes) 000010 1781623047.877051 mapread 0x0198a4 thru 0x0262ab (0xca08 bytes) 000011 1781623047.912784 mapread 0x01a117 thru 0x01d6e5 (0x35cf bytes) 000013 1781623047.940905 trunc from 0x0384d6 to 0x022c3f (0x15896 bytes) 000014 1781623048.036731 write 0x0004b4 thru 0x008d19 (0x8866 bytes) 000021 1781623048.088376 write 0x03a76a thru 0x03db1d (0x33b4 bytes) 000023 1781623048.095664 write 0x00a7cb thru 0x018457 (0xdc8d bytes) 000026 1781623048.127388 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3db1e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3db1e): Mirror 2: 0xf26bc358 (stale) CRC-32 checksum value for chunk [0, 0x3db1e): Mirror 1: 0xb44f703e Mirror 3: 0xb44f703e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3db1e: skipped 000027 1781623049.315138 read 0x00bfb8 thru 0x00ed5e (0x2da7 bytes) 000029 1781623049.344583 read 0x003880 thru 0x010786 (0xcf07 bytes) 000030 1781623049.371839 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3db1e): Mirror 2: 0xf26bc358 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3db1e): Mirror 2: 0xf26bc358 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3db1e: skipped 000031 1781623049.801711 mapread 0x00a78a thru 0x00e8cb (0x4142 bytes) 000032 1781623049.838748 read 0x020d33 thru 0x02d8c0 (0xcb8e bytes) 000033 1781623049.863111 mapread 0x010adf thru 0x01641d (0x593f bytes) 000034 1781623049.878946 mapwrite 0x013009 thru 0x01c126 (0x911e bytes) 000035 1781623050.072723 read 0x02a3e1 thru 0x02ca4f (0x266f bytes) 000039 1781623050.098933 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3db1e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3db1e): Mirror 2: 0xf26bc358 (stale) CRC-32 checksum value for chunk [0, 0x3db1e): Mirror 1: 0xef6193c1 Mirror 3: 0xef6193c1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3db1e: skipped 000041 1781623051.157620 mapwrite 0x0130b4 thru 0x0213d6 (0xe323 bytes) 000042 1781623051.399880 mapread 0x0048e9 thru 0x00f2c7 (0xa9df bytes) 000046 1781623051.463963 mapwrite 0x036c78 thru 0x03daa5 (0x6e2e bytes) 000047 1781623051.553373 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3db1e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3db1e): Mirror 2: 0xf26bc358 (stale) Mirror 3: 0xef6193c1 (stale) CRC-32 checksum value for chunk [0, 0x3db1e): Mirror 1: 0x49752ffd Mirror 4: 0x49752ffd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3db1e: skipped 000049 1781623052.729097 mapwrite 0x00b7f4 thru 0x019a35 (0xe242 bytes) 000050 1781623052.923615 mapwrite 0x003953 thru 0x010c44 (0xd2f2 bytes) 000051 1781623052.969854 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3db1e): Mirror 3: 0xef6193c1 (stale) Mirror 4: 0x49752ffd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3db1e): Mirror 3: 0xef6193c1 (stale) Mirror 4: 0x49752ffd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3db1e: skipped 000052 1781623053.341034 mapread 0x01f631 thru 0x02ba2f (0xc3ff bytes) 000053 1781623053.390484 MIRROR_RESYNC 3 mirrors truncating to largest ever: 0x3bd0f 000057 1781623054.443984 trunc from 0x03db1e to 0x03bd0e (0x1e0f bytes) 000062 1781623054.596552 mapread 0x01ab0e thru 0x028f68 (0xe45b bytes) 000066 1781623054.655674 trunc from 0x03bd0f to 0x021358 (0x1a9b6 bytes) 000067 1781623054.689175 mapwrite 0x035fd1 thru 0x03d010 (0x7040 bytes) 000068 1781623054.777101 write 0x00e548 thru 0x01449b (0x5f54 bytes) 000070 1781623054.819809 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d011): Mirror 4: 0x281baf4a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d011): Mirror 4: 0x281baf4a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d011: skipped 000071 1781623055.380129 trunc from 0x03d011 to 0x00e51d (0x2eaf3 bytes) 000072 1781623055.424372 mapwrite 0x036648 thru 0x03ffff (0x99b8 bytes) 000073 1781623055.551740 trunc from 0x040000 to 0x008797 (0x37868 bytes) 000075 1781623055.574997 trunc from 0x008798 to 0x031c32 (0x2949b bytes) 000078 1781623055.598741 read 0x031391 thru 0x031c32 (0x8a2 bytes) 000079 1781623055.605549 mapwrite 0x002809 thru 0x006074 (0x386c bytes) 000081 1781623055.664891 trunc from 0x031c33 to 0x00cf80 (0x24cb2 bytes) 000082 1781623055.685051 MIRROR_RESYNC 2 mirrors 000083 1781623056.368404 read 0x008c8a thru 0x00a3a8 (0x171f bytes) 000085 1781623056.385142 MIRROR_EXTEND to 3 mirrors 000086 1781623057.199720 mapread 0x00204b thru 0x00614c (0x4102 bytes) 000087 1781623057.244639 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) 000088 1781623057.730047 trunc from 0x00cf81 to 0x034ca5 (0x27d25 bytes) 000089 1781623057.817624 read 0x02576a thru 0x02f4c0 (0x9d57 bytes) 000091 1781623057.842547 read 0x014b46 thru 0x0160e7 (0x15a2 bytes) 000095 1781623057.870911 trunc from 0x034ca6 to 0x0155d0 (0x1f6d5 bytes) 000096 1781623057.925682 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x155d1): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x155d1): Mirror 5: 0xc3efb0e0 (stale) CRC-32 checksum value for chunk [0, 0x155d1): Mirror 1: 0xc3efb0e0 Mirror 6: 0xc3efb0e0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x155d1: skipped 000097 1781623058.908946 trunc from 0x0155d1 to 0x020177 (0xaba7 bytes) 000098 1781623058.990910 write 0x0387dd thru 0x03a72a (0x1f4e bytes) 000099 1781623058.992174 mapread 0x00fee1 thru 0x015c17 (0x5d37 bytes) 000100 1781623059.016771 mapread 0x02dce2 thru 0x032eac (0x51cb bytes) 000101 1781623059.032066 read 0x025aa3 thru 0x027108 (0x1666 bytes) 000103 1781623059.043964 mapread 0x022e52 thru 0x03209d (0xf24c bytes) 000104 1781623059.051906 mapwrite 0x03f15b thru 0x03ffff (0xea5 bytes) 000108 1781623059.235038 mapwrite 0x01d9ba thru 0x01ff10 (0x2557 bytes) 000109 1781623059.274977 trunc from 0x040000 to 0x00ff09 (0x300f6 bytes) 000110 1781623059.288926 write 0x003b18 thru 0x007b29 (0x4012 bytes) 000111 1781623059.300057 mapwrite 0x034755 thru 0x03ffff (0xb8ab bytes) 000112 1781623059.388851 MIRROR_RESYNC 3 mirrors 000114 1781623060.310119 MIRROR_RESYNC 3 mirrors 000115 1781623060.704985 MIRROR_RESYNC 3 mirrors 000116 1781623061.170969 trunc from 0x040000 to 0x02eeec (0x11113 bytes) 000117 1781623061.332674 read 0x0048c2 thru 0x005303 (0xa42 bytes) 000118 1781623061.339933 mapread 0x02c830 thru 0x02eeec (0x26bd bytes) 000119 1781623061.360712 mapread 0x0197df thru 0x01d4d7 (0x3cf9 bytes) 000120 1781623061.372884 read 0x0039a4 thru 0x007968 (0x3fc5 bytes) 000122 1781623061.388326 mapwrite 0x006732 thru 0x00e42a (0x7cf9 bytes) 000123 1781623061.432220 read 0x02812e thru 0x02eeec (0x6dbf bytes) 000124 1781623061.433706 trunc from 0x02eeed to 0x0372ff (0x8413 bytes) 000125 1781623061.453983 read 0x033461 thru 0x0372ff (0x3e9f bytes) 000127 1781623061.466353 mapread 0x02de3e thru 0x0372ff (0x94c2 bytes) 000128 1781623061.467809 read 0x02bb4e thru 0x0372ff (0xb7b2 bytes) 000129 1781623061.469141 trunc from 0x037300 to 0x00bfae (0x2b351 bytes) 000130 1781623061.484384 mapread 0x009e64 thru 0x00bfae (0x214b bytes) 000131 1781623061.486708 read 0x0045d3 thru 0x00bfae (0x79dc bytes) 000133 1781623061.488692 mapwrite 0x031cd0 thru 0x03afc9 (0x92fa bytes) 000135 1781623061.564977 mapwrite 0x0082c5 thru 0x0129f9 (0xa735 bytes) 000138 1781623061.629583 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3afca): Mirror 6: 0x86f606f7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3afca): Mirror 6: 0x86f606f7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3afca: skipped 000140 1781623061.927126 mapread 0x025467 thru 0x028abd (0x3657 bytes) 000141 1781623061.959539 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3afca): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3afca): Mirror 6: 0x86f606f7 (stale) CRC-32 checksum value for chunk [0, 0x3afca): Mirror 1: 0xe57912fd Mirror 7: 0xe57912fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3afca: skipped 000142 1781623062.700184 write 0x01fc49 thru 0x02b349 (0xb701 bytes) 000146 1781623062.814746 mapwrite 0x01866a thru 0x01b8d1 (0x3268 bytes) 000148 1781623062.887839 write 0x01d5d6 thru 0x02b028 (0xda53 bytes) 000149 1781623062.889249 read 0x0376ec thru 0x03afc9 (0x38de bytes) 000150 1781623062.894638 mapwrite 0x02fef9 thru 0x039457 (0x955f bytes) 000151 1781623062.942549 trunc from 0x03afca to 0x009355 (0x31c74 bytes) truncating to largest ever: 0x3f0cb 000153 1781623062.971930 trunc from 0x009356 to 0x03f0ca (0x35d75 bytes) 000155 1781623062.997349 mapread 0x008ba6 thru 0x010698 (0x7af3 bytes) 000156 1781623063.009613 read 0x032660 thru 0x03aa52 (0x83f3 bytes) 000157 1781623063.021021 write 0x0347b0 thru 0x03ffff (0xb850 bytes) 000158 1781623063.027517 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x3454de3e (stale) Mirror 7: 0xa24dd931 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9be0f6fe Mirror 8: 0x9be0f6fe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000160 1781623063.924793 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x3454de3e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x3454de3e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9be0f6fe Mirror 8: 0x9be0f6fe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000161 1781623064.456164 write 0x018b16 thru 0x01aada (0x1fc5 bytes) 000162 1781623064.521223 mapwrite 0x007166 thru 0x00a421 (0x32bc bytes) 000164 1781623064.581776 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x3454de3e (stale) Mirror 8: 0x9be0f6fe (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6df17386 Mirror 9: 0x6df17386 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000168 1781623065.677611 MIRROR_RESYNC 4 mirrors 000169 1781623066.594905 write 0x03a9a2 thru 0x03ffff (0x565e bytes) 000170 1781623066.669037 read 0x03c368 thru 0x03e232 (0x1ecb bytes) 000172 1781623066.671006 read 0x01b012 thru 0x021b33 (0x6b22 bytes) 000173 1781623066.682767 mapwrite 0x02a210 thru 0x0317c9 (0x75ba bytes) 000174 1781623066.742993 trunc from 0x040000 to 0x03105a (0xefa5 bytes) 000175 1781623066.775745 trunc from 0x03105b to 0x02b336 (0x5d24 bytes) 000177 1781623066.795115 mapread 0x01d11f thru 0x02538f (0x8271 bytes) 000178 1781623066.798123 trunc from 0x02b337 to 0x0167c1 (0x14b75 bytes) truncating to largest ever: 0x3f787 000182 1781623066.819847 trunc from 0x0167c2 to 0x03f786 (0x28fc5 bytes) 000183 1781623066.840261 mapread 0x0079b4 thru 0x00868a (0xcd7 bytes) 000184 1781623066.841502 mapwrite 0x01b135 thru 0x02a1fe (0xf0ca bytes) 000185 1781623066.979756 write 0x01bf54 thru 0x01fa17 (0x3ac4 bytes) 000187 1781623066.986135 read 0x016c41 thru 0x02395d (0xcd1d bytes) 000191 1781623067.004473 mapwrite 0x011d7d thru 0x013868 (0x1aec bytes) 000193 1781623067.069070 trunc from 0x03f787 to 0x00f9ef (0x2fd97 bytes) 000194 1781623067.100020 mapwrite 0x006956 thru 0x010f57 (0xa602 bytes) 000195 1781623067.279091 trunc from 0x010f58 to 0x01d8e3 (0xc98c bytes) 000200 1781623067.316885 write 0x010cff thru 0x014052 (0x3354 bytes) 000202 1781623067.342988 write 0x00e56d thru 0x011660 (0x30f4 bytes) 000203 1781623067.344772 write 0x02d521 thru 0x031135 (0x3c15 bytes) 000209 1781623067.349881 mapwrite 0x001da4 thru 0x00b794 (0x99f1 bytes) 000210 1781623067.440434 mapread 0x008856 thru 0x016a90 (0xe23b bytes) 000211 1781623067.463451 mapread 0x00d783 thru 0x013a79 (0x62f7 bytes) 000214 1781623067.474806 write 0x00c2f1 thru 0x0193fb (0xd10b bytes) 000216 1781623067.498975 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x31136): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31136): Mirror 6: 0x3b588976 (stale) Mirror 8: 0x3b588976 (stale) Mirror 9: 0x3b588976 (stale) CRC-32 checksum value for chunk [0, 0x31136): Mirror 1: 0x5519abed Mirror 10: 0x5519abed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31136: skipped 000218 1781623068.519982 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31136): Mirror 8: 0x3b588976 (stale) Mirror 9: 0x3b588976 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31136): Mirror 8: 0x3b588976 (stale) Mirror 9: 0x3b588976 (stale) CRC-32 checksum value for chunk [0, 0x31136): Mirror 1: 0x5519abed Mirror 10: 0x5519abed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31136: skipped 000220 1781623069.203936 MIRROR_RESYNC 4 mirrors 000222 1781623070.053150 mapwrite 0x002396 thru 0x005c9a (0x3905 bytes) 000223 1781623070.262266 mapwrite 0x01e4ee thru 0x0269e1 (0x84f4 bytes) 000225 1781623070.422701 mapwrite 0x008038 thru 0x00f527 (0x74f0 bytes) 000226 1781623070.492135 MIRROR_RESYNC 4 mirrors 000227 1781623071.582241 read 0x00d371 thru 0x015822 (0x84b2 bytes) 000228 1781623071.596002 write 0x03bed6 thru 0x03ffff (0x412a bytes) 000229 1781623071.668000 mapwrite 0x03595d thru 0x03ffff (0xa6a3 bytes) 000230 1781623071.752848 mapwrite 0x017eb9 thru 0x023d0c (0xbe54 bytes) 000231 1781623071.828281 write 0x028877 thru 0x037c8f (0xf419 bytes) 000232 1781623071.830210 mapread 0x02714a thru 0x0363c5 (0xf27c bytes) 000233 1781623071.838722 mapread 0x02b169 thru 0x02b3c0 (0x258 bytes) 000234 1781623071.852906 MIRROR_RESYNC 4 mirrors 000236 1781623072.860506 read 0x0234b1 thru 0x02afa1 (0x7af1 bytes) 000237 1781623072.873983 read 0x029cbc thru 0x03923c (0xf581 bytes) 000241 1781623072.903870 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) 000244 1781623073.309503 read 0x002893 thru 0x00b36e (0x8adc bytes) 000245 1781623073.317755 mapwrite 0x01197f thru 0x015546 (0x3bc8 bytes) 000246 1781623073.503877 write 0x013aaa thru 0x01a8f0 (0x6e47 bytes) 000249 1781623073.511258 mapwrite 0x02f160 thru 0x039c1f (0xaac0 bytes) 000250 1781623073.595785 mapwrite 0x0066cc thru 0x00c762 (0x6097 bytes) 000253 1781623073.664520 trunc from 0x040000 to 0x01706b (0x28f94 bytes) 000256 1781623073.696174 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1706c): Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1706c): Mirror 8: 0x21366037 (stale) Mirror 10: 0x21366037 (stale) CRC-32 checksum value for chunk [0, 0x1706c): Mirror 1: 0x1520597a Mirror 11: 0x1520597a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1706c: skipped 000260 1781623074.643919 trunc from 0x01706c to 0x0252b6 (0xe24b bytes) 000262 1781623074.833733 write 0x03dc3d thru 0x03ffff (0x23c3 bytes) 000263 1781623074.836977 mapwrite 0x00f2c4 thru 0x0142d0 (0x500d bytes) 000266 1781623074.943567 mapwrite 0x0210d9 thru 0x025522 (0x444a bytes) 000270 1781623075.032889 write 0x00df28 thru 0x01d8f6 (0xf9cf bytes) 000271 1781623075.035505 trunc from 0x040000 to 0x004993 (0x3b66c bytes) 000276 1781623075.061548 read 0x002b51 thru 0x004993 (0x1e43 bytes) 000277 1781623075.073442 MIRROR_RESYNC 4 mirrors 000280 1781623076.006958 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) 000281 1781623076.418665 mapread 0x000abb thru 0x004993 (0x3ed9 bytes) 000282 1781623076.465837 read 0x002a9a thru 0x004993 (0x1efa bytes) 000285 1781623076.470166 mapread 0x002147 thru 0x004993 (0x284d bytes) 000287 1781623076.474039 mapwrite 0x03e91d thru 0x03ffff (0x16e3 bytes) 000288 1781623076.723614 mapread 0x02ca0c thru 0x031b2b (0x5120 bytes) 000289 1781623076.745689 trunc from 0x040000 to 0x01ee41 (0x211be bytes) 000293 1781623076.801499 mapwrite 0x01866b thru 0x01aca2 (0x2638 bytes) 000297 1781623076.875688 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ee42): Mirror 8: 0xfd0f2046 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ee42): Mirror 8: 0xfd0f2046 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ee42: skipped 000298 1781623077.233095 write 0x03e68b thru 0x03ffff (0x1975 bytes) 000300 1781623077.246806 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xc3328091 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x15fd6a6 Mirror 9: 0x15fd6a6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000301 1781623078.174494 mapwrite 0x00a2fc thru 0x0173fc (0xd101 bytes) 000302 1781623078.363788 read 0x00a0e6 thru 0x00a48f (0x3aa bytes) 000304 1781623078.365087 read 0x02241a thru 0x024a93 (0x267a bytes) 000305 1781623078.376924 read 0x010ebc thru 0x01dc2e (0xcd73 bytes) 000307 1781623078.378619 trunc from 0x040000 to 0x029b1d (0x164e2 bytes) 000309 1781623078.407782 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29b1e): Mirror 9: 0x770e9b49 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29b1e): Mirror 9: 0x770e9b49 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29b1e: skipped 000310 1781623078.845260 write 0x037369 thru 0x03b3fc (0x4094 bytes) 000312 1781623078.851270 mapread 0x00da42 thru 0x015e08 (0x83c7 bytes) 000313 1781623078.887493 mapread 0x03762a thru 0x03b3fc (0x3dd3 bytes) 000316 1781623078.890826 mapread 0x01db27 thru 0x02b733 (0xdc0d bytes) 000317 1781623078.915471 trunc from 0x03b3fd to 0x023d90 (0x1766c bytes) 000318 1781623078.934035 write 0x03326e thru 0x03c1fa (0x8f8d bytes) 000319 1781623078.937805 read 0x00d799 thru 0x013e28 (0x6690 bytes) 000321 1781623078.940223 write 0x003b37 thru 0x012886 (0xed50 bytes) 000322 1781623078.949113 read 0x02e421 thru 0x03ac7c (0xc85c bytes) 000324 1781623078.960555 read 0x0171d2 thru 0x018b97 (0x19c6 bytes) 000326 1781623078.967150 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3c1fb): Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c1fb): Mirror 9: 0x48f7efab (stale) CRC-32 checksum value for chunk [0, 0x3c1fb): Mirror 1: 0x54c291bb Mirror 10: 0x54c291bb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c1fb: skipped 000329 1781623079.752708 write 0x0256cd thru 0x030bab (0xb4df bytes) 000336 1781623079.867784 MIRROR_RESYNC 3 mirrors 000337 1781623080.589826 mapread 0x004424 thru 0x00ccdc (0x88b9 bytes) 000339 1781623080.613868 mapwrite 0x003551 thru 0x012db8 (0xf868 bytes) 000340 1781623080.761679 MIRROR_RESYNC 3 mirrors 000342 1781623081.521544 mapread 0x0000eb thru 0x00c62a (0xc540 bytes) 000343 1781623081.550897 read 0x028913 thru 0x0371ba (0xe8a8 bytes) 000344 1781623081.588479 MIRROR_EXTEND to 4 mirrors 000346 1781623082.277725 read 0x01a0f5 thru 0x01bc24 (0x1b30 bytes) 000350 1781623082.284511 read 0x03baf6 thru 0x03c1fa (0x705 bytes) 000351 1781623082.295111 mapread 0x01fcb6 thru 0x02d441 (0xd78c bytes) 000352 1781623082.296342 mapwrite 0x00331f thru 0x00df84 (0xac66 bytes) 000354 1781623082.616188 trunc from 0x03c1fb to 0x00df1a (0x2e2e0 bytes) 000357 1781623082.651015 mapwrite 0x03afee thru 0x03f178 (0x418b bytes) 000360 1781623082.773884 mapread 0x00fb09 thru 0x017506 (0x79fe bytes) 000362 1781623082.824469 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3f179): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f179): Mirror 9: 0x14bb4efd (stale) Mirror 10: 0x14bb4efd (stale) Mirror 11: 0x14bb4efd (stale) CRC-32 checksum value for chunk [0, 0x3f179): Mirror 1: 0x5661272b Mirror 12: 0x5661272b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f179: skipped 000363 1781623083.636084 trunc from 0x03f179 to 0x025610 (0x19b68 bytes) 000364 1781623083.708784 MIRROR_RESYNC 5 mirrors 000365 1781623084.520906 MIRROR_EXTEND to 6 mirrors 000366 1781623085.473064 write 0x0195f7 thru 0x027452 (0xde5c bytes) 000367 1781623085.570664 mapread 0x02187f thru 0x027452 (0x5bd4 bytes) 000369 1781623085.572605 write 0x01624e thru 0x0184ef (0x22a2 bytes) 000370 1781623085.605227 read 0x0037b0 thru 0x00a928 (0x7179 bytes) 000371 1781623085.625697 MIRROR_RESYNC 6 mirrors 000372 1781623086.736802 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) 000373 1781623087.046809 MIRROR_EXTEND to 6 mirrors 000378 1781623087.891203 write 0x008251 thru 0x00b213 (0x2fc3 bytes) 000381 1781623087.985473 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27453): Mirror 9: 0x3195f269 (stale) Mirror 11: 0x3195f269 (stale) Mirror 13: 0x3195f269 (stale) Mirror 14: 0x3195f269 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27453): Mirror 9: 0x3195f269 (stale) Mirror 11: 0x3195f269 (stale) Mirror 13: 0x3195f269 (stale) Mirror 14: 0x3195f269 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27453: skipped 000382 1781623088.367221 write 0x03e3d0 thru 0x03ffff (0x1c30 bytes) 000385 1781623088.370185 mapwrite 0x03fe12 thru 0x03ffff (0x1ee bytes) 000386 1781623088.427471 trunc from 0x040000 to 0x0058d7 (0x3a728 bytes) 000388 1781623088.443027 mapwrite 0x02b884 thru 0x035789 (0x9f06 bytes) 000389 1781623088.564076 mapread 0x01103d thru 0x01a752 (0x9716 bytes) 000390 1781623088.607085 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3578a): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3578a): Mirror 9: 0xbd769b2b (stale) Mirror 11: 0xbd769b2b (stale) Mirror 13: 0xbd769b2b (stale) Mirror 14: 0xbd769b2b (stale) CRC-32 checksum value for chunk [0, 0x3578a): Mirror 1: 0x855cab77 Mirror 15: 0x855cab77 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3578a: skipped 000391 1781623089.530905 write 0x001e4c thru 0x006a3a (0x4bef bytes) 000392 1781623089.661059 read 0x02c90f thru 0x035789 (0x8e7b bytes) 000393 1781623089.702796 MIRROR_RESYNC 6 mirrors 000396 1781623090.812533 mapwrite 0x005f8a thru 0x011868 (0xb8df bytes) 000398 1781623090.978074 mapread 0x006a82 thru 0x0098a4 (0x2e23 bytes) 000400 1781623090.986387 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3578a): Mirror 11: 0xbef0099c (stale) Mirror 13: 0xbef0099c (stale) Mirror 14: 0xbef0099c (stale) Mirror 15: 0xbef0099c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3578a): Mirror 11: 0xbef0099c (stale) Mirror 13: 0xbef0099c (stale) Mirror 14: 0xbef0099c (stale) Mirror 15: 0xbef0099c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3578a: skipped 000401 1781623091.411530 mapread 0x019ced thru 0x028ec5 (0xf1d9 bytes) 000402 1781623091.449329 trunc from 0x03578a to 0x02c2e1 (0x94a8 bytes) 000403 1781623091.476107 trunc from 0x02c2e2 to 0x019764 (0x12b7d bytes) 000404 1781623091.489666 write 0x03a2af thru 0x03bbe3 (0x1935 bytes) 000408 1781623091.498568 MIRROR_RESYNC 5 mirrors 000410 1781623092.416450 mapread 0x02a480 thru 0x03926e (0xedef bytes) 000411 1781623092.473401 trunc from 0x03bbe4 to 0x024a09 (0x171da bytes) 000415 1781623092.537171 write 0x0294b6 thru 0x02d5e3 (0x412e bytes) 000418 1781623092.540203 read 0x009a06 thru 0x015adc (0xc0d7 bytes) 000419 1781623092.554670 write 0x004a1d thru 0x012714 (0xdcf8 bytes) 000421 1781623092.569392 write 0x02ca6c thru 0x0340e4 (0x7679 bytes) 000423 1781623092.582737 MIRROR_RESYNC 5 mirrors 000424 1781623093.469814 mapwrite 0x02e432 thru 0x031a75 (0x3644 bytes) 000426 1781623093.599492 read 0x013494 thru 0x0135f3 (0x160 bytes) 000427 1781623093.609669 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x340e5): Mirror 13: 0xfe010e33 (stale) Mirror 14: 0xfe010e33 (stale) Mirror 15: 0xfe010e33 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x340e5): Mirror 13: 0xfe010e33 (stale) Mirror 14: 0xfe010e33 (stale) Mirror 15: 0xfe010e33 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x340e5: skipped 000428 1781623093.936944 read 0x010811 thru 0x012155 (0x1945 bytes) 000430 1781623093.949262 write 0x02c755 thru 0x03b273 (0xeb1f bytes) 000436 1781623093.969257 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3b274): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b274): Mirror 13: 0xbb60a2ac (stale) Mirror 14: 0xbb60a2ac (stale) Mirror 15: 0xbb60a2ac (stale) CRC-32 checksum value for chunk [0, 0x3b274): Mirror 1: 0xdb53f7ff Mirror 16: 0xdb53f7ff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b274: skipped 000437 1781623094.826842 trunc from 0x03b274 to 0x03539c (0x5ed7 bytes) 000442 1781623094.881710 read 0x01e5bd thru 0x029fdf (0xba23 bytes) 000443 1781623094.895231 mapwrite 0x02cd51 thru 0x02d0d7 (0x387 bytes) 000444 1781623094.975408 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3539d): Mirror 13: 0xca10a76e (stale) Mirror 14: 0xca10a76e (stale) Mirror 15: 0xca10a76e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3539d): Mirror 13: 0xca10a76e (stale) Mirror 14: 0xca10a76e (stale) Mirror 15: 0xca10a76e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3539d: skipped 000446 1781623095.381615 mapread 0x00a172 thru 0x00f043 (0x4ed2 bytes) 000447 1781623095.416939 write 0x02c771 thru 0x034cb9 (0x8549 bytes) 000448 1781623095.432854 mapread 0x023464 thru 0x02d3d2 (0x9f6f bytes) 000450 1781623095.452581 mapwrite 0x0217ad thru 0x02da42 (0xc296 bytes) 000452 1781623095.528127 trunc from 0x03539d to 0x025c6d (0xf72f bytes) 000453 1781623095.564753 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x25c6e): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25c6e): Mirror 13: 0x46f0acb (stale) Mirror 14: 0x46f0acb (stale) Mirror 15: 0x46f0acb (stale) CRC-32 checksum value for chunk [0, 0x25c6e): Mirror 1: 0x153b2b89 Mirror 16: 0x153b2b89 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25c6e: skipped 000455 1781623096.359661 trunc from 0x025c6e to 0x018ca9 (0xcfc4 bytes) 000460 1781623096.450647 write 0x00cbbd thru 0x01bd2d (0xf171 bytes) 000461 1781623096.461325 mapwrite 0x02ade1 thru 0x03588e (0xaaae bytes) 000463 1781623096.560077 write 0x02d237 thru 0x02ec99 (0x1a63 bytes) 000464 1781623096.560949 mapwrite 0x000183 thru 0x00f552 (0xf3d0 bytes) 000466 1781623096.625182 read 0x018195 thru 0x0202e5 (0x8151 bytes) 000467 1781623096.635276 read 0x0148d4 thru 0x020112 (0xb83f bytes) 000471 1781623096.638467 mapwrite 0x033fb1 thru 0x03ffff (0xc04f bytes) 000472 1781623096.710540 mapread 0x03b38b thru 0x03ffff (0x4c75 bytes) 000473 1781623096.711861 read 0x014677 thru 0x01e874 (0xa1fe bytes) 000475 1781623096.715245 read 0x031e8d thru 0x034e23 (0x2f97 bytes) 000476 1781623096.716783 mapwrite 0x008e31 thru 0x012e7a (0xa04a bytes) 000477 1781623096.790859 mapwrite 0x029f7d thru 0x0304c0 (0x6544 bytes) 000478 1781623096.844667 mapread 0x03f91a thru 0x03ffff (0x6e6 bytes) 000480 1781623096.849646 mapread 0x01b805 thru 0x022290 (0x6a8c bytes) 000482 1781623096.851986 trunc from 0x040000 to 0x026e44 (0x191bb bytes) 000485 1781623096.866359 read 0x022a53 thru 0x026e44 (0x43f2 bytes) 000486 1781623096.867236 mapread 0x00e23d thru 0x01882f (0xa5f3 bytes) 000487 1781623096.868809 mapwrite 0x03dabe thru 0x03ffff (0x2542 bytes) 000489 1781623096.988831 write 0x03a0c4 thru 0x03f976 (0x58b3 bytes) 000490 1781623096.991160 write 0x013fbd thru 0x01d59a (0x95de bytes) 000491 1781623096.993509 write 0x0284cf thru 0x02b215 (0x2d47 bytes) 000492 1781623096.995831 mapread 0x001a26 thru 0x00bbbd (0xa198 bytes) 000493 1781623096.997540 mapread 0x0368f9 thru 0x03c94a (0x6052 bytes) 000495 1781623097.002654 mapread 0x03b91f thru 0x03fa67 (0x4149 bytes) 000497 1781623097.006169 trunc from 0x040000 to 0x029ea6 (0x16159 bytes) 000499 1781623097.037574 read 0x02964a thru 0x029ea6 (0x85d bytes) 000504 1781623097.042051 read 0x024f91 thru 0x029ea6 (0x4f16 bytes) 000505 1781623097.047275 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29ea7): Mirror 13: 0xf1429935 (stale) Mirror 14: 0xf1429935 (stale) Mirror 15: 0xf1429935 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29ea7): Mirror 13: 0xf1429935 (stale) Mirror 14: 0xf1429935 (stale) Mirror 15: 0xf1429935 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29ea7: skipped 000506 1781623097.394738 mapwrite 0x03a2ad thru 0x03ffff (0x5d53 bytes) 000507 1781623097.475251 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x6cceafc6 (stale) Mirror 14: 0x6cceafc6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x6cceafc6 (stale) Mirror 14: 0x6cceafc6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000508 1781623097.810155 mapread 0x007e9a thru 0x014d28 (0xce8f bytes) 000510 1781623097.838793 mapwrite 0x00d46d thru 0x01b716 (0xe2aa bytes) 000513 1781623097.899478 read 0x03406b thru 0x03ffff (0xbf95 bytes) 000514 1781623097.921114 read 0x00b404 thru 0x00c91f (0x151c bytes) 000515 1781623097.922865 write 0x006290 thru 0x006fa9 (0xd1a bytes) 000518 1781623097.927009 trunc from 0x040000 to 0x013a52 (0x2c5ad bytes) 000522 1781623097.967069 MIRROR_RESYNC 3 mirrors 000523 1781623098.687738 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) 000524 1781623099.161631 mapwrite 0x020a1e thru 0x02ec6f (0xe252 bytes) 000525 1781623099.376267 trunc from 0x02ec70 to 0x0130b3 (0x1bbbc bytes) 000526 1781623099.407523 write 0x011e10 thru 0x0168ae (0x4a9f bytes) 000527 1781623099.418079 read 0x007098 thru 0x00c156 (0x50bf bytes) 000528 1781623099.435740 read 0x01416c thru 0x0168ae (0x2743 bytes) 000529 1781623099.447642 mapread 0x002763 thru 0x00e966 (0xc204 bytes) 000530 1781623099.463723 write 0x0148f5 thru 0x021ac5 (0xd1d1 bytes) 000532 1781623099.467530 write 0x039832 thru 0x03ffff (0x67ce bytes) 000533 1781623099.470265 mapwrite 0x00c5ac thru 0x01174e (0x51a3 bytes) 000535 1781623099.548209 mapread 0x016123 thru 0x017441 (0x131f bytes) 000536 1781623099.550479 read 0x01b2f2 thru 0x01f8aa (0x45b9 bytes) 000540 1781623099.554269 mapread 0x000b20 thru 0x00fa52 (0xef33 bytes) 000541 1781623099.560142 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x3692ecbb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbdbf7352 Mirror 15: 0xbdbf7352 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000543 1781623100.330879 mapread 0x00524e thru 0x01290d (0xd6c0 bytes) 000544 1781623100.364573 write 0x0041a8 thru 0x0115e9 (0xd442 bytes) 000545 1781623100.448929 read 0x006897 thru 0x0148c3 (0xe02d bytes) 000546 1781623100.460265 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xbdbf7352 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xbdbf7352 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000547 1781623100.772136 write 0x0306c9 thru 0x0350d2 (0x4a0a bytes) 000548 1781623100.781834 read 0x00e4d8 thru 0x0183bc (0x9ee5 bytes) 000549 1781623100.792814 trunc from 0x040000 to 0x029bcd (0x16432 bytes) 000550 1781623100.831547 mapwrite 0x00fb54 thru 0x01793e (0x7deb bytes) 000551 1781623100.875703 write 0x0318bd thru 0x033061 (0x17a5 bytes) 000555 1781623100.878124 read 0x031f4c thru 0x032aae (0xb63 bytes) 000556 1781623100.879081 trunc from 0x033062 to 0x00b104 (0x27f5d bytes) 000558 1781623100.896763 mapwrite 0x019011 thru 0x022844 (0x9834 bytes) 000559 1781623100.966271 read 0x0127f5 thru 0x017b12 (0x531e bytes) 000560 1781623100.968216 write 0x015588 thru 0x01f405 (0x9e7e bytes) 000561 1781623100.971450 read 0x01b99d thru 0x01f1b4 (0x3818 bytes) 000562 1781623100.972461 read 0x0114f4 thru 0x01b9d0 (0xa4dd bytes) 000563 1781623100.973114 read 0x004b54 thru 0x00844a (0x38f7 bytes) 000564 1781623100.974021 read 0x01a913 thru 0x01be80 (0x156e bytes) 000566 1781623100.975729 read 0x01d720 thru 0x020d10 (0x35f1 bytes) 000567 1781623100.976759 mapread 0x00c9a0 thru 0x014d6c (0x83cd bytes) 000568 1781623100.978121 mapread 0x00e7f9 thru 0x014f08 (0x6710 bytes) 000569 1781623100.979853 trunc from 0x022845 to 0x018fba (0x988a bytes) 000571 1781623101.003709 trunc from 0x018fbb to 0x032f96 (0x19fdc bytes) 000572 1781623101.024706 mapread 0x030332 thru 0x0310bd (0xd8c bytes) 000573 1781623101.047798 mapwrite 0x03b26d thru 0x03ec85 (0x3a19 bytes) 000574 1781623101.168845 MIRROR_RESYNC 2 mirrors 000575 1781623102.007972 MIRROR_RESYNC 2 mirrors 000577 1781623102.530015 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) 000579 1781623102.732350 write 0x03f33f thru 0x03ffff (0xcc1 bytes) 000581 1781623102.752113 mapwrite 0x019476 thru 0x019f02 (0xa8d bytes) 000584 1781623102.855561 mapwrite 0x014230 thru 0x01825a (0x402b bytes) 000585 1781623102.946625 write 0x00ccf9 thru 0x01b9da (0xece2 bytes) 000591 1781623102.962832 mapread 0x0194a2 thru 0x0217dd (0x833c bytes) 000594 1781623102.982219 write 0x02bf70 thru 0x036142 (0xa1d3 bytes) 000597 1781623103.002469 write 0x03ea65 thru 0x03ffff (0x159b bytes) 000598 1781623103.008807 mapwrite 0x01de7b thru 0x01ea62 (0xbe8 bytes) 000600 1781623103.066562 write 0x02f759 thru 0x03e30b (0xebb3 bytes) 000601 1781623103.069693 trunc from 0x040000 to 0x00db3d (0x324c2 bytes) 000602 1781623103.118166 mapwrite 0x03e32a thru 0x03ffff (0x1cd6 bytes) 000607 1781623103.192689 trunc from 0x040000 to 0x02d036 (0x12fc9 bytes) 000611 1781623103.218459 mapread 0x007c33 thru 0x00ddfd (0x61cb bytes) 000612 1781623103.231751 read 0x00425a thru 0x00f223 (0xafca bytes) 000613 1781623103.232897 mapwrite 0x027625 thru 0x0305c3 (0x8f9f bytes) 000614 1781623103.309901 read 0x024216 thru 0x02428c (0x77 bytes) 000619 1781623103.315544 read 0x00ab90 thru 0x0189bf (0xde30 bytes) 000620 1781623103.344050 MIRROR_EXTEND to 2 mirrors 000622 1781623104.175713 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) 000624 1781623104.379855 MIRROR_EXTEND to 2 mirrors 000625 1781623105.198640 mapwrite 0x026344 thru 0x0346b5 (0xe372 bytes) 000626 1781623105.460980 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x346b6): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x346b6): Mirror 2: 0x38d8c56 (stale) CRC-32 checksum value for chunk [0, 0x346b6): Mirror 1: 0x42352bd2 Mirror 3: 0x42352bd2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x346b6: skipped 000628 1781623106.248761 write 0x018115 thru 0x023e8d (0xbd79 bytes) 000629 1781623106.341390 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x346b6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x346b6): Mirror 2: 0x38d8c56 (stale) Mirror 3: 0x42352bd2 (stale) CRC-32 checksum value for chunk [0, 0x346b6): Mirror 1: 0x8890bb01 Mirror 4: 0x8890bb01 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x346b6: skipped 000630 1781623107.342188 MIRROR_RESYNC 4 mirrors 000631 1781623108.248381 read 0x028a53 thru 0x0341a0 (0xb74e bytes) 000632 1781623108.258950 MIRROR_RESYNC 4 mirrors 000634 1781623108.735833 MIRROR_RESYNC 4 mirrors 000636 1781623109.154900 write 0x018ada thru 0x024c31 (0xc158 bytes) 000637 1781623109.225383 read 0x028ed8 thru 0x02fdbb (0x6ee4 bytes) 000638 1781623109.234724 write 0x006d77 thru 0x010963 (0x9bed bytes) 000639 1781623109.255704 read 0x00a410 thru 0x016145 (0xbd36 bytes) 000640 1781623109.270009 mapwrite 0x02a980 thru 0x02d6d1 (0x2d52 bytes) 000642 1781623109.375228 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x346b6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x346b6): Mirror 2: 0x8890bb01 (stale) Mirror 3: 0x8890bb01 (stale) Mirror 4: 0x8890bb01 (stale) CRC-32 checksum value for chunk [0, 0x346b6): Mirror 1: 0x43d00bcf Mirror 5: 0x43d00bcf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x346b6: skipped 000645 1781623110.387196 trunc from 0x0346b6 to 0x008a3d (0x2bc78 bytes) 000646 1781623110.547035 mapwrite 0x01fb73 thru 0x02e13a (0xe5c8 bytes) 000647 1781623110.721146 mapwrite 0x00e892 thru 0x019aa0 (0xb20f bytes) 000648 1781623110.831311 write 0x03f69f thru 0x03ffff (0x961 bytes) 000650 1781623110.835656 read 0x006e6f thru 0x014431 (0xd5c3 bytes) 000651 1781623110.843754 mapwrite 0x03b7a3 thru 0x03ffff (0x485d bytes) 000652 1781623110.897042 read 0x03b0e4 thru 0x03ffff (0x4f1c bytes) 000657 1781623110.901636 write 0x0376cd thru 0x03ffff (0x8933 bytes) 000659 1781623110.904583 mapwrite 0x005fe1 thru 0x0099c6 (0x39e6 bytes) 000660 1781623110.956593 mapwrite 0x02f862 thru 0x034016 (0x47b5 bytes) 000661 1781623111.003740 mapread 0x03b3a0 thru 0x03ffff (0x4c60 bytes) 000664 1781623111.007788 mapread 0x00709b thru 0x0128f0 (0xb856 bytes) 000665 1781623111.009674 trunc from 0x040000 to 0x005f6b (0x3a094 bytes) 000666 1781623111.031554 mapwrite 0x01d89a thru 0x028194 (0xa8fb bytes) 000667 1781623111.142771 mapread 0x025d65 thru 0x028194 (0x2430 bytes) 000668 1781623111.144390 mapread 0x020e21 thru 0x02810b (0x72eb bytes) 000670 1781623111.146656 read 0x01dee1 thru 0x027122 (0x9242 bytes) 000671 1781623111.148132 trunc from 0x028195 to 0x011c00 (0x16594 bytes) 000672 1781623111.184861 mapread 0x006d03 thru 0x00e16a (0x7468 bytes) 000673 1781623111.196665 read 0x00ea9e thru 0x011c00 (0x3163 bytes) 000675 1781623111.201088 mapwrite 0x03468e thru 0x03ffff (0xb972 bytes) 000676 1781623111.262958 trunc from 0x040000 to 0x02f4db (0x10b24 bytes) 000678 1781623111.279314 mapread 0x012324 thru 0x01cf04 (0xabe1 bytes) 000680 1781623111.286655 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2f4dc): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f4dc): Mirror 2: 0xd6c55620 (stale) Mirror 3: 0xd6c55620 (stale) Mirror 4: 0xd6c55620 (stale) Mirror 5: 0x5094cc88 (stale) CRC-32 checksum value for chunk [0, 0x2f4dc): Mirror 1: 0x49602284 Mirror 6: 0x49602284 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f4dc: skipped 000682 1781623111.893276 write 0x029a24 thru 0x02fe5f (0x643c bytes) 000683 1781623111.965339 trunc from 0x02fe60 to 0x0204e3 (0xf97c bytes) 000684 1781623112.008765 trunc from 0x0204e4 to 0x0207c3 (0x2e0 bytes) 000688 1781623112.031155 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x207c4): Mirror 2: 0xce79873d (stale) Mirror 4: 0xce79873d (stale) Mirror 5: 0x43926320 (stale) Mirror 6: 0xc9871d6d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x207c4): Mirror 2: 0xce79873d (stale) Mirror 4: 0xce79873d (stale) Mirror 5: 0x43926320 (stale) Mirror 6: 0xc9871d6d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x207c4: skipped 000689 1781623112.402951 write 0x0085a8 thru 0x0135a3 (0xaffc bytes) 000691 1781623112.418911 write 0x022436 thru 0x02c4c5 (0xa090 bytes) 000692 1781623112.421047 write 0x030eee thru 0x038f75 (0x8088 bytes) 000693 1781623112.424092 mapwrite 0x0152b8 thru 0x0251e8 (0xff31 bytes) 000694 1781623112.518961 write 0x0269f9 thru 0x02895e (0x1f66 bytes) 000695 1781623112.526762 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38f76): Mirror 2: 0x73fb3213 (stale) Mirror 4: 0x73fb3213 (stale) Mirror 5: 0x2b9c6a1d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38f76): Mirror 2: 0x73fb3213 (stale) Mirror 4: 0x73fb3213 (stale) Mirror 5: 0x2b9c6a1d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38f76: skipped 000697 1781623112.979867 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x38f76): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38f76): Mirror 2: 0x73fb3213 (stale) Mirror 4: 0x73fb3213 (stale) Mirror 5: 0x2b9c6a1d (stale) CRC-32 checksum value for chunk [0, 0x38f76): Mirror 1: 0x28db8583 Mirror 6: 0x28db8583 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38f76: skipped 000699 1781623114.048078 mapwrite 0x029c8c thru 0x0329ee (0x8d63 bytes) 000701 1781623114.334116 trunc from 0x038f76 to 0x0012f7 (0x37c7e bytes) 000702 1781623114.361626 mapread 0x0010ee thru 0x0012f7 (0x20a bytes) 000705 1781623114.373460 mapwrite 0x02d0b0 thru 0x035409 (0x835a bytes) 000706 1781623114.445869 read 0x018db8 thru 0x019d7a (0xfc3 bytes) 000707 1781623114.451066 trunc from 0x03540a to 0x021d81 (0x13688 bytes) 000710 1781623114.471002 write 0x01fad0 thru 0x02f27a (0xf7ab bytes) 000711 1781623114.481533 MIRROR_RESYNC 5 mirrors 000712 1781623115.633842 write 0x029729 thru 0x0381d3 (0xeaab bytes) 000717 1781623115.704373 write 0x029922 thru 0x02dda2 (0x4481 bytes) 000718 1781623115.710925 mapwrite 0x02e427 thru 0x03a455 (0xc02f bytes) 000720 1781623115.858001 mapread 0x023035 thru 0x02ff0e (0xceda bytes) 000724 1781623115.885503 trunc from 0x03a456 to 0x000f12 (0x39543 bytes) 000729 1781623115.904991 read 0x00018e thru 0x000f12 (0xd85 bytes) 000730 1781623115.925075 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf13): Mirror 2: 0x7a56982c (stale) Mirror 5: 0x7a56982c (stale) Mirror 6: 0x7a56982c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf13): Mirror 2: 0x7a56982c (stale) Mirror 5: 0x7a56982c (stale) Mirror 6: 0x7a56982c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf13: skipped 000731 1781623116.358379 mapwrite 0x03c9ae thru 0x03e1d7 (0x182a bytes) 000733 1781623116.456078 trunc from 0x03e1d8 to 0x035b69 (0x866e bytes) 000734 1781623116.478935 mapread 0x013289 thru 0x01446d (0x11e5 bytes) 000736 1781623116.490976 mapread 0x01dc90 thru 0x02d116 (0xf487 bytes) 000740 1781623116.517569 mapread 0x000d0f thru 0x00eb0f (0xde01 bytes) 000741 1781623116.530919 mapwrite 0x027395 thru 0x02e95f (0x75cb bytes) 000742 1781623116.585237 write 0x03e0b1 thru 0x03ffff (0x1f4f bytes) 000743 1781623116.594247 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x98120270 (stale) Mirror 6: 0x98120270 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x98120270 (stale) Mirror 6: 0x98120270 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000746 1781623117.070049 mapwrite 0x039fae thru 0x03ffff (0x6052 bytes) 000747 1781623117.198627 read 0x01f6e7 thru 0x0292f7 (0x9c11 bytes) 000748 1781623117.210909 mapwrite 0x022464 thru 0x024a12 (0x25af bytes) 000749 1781623117.258946 trunc from 0x040000 to 0x00c080 (0x33f7f bytes) 000750 1781623117.289795 mapread 0x003369 thru 0x009e15 (0x6aad bytes) 000751 1781623117.309542 read 0x0059d6 thru 0x00c080 (0x66ab bytes) 000752 1781623117.313315 mapwrite 0x0345cb thru 0x0385fb (0x4031 bytes) 000755 1781623117.413418 read 0x00c78d thru 0x016ca7 (0xa51b bytes) 000756 1781623117.414689 trunc from 0x0385fc to 0x01876d (0x1fe8e bytes) 000757 1781623117.436779 mapread 0x0137ef thru 0x01876d (0x4f7f bytes) 000758 1781623117.438768 write 0x03a792 thru 0x03ffff (0x586e bytes) 000759 1781623117.451465 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x98120270 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x98120270 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000760 1781623117.727271 write 0x010aee thru 0x018e92 (0x83a5 bytes) 000761 1781623117.740002 read 0x01ec7d thru 0x0219e2 (0x2d66 bytes) 000762 1781623117.746992 read 0x0192b9 thru 0x026a22 (0xd76a bytes) 000763 1781623117.759041 mapwrite 0x0145e6 thru 0x016c2e (0x2649 bytes) 000765 1781623117.803775 mapread 0x0174d8 thru 0x02475d (0xd286 bytes) 000766 1781623117.810864 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x98120270 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x953e3ccf Mirror 7: 0x953e3ccf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000767 1781623118.704848 read 0x03ae02 thru 0x03ffff (0x51fe bytes) 000768 1781623118.728973 read 0x03b7a4 thru 0x03ffff (0x485c bytes) 000770 1781623118.749181 MIRROR_RESYNC 3 mirrors 000772 1781623119.466947 read 0x02117d thru 0x02dcaf (0xcb33 bytes) 000773 1781623119.476342 mapread 0x01f424 thru 0x02af8a (0xbb67 bytes) 000774 1781623119.488434 mapread 0x02617e thru 0x02c415 (0x6298 bytes) 000777 1781623119.494739 MIRROR_EXTEND to 4 mirrors truncating to largest ever: 0x3ff20 000778 1781623120.257236 trunc from 0x040000 to 0x03ff1f (0xe0 bytes) 000779 1781623120.414802 read 0x0051ca thru 0x008a94 (0x38cb bytes) 000781 1781623120.423223 mapwrite 0x008d39 thru 0x00dbdd (0x4ea5 bytes) 000782 1781623120.512798 write 0x023cd4 thru 0x02724d (0x357a bytes) 000783 1781623120.545622 MIRROR_RESYNC 4 mirrors 000784 1781623121.530660 MIRROR_EXTEND to 5 mirrors 000785 1781623122.196538 mapwrite 0x0083f8 thru 0x00d637 (0x5240 bytes) 000786 1781623122.353547 mapwrite 0x03a881 thru 0x03c188 (0x1908 bytes) 000790 1781623122.421927 mapwrite 0x0369ab thru 0x03ffff (0x9655 bytes) 000791 1781623122.516475 write 0x015190 thru 0x020602 (0xb473 bytes) 000792 1781623122.527165 mapwrite 0x03b1f8 thru 0x03ffff (0x4e08 bytes) 000794 1781623122.583366 mapwrite 0x00786f thru 0x0097a7 (0x1f39 bytes) 000798 1781623122.620983 trunc from 0x040000 to 0x03a0eb (0x5f14 bytes) 000799 1781623122.648219 write 0x03983b thru 0x03ffff (0x67c5 bytes) 000801 1781623122.653249 trunc from 0x040000 to 0x00233b (0x3dcc4 bytes) 000802 1781623122.669631 trunc from 0x00233c to 0x00a431 (0x80f6 bytes) 000804 1781623122.701130 write 0x025f41 thru 0x02f2ff (0x93bf bytes) 000806 1781623122.703758 mapwrite 0x039d17 thru 0x03ffff (0x62e9 bytes) 000809 1781623122.783894 trunc from 0x040000 to 0x0032b3 (0x3cd4c bytes) 000810 1781623122.819840 write 0x007b41 thru 0x01687d (0xed3d bytes) 000811 1781623122.821848 read 0x0015ce thru 0x00e00d (0xca40 bytes) 000812 1781623122.831759 trunc from 0x01687e to 0x039520 (0x22ca3 bytes) 000814 1781623122.852699 read 0x007d3c thru 0x01683f (0xeb04 bytes) 000817 1781623122.862587 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39521): Mirror 7: 0x4f720ea0 (stale) Mirror 8: 0x4f720ea0 (stale) Mirror 9: 0x4f720ea0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39521): Mirror 7: 0x4f720ea0 (stale) Mirror 8: 0x4f720ea0 (stale) Mirror 9: 0x4f720ea0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39521: skipped 000818 1781623123.271634 mapwrite 0x02dc67 thru 0x03588a (0x7c24 bytes) 000820 1781623123.326365 mapwrite 0x0368c7 thru 0x037b1d (0x1257 bytes) 000822 1781623123.358350 mapwrite 0x02affb thru 0x0354f1 (0xa4f7 bytes) 000824 1781623123.402830 trunc from 0x039521 to 0x0243f5 (0x1512b bytes) 000825 1781623123.419785 read 0x02021e thru 0x0243f5 (0x41d8 bytes) 000828 1781623123.428854 mapwrite 0x01ec6d thru 0x02a586 (0xb91a bytes) 000829 1781623123.508791 mapwrite 0x0253fc thru 0x032235 (0xce3a bytes) 000831 1781623123.574765 mapwrite 0x019a13 thru 0x029910 (0xfefe bytes) 000834 1781623123.624118 read 0x01579f thru 0x019cf4 (0x4556 bytes) 000836 1781623123.631782 read 0x008fb1 thru 0x0164cf (0xd51f bytes) 000839 1781623123.634563 write 0x009f15 thru 0x015950 (0xba3c bytes) 000840 1781623123.636567 write 0x008d1d thru 0x016ff6 (0xe2da bytes) 000842 1781623123.638353 mapread 0x02d4cd thru 0x02ef43 (0x1a77 bytes) 000845 1781623123.640781 read 0x01514a thru 0x017074 (0x1f2b bytes) 000848 1781623123.642818 read 0x005857 thru 0x00e434 (0x8bde bytes) 000850 1781623123.644749 mapwrite 0x01d263 thru 0x026399 (0x9137 bytes) 000854 1781623123.692616 mapwrite 0x022f86 thru 0x02e258 (0xb2d3 bytes) 000858 1781623123.749725 read 0x004c1e thru 0x005750 (0xb33 bytes) 000861 1781623123.751873 mapread 0x00e915 thru 0x012903 (0x3fef bytes) 000863 1781623123.754203 write 0x0335bc thru 0x0372ef (0x3d34 bytes) 000865 1781623123.756189 write 0x02add9 thru 0x037555 (0xc77d bytes) 000868 1781623123.763070 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x37556): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37556): Mirror 7: 0x67800553 (stale) Mirror 8: 0x67800553 (stale) Mirror 9: 0x67800553 (stale) CRC-32 checksum value for chunk [0, 0x37556): Mirror 1: 0x6ce62e1f Mirror 10: 0x6ce62e1f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37556: skipped 000869 1781623124.591086 read 0x03641e thru 0x037555 (0x1138 bytes) 000871 1781623124.613065 mapread 0x026a89 thru 0x02ae92 (0x440a bytes) 000874 1781623124.720267 mapread 0x02240f thru 0x02707b (0x4c6d bytes) 000879 1781623124.794789 mapwrite 0x012280 thru 0x01a58d (0x830e bytes) 000880 1781623125.031859 mapwrite 0x00a8da thru 0x014761 (0x9e88 bytes) 000883 1781623125.087780 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37556): Mirror 8: 0x67800553 (stale) Mirror 9: 0x67800553 (stale) Mirror 10: 0x6ce62e1f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37556): Mirror 8: 0x67800553 (stale) Mirror 9: 0x67800553 (stale) Mirror 10: 0x6ce62e1f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37556: skipped 000885 1781623125.500626 mapwrite 0x03e62c thru 0x03ffff (0x19d4 bytes) 000886 1781623125.558093 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x26733f4 (stale) Mirror 9: 0x26733f4 (stale) Mirror 10: 0xb8965f3b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x94f2e747 Mirror 11: 0x94f2e747 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000887 1781623126.314036 mapread 0x0192b7 thru 0x022cd2 (0x9a1c bytes) 000888 1781623126.349021 read 0x0100d1 thru 0x01f05d (0xef8d bytes) 000889 1781623126.367091 mapread 0x03ec00 thru 0x03ffff (0x1400 bytes) 000890 1781623126.391470 mapwrite 0x013f4c thru 0x022f6f (0xf024 bytes) 000891 1781623126.591807 mapread 0x039be9 thru 0x03ffff (0x6417 bytes) 000892 1781623126.600803 trunc from 0x040000 to 0x001c0e (0x3e3f1 bytes) 000893 1781623126.654991 mapread 0x00030e thru 0x001c0e (0x1901 bytes) 000896 1781623126.675750 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c0f): Mirror 8: 0x994eeb7e (stale) Mirror 10: 0x994eeb7e (stale) Mirror 11: 0x994eeb7e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c0f): Mirror 8: 0x994eeb7e (stale) Mirror 10: 0x994eeb7e (stale) Mirror 11: 0x994eeb7e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c0f: skipped 000897 1781623127.004750 read 0x0011f3 thru 0x001c0e (0xa1c bytes) 000899 1781623127.014819 mapread 0x0017ea thru 0x001c0e (0x425 bytes) 000903 1781623127.022825 mapwrite 0x00a7b7 thru 0x01233e (0x7b88 bytes) 000905 1781623127.158251 MIRROR_RESYNC 4 mirrors 000907 1781623128.017097 read 0x006f4e thru 0x0099a5 (0x2a58 bytes) 000910 1781623128.027264 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) 000911 1781623128.288501 mapwrite 0x0240b7 thru 0x027198 (0x30e2 bytes) 000912 1781623128.469397 mapread 0x000879 thru 0x01024c (0xf9d4 bytes) 000913 1781623128.507637 MIRROR_RESYNC 3 mirrors 000914 1781623129.364749 MIRROR_RESYNC 3 mirrors 000920 1781623129.799618 read 0x0215c8 thru 0x027198 (0x5bd1 bytes) 000922 1781623129.815079 write 0x01f0fe thru 0x020a61 (0x1964 bytes) 000923 1781623129.910777 read 0x003338 thru 0x010eaa (0xdb73 bytes) 000924 1781623129.928451 write 0x004464 thru 0x0108d1 (0xc46e bytes) 000928 1781623129.936814 read 0x012bce thru 0x01d0b9 (0xa4ec bytes) 000929 1781623129.943451 MIRROR_RESYNC 3 mirrors 000930 1781623130.917663 trunc from 0x027199 to 0x022ed8 (0x42c0 bytes) 000931 1781623130.987992 trunc from 0x022ed9 to 0x03106a (0xe192 bytes) 000932 1781623131.013148 read 0x021759 thru 0x023488 (0x1d30 bytes) 000933 1781623131.022333 mapwrite 0x0331ca thru 0x03e747 (0xb57e bytes) 000934 1781623131.133925 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3e748): Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e748): Mirror 10: 0xb9ca25d (stale) Mirror 11: 0xb9ca25d (stale) CRC-32 checksum value for chunk [0, 0x3e748): Mirror 1: 0x4fc1a951 Mirror 12: 0x4fc1a951 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e748: skipped 000936 1781623131.862219 write 0x00ea36 thru 0x01d886 (0xee51 bytes) 000938 1781623131.963572 trunc from 0x03e748 to 0x03d205 (0x1542 bytes) 000939 1781623131.990850 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3d206): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d206): Mirror 10: 0x4ebf56f1 (stale) Mirror 11: 0x4ebf56f1 (stale) Mirror 12: 0x607348ae (stale) CRC-32 checksum value for chunk [0, 0x3d206): Mirror 1: 0xd7577028 Mirror 13: 0xd7577028 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d206: skipped 000940 1781623132.698224 MIRROR_RESYNC 5 mirrors 000941 1781623133.705257 MIRROR_EXTEND to 6 mirrors 000942 1781623134.378560 write 0x02ddec thru 0x03c789 (0xe99e bytes) 000944 1781623134.470867 trunc from 0x03d206 to 0x0154f2 (0x27d13 bytes) 000945 1781623134.501701 write 0x018884 thru 0x025724 (0xcea1 bytes) 000949 1781623134.507395 trunc from 0x025725 to 0x0366d8 (0x10fb4 bytes) 000950 1781623134.527242 mapwrite 0x0249ba thru 0x028250 (0x3897 bytes) 000951 1781623134.607852 mapwrite 0x028e9e thru 0x02a8f2 (0x1a55 bytes) 000952 1781623134.646771 trunc from 0x0366d9 to 0x02ab36 (0xbba2 bytes) 000953 1781623134.676155 mapread 0x01ffb3 thru 0x0260af (0x60fd bytes) 000956 1781623134.679998 trunc from 0x02ab37 to 0x034e67 (0xa331 bytes) 000958 1781623134.697086 trunc from 0x034e68 to 0x03f248 (0xa3e1 bytes) 000959 1781623134.721948 mapwrite 0x00ed6e thru 0x0156e1 (0x6974 bytes) 000961 1781623134.777244 trunc from 0x03f249 to 0x022343 (0x1cf05 bytes) 000963 1781623134.794919 mapread 0x002f8b thru 0x00f0ba (0xc130 bytes) 000965 1781623134.805110 mapread 0x00f4ad thru 0x013349 (0x3e9d bytes) 000969 1781623134.813101 MIRROR_RESYNC 6 mirrors 000970 1781623135.962998 mapwrite 0x022185 thru 0x0291a7 (0x7023 bytes) 000972 1781623136.103417 write 0x00a57d thru 0x0199fe (0xf482 bytes) 000973 1781623136.120839 mapwrite 0x00d1f5 thru 0x017013 (0x9e1f bytes) 000974 1781623136.205861 trunc from 0x0291a8 to 0x0144d4 (0x14cd3 bytes) 000975 1781623136.232868 read 0x007cf7 thru 0x009097 (0x13a1 bytes) 000977 1781623136.249131 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x144d5): Mirror 11: 0x80ade690 (stale) Mirror 12: 0x80ade690 (stale) Mirror 13: 0x80ade690 (stale) Mirror 14: 0x80ade690 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x144d5): Mirror 11: 0x80ade690 (stale) Mirror 12: 0x80ade690 (stale) Mirror 13: 0x80ade690 (stale) Mirror 14: 0x80ade690 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x144d5: skipped 000979 1781623136.653266 mapwrite 0x02c757 thru 0x036fc5 (0xa86f bytes) 000980 1781623136.793323 write 0x0356c9 thru 0x03c3ce (0x6d06 bytes) 000982 1781623136.796607 read 0x014c06 thru 0x016474 (0x186f bytes) 000983 1781623136.803823 mapwrite 0x03c931 thru 0x03ffff (0x36cf bytes) 000984 1781623136.894482 write 0x02eef7 thru 0x03eb64 (0xfc6e bytes) 000985 1781623136.897544 write 0x035608 thru 0x03ffff (0xa9f8 bytes) 000986 1781623136.909087 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x117dcbd6 (stale) Mirror 12: 0x117dcbd6 (stale) Mirror 13: 0x117dcbd6 (stale) Mirror 14: 0x117dcbd6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8a198e45 Mirror 15: 0x8a198e45 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000987 1781623138.033746 mapread 0x010432 thru 0x012ac9 (0x2698 bytes) 000988 1781623138.067794 write 0x029558 thru 0x02ae2e (0x18d7 bytes) 000989 1781623138.151857 read 0x02a36a thru 0x02c5db (0x2272 bytes) 000990 1781623138.152786 mapread 0x025b86 thru 0x028e4d (0x32c8 bytes) 000991 1781623138.161850 read 0x02ef70 thru 0x03be21 (0xceb2 bytes) 000992 1781623138.179961 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x117dcbd6 (stale) Mirror 12: 0x117dcbd6 (stale) Mirror 13: 0x117dcbd6 (stale) Mirror 14: 0x117dcbd6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x117dcbd6 (stale) Mirror 12: 0x117dcbd6 (stale) Mirror 13: 0x117dcbd6 (stale) Mirror 14: 0x117dcbd6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000993 1781623138.606919 write 0x00ffcb thru 0x010480 (0x4b6 bytes) 000995 1781623138.627579 trunc from 0x040000 to 0x02f14d (0x10eb2 bytes) 000999 1781623138.645051 trunc from 0x02f14e to 0x029c49 (0x5504 bytes) 001000 1781623138.666479 mapwrite 0x0090e0 thru 0x014d83 (0xbca4 bytes) All operations completed A-OK! PASS 70a (99s) == sanity-flr test 71: check mirror extend parent fid ==== 11:19:03 (1781623143) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 11:19:06 (1781623146) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.263018 s, 8.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113115 s, 9.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 11:19:15 (1781623155) Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre2 Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3131200 .. Extending file size to 8166656 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6232416 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8342304 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6929984 .. Extending file size to 8533056 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8976384 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9007200 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8424128 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9285696 .. failed Waiting 56924 56925 56927 56929 56930 192.168.201.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg123-client.virtnet /mnt/lustre2 (opts:) 192.168.201.123@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg123-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (78s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 11:20:33 (1781623233) Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre2 Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7749344 .. Extending file size to 8113760 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8508128 .. Extending file size to 8735200 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8807808 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8805184 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8898464 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9058688 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58406 58407 58408 58409 58410 done 192.168.201.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg123-client.virtnet /mnt/lustre2 (opts:) 192.168.201.123@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg123-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (71s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 11:21:44 (1781623304) Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0744157 s, 28.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.332087 s, 6.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x92a0c0b7 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg123-client.virtnet /mnt/lustre2 (opts:) PASS 200c (14s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 11:21:59 (1781623319) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0553306 s, 37.9 MB/s PASS 202 (5s) == sanity-flr test 203: mirror file preserve mirror ID === 11:22:04 (1781623324) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f2:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f8:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } PASS 203 (5s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 11:22:09 (1781623329) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.124135 s, 16.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (5s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 11:22:14 (1781623334) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.141689 s, 14.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } PASS 204b (6s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 11:22:20 (1781623340) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.167977 s, 12.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x318:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x318:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x319:0x0] } PASS 204c (29s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 11:22:49 (1781623369) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg123-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108122 s, 9.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fd:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fd:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fe:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 11:23:45 (1781623425) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg123-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.233736 s, 9.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ff:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ff:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (56s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 11:24:41 (1781623481) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg123-server mds-ost sync done. Creating new pool test_204f oleg123-server: Pool lustre.test_204f created Adding targets to pool oleg123-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg123-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.176421 s, 11.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x300:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x300:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg123-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg123-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg123-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (68s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 11:25:49 (1781623549) PASS 205a (4s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 11:25:53 (1781623553) PASS 205b (5s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 11:25:58 (1781623558) oleg123-server: Pool lustre.test_206 created Adding targets to pool oleg123-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg123-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg123-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg123-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg123-server: Pool lustre.test_206 destroyed PASS 206 (21s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 11:26:19 (1781623579) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631736 s, 16.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364984 s, 28.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xf43b574a Mirror 3: 0xf43b574a PASS 207 (7s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 11:26:26 (1781623586) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 11:26:29 (1781623589) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 11:26:31 (1781623591) Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.106985 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1ac0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xab0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xab0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xab0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xc60000 Read this (4 bytes): BBBB 192.168.201.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg123-client.virtnet /mnt/lustre2 (opts:) PASS 209a (26s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 11:26:57 (1781623617) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0101316 s, 809 kB/s the very first read read with warm cache PASS 209b (5s) == sanity-flr test 210a: handle broken mirrored lovea ==== 11:27:02 (1781623622) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358043 s, 29.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x217 lmm_fid: [0x200000404:0x217:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x217 lmm_fid: [0x200000404:0x217:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30a:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834455 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-16 11:27:03.000000000 -0400 Modify: 2026-06-16 11:27:03.000000000 -0400 Change: 2026-06-16 11:27:04.000000000 -0400 Birth: 2026-06-16 11:27:03.000000000 -0400 PASS 210a (5s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 11:27:07 (1781623627) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 11:27:09 (1781623629) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.342971 s, 119 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0450972 s, 90.8 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 16 11:27 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x219:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 16 11:27 /mnt/lustre/f211.sanity-flr PASS 211 (5s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 11:27:15 (1781623635) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (6s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 11:27:21 (1781623641) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (6s) == sanity-flr test complete, duration 1588 sec =========== 11:27:27 (1781623647) === sanity-flr: start cleanup 11:27:28 (1781623648) === === sanity-flr: finish cleanup 11:27:32 (1781623652) ===