-----============= acceptance-small: sanity-flr ============----- Fri Oct 2 01:56:24 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky use SANITY_FLR_ONLY='' SANITY_FLR_EXCEPT='' oleg216-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 01:56:45 (1790920605) === oleg216-client.virtnet: executing check_config_client /mnt/lustre oleg216-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg216-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9e6502af3800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e6502af3800.idle_timeout=debug disable quota as required oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 01:57:37 (1790920657) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8226] == sanity-flr test 0a: lfs mirror create with -N option == 01:57:40 (1790920660) PASS 0a (23s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:58:03 (1790920683) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:58:08 (1790920688) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:58:12 (1790920692) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:58:38 (1790920718) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:58:42 (1790920722) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 01:58:46 (1790920726) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9e6502af3800.stats=clear osc.lustre-OST0001-osc-ffff9e6502af3800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:58:55 (1790920735) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--ec-map] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:59:06 (1790920746) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:59:16 (1790920756) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 01:59:20 (1790920760) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:59:29 (1790920769) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 01:59:41 (1790920781) PASS 2 (11s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:59:53 (1790920793) PASS 3 (10s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 02:00:03 (1790920803) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (11s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 02:00:14 (1790920814) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (9s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 02:00:27 (1790920827) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 02:00:36 (1790920836) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 1.43049 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997659 s, 10.5 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 02:00:49 (1790920849) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 1.2412 s, 15.2 MB/s PASS 22 (12s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 02:01:01 (1790920861) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 02:01:14 (1790920874) Stopping /mnt/lustre-ost1 (opts:) on oleg216-server oleg216-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e6502af3800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e6502af3800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg216-server: oleg216-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg216-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg216-server oleg216-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9e6502af3800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e6502af3800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg216-server: oleg216-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg216-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (88s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 02:02:42 (1790920962) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.369354 s, 8.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg216-server oleg216-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e6502af3800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e6502af3800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg216-server: oleg216-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg216-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (47s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 02:03:30 (1790921010) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg216-server oleg216-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e6502af3800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e6502af3800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg216-server: oleg216-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg216-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg216-server oleg216-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9e6502af3800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e6502af3800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg216-server: oleg216-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg216-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (92s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 02:05:02 (1790921102) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.74334 s, 14.0 MB/s -rw-r--r-- 1 root root 52428800 Oct 2 02:05 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.351s user 0m0.000s sys 0m0.206s Stopping /mnt/lustre-ost1 (opts:) on oleg216-server oleg216-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e6502af3800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e6502af3800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.336s user 0m0.001s sys 0m0.223s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg216-server: oleg216-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg216-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg216-server oleg216-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9e6502af3800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e6502af3800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.332s user 0m0.000s sys 0m0.199s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg216-server: oleg216-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg216-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg216-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e6502af3800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e6502af3800.ost_server_uuid in FULL state after 0 sec PASS 33b (105s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 02:06:47 (1790921207) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 02:06:51 (1790921211) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 02:06:54 (1790921214) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 02:06:58 (1790921218) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 02:07:05 (1790921225) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.597394 s, 21.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 02:07:18 (1790921238) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249936 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281173 s, 37.3 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 02:07:25 (1790921245) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 02:07:33 (1790921253) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000751132 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00129534 s, 791 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00210381 s, 487 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00123189 s, 831 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 02:07:44 (1790921264) create test files with size 15750657 .. 7cb3a8969b113af6c791785531ec1922 - 17371d6d58a6f5852646f650c65b7992 - dc39b80a50131ad993aec6af6f63a179 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg216-client.virtnet /mnt/lustre (opts:) Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 02:08:10 (1790921290) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 02:08:23 (1790921303) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 02:08:30 (1790921310) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07855 s, 13.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaa:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381107 s, 27.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xab:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xac:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 02:08:40 (1790921320) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0682457 s, 30.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.219763 s, 19.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.224366 s, 9.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 02:08:56 (1790921336) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 02:09:00 (1790921340) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 02:09:04 (1790921344) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191623 s, 54.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 02:09:13 (1790921353) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.389987 s, 8.1 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 02:09:24 (1790921364) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb6:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb6:0x0] } split mirror_id 2 id 131073 error lfs split: file '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' has 1 component, cannot split make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 02:09:35 (1790921375) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.398756 s, 26.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg216-client.virtnet /mnt/lustre (opts:) Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 02:09:49 (1790921389) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9e650901d800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.340765 s, 30.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg216-client.virtnet /mnt/lustre (opts:) Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9e650648b000.xattr_cache=1 PASS 44e (16s) == sanity-flr test 44f: lfs mirror split does not leak the victim file descriptor ========================================================== 02:10:06 (1790921406) error lfs split: create victim file failed: File exists PASS 44f (20s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 02:10:25 (1790921425) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (47s) == sanity-flr test 46: Verify setstripe --copy option ==== 02:11:12 (1790921472) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (44s) == sanity-flr test 47: Verify mirror obj alloc =========== 02:11:57 (1790921517) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 02:12:01 (1790921521) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.28833 s, 10.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.305099 s, 10.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x965cb31f (nosync,stale) Mirror 2: 0x965cb31f (nosync,stale) ** verify mirror 2 doesn't change original checksum: 64ed02d057b575d08514e445364bb015 - mirror 1 checksum: 606ea46c336d369f949b7d07e02c96ab - mirror 2 checksum: 64ed02d057b575d08514e445364bb015 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-10-02T02:12:03-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xeb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-10-02T02:12:03-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 64ed02d057b575d08514e445364bb015 - mirror 1 checksum: 606ea46c336d369f949b7d07e02c96ab - mirror 2 checksum: 606ea46c336d369f949b7d07e02c96ab - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-10-02T02:12:03-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xeb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-10-02T02:12:03-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 02:12:14 (1790921534) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632501 s, 16.6 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (9s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 02:12:22 (1790921542) Failing mds1 on oleg216-server Stopping /mnt/lustre-mds1 (opts:) on oleg216-server 02:12:29 (1790921549) shut down facet: mds1 facet_host: oleg216-server facet_failover_host: oleg216-server Failover mds1 to oleg216-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-MDT0000 02:13:04 (1790921584) targets are mounted 02:13:04 (1790921584) facet_failover done oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391792 s, 26.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x240000404 lmm_object_id: 0x8 lmm_fid: [0x240000404:0x8:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e:0x0] } PASS 50A (64s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 02:13:26 (1790921606) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0485883 s, 422 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0141663 s, 1.4 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (13s) == sanity-flr test 50b: mirror rsync handles sparseness == 02:13:39 (1790921619) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0820762 s, 250 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0106488 s, 1.9 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (15s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 02:13:54 (1790921634) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0113531 s, 1.4 MB/s PASS 50c (12s) == sanity-flr test 50d: mirror rsync keep holes ========== 02:14:06 (1790921646) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0964925 s, 212 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (14s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 02:14:21 (1790921661) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00959529 s, 427 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0353999 s, 116 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 02:14:30 (1790921670) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) PASS 60b (26s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 02:14:57 (1790921697) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 02:15:28 (1790921728) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 02:15:53 (1790921753) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (27s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 02:16:20 (1790921780) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (10s) == sanity-flr test 70: mirror create and split race ====== 02:16:30 (1790921790) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 58283 mirror split pid 58285 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5127: 58283 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5127: 58285 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 02:17:39 (1790921859) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 1861 000002 1790921861.865368 mapwrite 0x020c32 thru 0x028223 (0x75f2 bytes) 000003 1790921861.931182 punch from 0x0063c8 to 0x010d82 (0xa9bb bytes) 000004 1790921861.936646 mapwrite 0x03e972 thru 0x03ffff (0x168e bytes) fallocating to largest ever: 0x3e766 000005 1790921861.995061 fallocate 0x02faed thru 0x03e765 (0xec79 bytes) 000006 1790921862.003387 read 0x00e85f thru 0x01db7a (0xf31c bytes) 000007 1790921862.013212 trunc from 0x0220b9 to 0x02764a (0x5592 bytes) 000009 1790921862.018639 read 0x0004c1 thru 0x005ce9 (0x5829 bytes) 000010 1790921862.032086 write 0x008bae thru 0x010d61 (0x81b4 bytes) 000011 1790921862.033727 trunc from 0x008ebf to 0x0145fc (0xb73e bytes) 000012 1790921862.056241 write 0x00a652 thru 0x00b9b7 (0x1366 bytes) 000013 1790921862.066850 read 0x02ceed thru 0x03b41f (0xe533 bytes) 000015 1790921862.069402 mapwrite 0x030796 thru 0x038caa (0x8515 bytes) 000016 1790921862.093851 read 0x0063c0 thru 0x00c625 (0x6266 bytes) 000017 1790921862.106057 trunc from 0x040000 to 0x02b4ba (0x14b45 bytes) 000018 1790921862.122718 write 0x0079ec thru 0x00a221 (0x2836 bytes) 000019 1790921862.126462 fallocate 0x00c56a thru 0x00ee65 (0x28fc bytes) 000020 1790921862.131704 punch from 0x0074ff to 0x009ca1 (0x27a3 bytes) 000022 1790921862.142306 trunc from 0x02b4bb to 0x022c8a (0x8830 bytes) 000023 1790921862.164741 trunc from 0x022c8b to 0x0278f7 (0x4c6d bytes) 000024 1790921862.187030 fallocate 0x01bc81 thru 0x01e0ab (0x242b bytes) 000025 1790921862.194487 punch from 0x004f29 to 0x00762c (0x2704 bytes) 000026 1790921862.201753 mapwrite 0x019615 thru 0x02555c (0xbf48 bytes) 000027 1790921862.252817 mapread 0x0117bc thru 0x015242 (0x3a87 bytes) 000028 1790921862.254157 punch from 0x001e74 to 0x005617 (0x37a4 bytes) 000029 1790921862.259697 fallocate 0x02fd47 thru 0x0326a8 (0x2962 bytes) 000030 1790921862.265017 write 0x006954 thru 0x00d805 (0x6eb2 bytes) 000031 1790921862.270997 read 0x005153 thru 0x0124da (0xd388 bytes) 000032 1790921862.282910 write 0x02ef0f thru 0x03b53f (0xc631 bytes) 000033 1790921862.286054 write 0x02d23f thru 0x036aae (0x9870 bytes) 000034 1790921862.293720 trunc from 0x03239f to 0x0334c4 (0x1126 bytes) 000036 1790921862.320367 trunc from 0x031322 to 0x036da6 (0x5a85 bytes) 000037 1790921862.325651 read 0x02051d thru 0x02d702 (0xd1e6 bytes) 000038 1790921862.329397 trunc from 0x01c79d to 0x01e760 (0x1fc4 bytes) 000039 1790921862.339429 MIRROR_EXTEND to 2 mirrors 000040 1790921863.603265 punch from 0x003f5c to 0x00b9ab (0x7a50 bytes) 000041 1790921863.647060 trunc from 0x01e880 to 0x0237de (0x4f5f bytes) 000042 1790921863.658755 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa6:0x0]: Invalid argument (22) 000043 1790921863.853678 trunc from 0x03b540 to 0x006b90 (0x349af bytes) 000044 1790921863.877806 punch from 0x003b7c to 0x006b90 (0x3015 bytes) 000045 1790921863.882937 write 0x008e65 thru 0x00a60c (0x17a8 bytes) 000046 1790921863.885047 punch from 0x0028a1 to 0x006555 (0x3cb5 bytes) 000048 1790921863.890409 punch from 0x001bf6 to 0x00a60c (0x8a17 bytes) 000049 1790921863.902664 trunc from 0x00a60d to 0x037618 (0x2d00c bytes) 000050 1790921863.917293 read 0x00a03b thru 0x00c113 (0x20d9 bytes) 000051 1790921863.923341 mapwrite 0x00fe4e thru 0x01cd06 (0xceb9 bytes) 000052 1790921863.970517 fallocate 0x020a85 thru 0x0241ed (0x3769 bytes) 000053 1790921863.976152 write 0x02aaf3 thru 0x03a4c7 (0xf9d5 bytes) 000054 1790921863.982945 write 0x00282a thru 0x006338 (0x3b0f bytes) 000055 1790921863.994749 trunc from 0x001cdd to 0x005f7e (0x42a2 bytes) 000057 1790921864.012427 mapread 0x038533 thru 0x03a4c7 (0x1f95 bytes) fallocating to largest ever: 0x40000 000059 1790921864.015382 fallocate 0x03a400 thru 0x03ffff (0x5c00 bytes) 000060 1790921864.023703 mapread 0x018d5f thru 0x019a5c (0xcfe bytes) 000061 1790921864.026112 mapwrite 0x03c312 thru 0x03ffff (0x3cee bytes) 000062 1790921864.156928 mapwrite 0x020cb8 thru 0x02f9bb (0xed04 bytes) 000063 1790921864.301804 MIRROR_EXTEND to 2 mirrors 000064 1790921865.639797 write 0x02827e thru 0x030bfd (0x8980 bytes) 000065 1790921865.698303 punch from 0x0171fe to 0x021f8b (0xad8e bytes) 000066 1790921865.703508 write 0x00aac4 thru 0x0148fe (0x9e3b bytes) 000067 1790921865.714463 write 0x01d83e thru 0x027545 (0x9d08 bytes) 000068 1790921865.729440 punch from 0x02d174 to 0x03b810 (0xe69d bytes) 000070 1790921865.751113 trunc from 0x040000 to 0x02cc28 (0x133d7 bytes) 000071 1790921865.774268 mapwrite 0x030514 thru 0x034eb1 (0x499e bytes) 000072 1790921865.860260 mapread 0x005b14 thru 0x00d5de (0x7acb bytes) 000073 1790921865.873754 trunc from 0x034eb2 to 0x00ba5c (0x29455 bytes) 000074 1790921865.900319 mapread 0x004b72 thru 0x00ba5c (0x6eeb bytes) 000076 1790921865.902737 punch from 0x005103 to 0x006671 (0x156f bytes) 000077 1790921865.909400 read 0x008b08 thru 0x00ba5c (0x2f55 bytes) 000078 1790921865.910098 write 0x00cc93 thru 0x0148ed (0x7c5b bytes) 000079 1790921865.912191 fallocate 0x01e520 thru 0x028679 (0xa15a bytes) 000080 1790921865.917446 trunc from 0x01c429 to 0x01efb9 (0x2b91 bytes) 000081 1790921865.922813 write 0x0389f0 thru 0x03dc12 (0x5223 bytes) 000082 1790921865.925699 mapread 0x017581 thru 0x01e8ec (0x736c bytes) 000083 1790921865.935441 mapread 0x0146a2 thru 0x02139c (0xccfb bytes) 000084 1790921865.944795 mapwrite 0x0025a2 thru 0x0059bb (0x341a bytes) 000085 1790921865.996286 MIRROR_RESYNC 2 mirrors 000086 1790921867.574131 trunc from 0x03dc13 to 0x009a3e (0x341d4 bytes) 000087 1790921867.634491 fallocate 0x0296a8 thru 0x02c9bd (0x3316 bytes) 000088 1790921867.660532 trunc from 0x00f44f to 0x012460 (0x3012 bytes) 000089 1790921867.669073 fallocate 0x027933 thru 0x029d79 (0x2447 bytes) 000090 1790921867.690327 MIRROR_RESYNC 2 mirrors 000091 1790921869.012614 write 0x03f440 thru 0x03fd8d (0x94e bytes) 000092 1790921869.101054 trunc from 0x03f76d to 0x03fd8d (0x621 bytes) 000093 1790921869.131155 mapwrite 0x00a16b thru 0x00da0c (0x38a2 bytes) 000094 1790921869.247170 punch from 0x01f612 to 0x027674 (0x8063 bytes) 000095 1790921869.280512 punch from 0x03700e to 0x03fd8d (0x8d80 bytes) 000096 1790921869.326627 mapread 0x021365 thru 0x02ed40 (0xd9dc bytes) 000097 1790921869.379209 trunc from 0x03fd8e to 0x004401 (0x3b98c bytes) 000099 1790921869.465464 mapwrite 0x01d4be thru 0x01e7ea (0x132d bytes) 000100 1790921869.587767 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1e7eb): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e7eb): Mirror 2: 0x13e16986 (stale) CRC-32 checksum value for chunk [0, 0x1e7eb): Mirror 1: 0xddddca83 Mirror 3: 0xddddca83 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e7eb: skipped 000101 1790921871.084959 mapread 0x0154e9 thru 0x01c007 (0x6b1f bytes) 000102 1790921871.149481 trunc from 0x01e7eb to 0x0106fb (0xe0ef bytes) 000103 1790921871.276634 trunc from 0x0106fc to 0x0280d0 (0x179d5 bytes) 000104 1790921871.318089 mapread 0x0111d8 thru 0x020f0b (0xfd34 bytes) 000105 1790921871.418668 fallocate 0x015f31 thru 0x025c23 (0xfcf3 bytes) 000106 1790921871.438135 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x280d1): Mirror 2: 0xf2df79d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x280d1): Mirror 2: 0xf2df79d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x280d1: skipped 000107 1790921871.970852 MIRROR_RESYNC 2 mirrors 000108 1790921872.911953 read 0x02156a thru 0x0280d0 (0x6b67 bytes) 000109 1790921872.931994 trunc from 0x024c8a to 0x0280d0 (0x3447 bytes) 000110 1790921873.012276 mapread 0x01a947 thru 0x01aaea (0x1a4 bytes) 000111 1790921873.022309 trunc from 0x0280d1 to 0x03330c (0xb23c bytes) 000112 1790921873.066745 fallocate 0x0205d1 thru 0x027061 (0x6a91 bytes) 000113 1790921873.098379 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3330d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3330d): Mirror 2: 0x83a3e30a (stale) CRC-32 checksum value for chunk [0, 0x3330d): Mirror 1: 0x83a3e30a Mirror 3: 0x83a3e30a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3330d: skipped 000114 1790921874.253700 mapwrite 0x0100c9 thru 0x01918b (0x90c3 bytes) 000115 1790921874.368637 punch from 0x01e79c to 0x026e87 (0x86ec bytes) 000116 1790921874.377598 punch from 0x014bf5 to 0x01b2be (0x66ca bytes) 000117 1790921874.387572 read 0x02655f thru 0x02e91b (0x83bd bytes) 000118 1790921874.409104 MIRROR_RESYNC 3 mirrors 000119 1790921875.614067 mapread 0x020c0f thru 0x026e3b (0x622d bytes) 000121 1790921875.678660 fallocate 0x018e98 thru 0x0287c9 (0xf932 bytes) 000122 1790921875.719462 punch from 0x02cf38 to 0x03330c (0x63d5 bytes) 000123 1790921875.741932 MIRROR_RESYNC 3 mirrors 000125 1790921876.770066 trunc from 0x03330d to 0x018f0a (0x1a402 bytes) 000126 1790921876.818760 fallocate 0x029151 thru 0x02de0a (0x4cba bytes) 000127 1790921876.832302 mapwrite 0x02b77d thru 0x03303f (0x78c3 bytes) 000128 1790921876.903629 trunc from 0x033040 to 0x03db5c (0xab1d bytes) 000129 1790921876.924287 write 0x01a2a0 thru 0x0268b9 (0xc61a bytes) 000130 1790921876.932774 trunc from 0x03db5d to 0x03a5ca (0x3592 bytes) 000131 1790921876.964426 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a5cb): Mirror 3: 0x9fe0d7a2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a5cb): Mirror 3: 0x9fe0d7a2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a5cb: skipped 000134 1790921877.476926 read 0x02c314 thru 0x02dee1 (0x1bce bytes) 000135 1790921877.491035 MIRROR_RESYNC 2 mirrors 000136 1790921878.834249 punch from 0x00afdb to 0x010aa2 (0x5ac8 bytes) 000137 1790921878.908512 mapwrite 0x01ac2e thru 0x029f08 (0xf2db bytes) 000138 1790921879.030880 mapread 0x01f397 thru 0x023479 (0x40e3 bytes) 000139 1790921879.043971 fallocate 0x0097cc thru 0x012400 (0x8c35 bytes) 000140 1790921879.058824 mapread 0x037b0f thru 0x03a5ca (0x2abc bytes) 000141 1790921879.079665 trunc from 0x03a5cb to 0x0138d0 (0x26cfa bytes) 000142 1790921879.111251 write 0x038eb8 thru 0x03ffff (0x7148 bytes) 000143 1790921879.112751 fallocate 0x035cb5 thru 0x039cde (0x402a bytes) 000144 1790921879.128775 trunc from 0x00751e to 0x015005 (0xdae8 bytes) 000145 1790921879.136687 read 0x00f289 thru 0x01578f (0x6507 bytes) 000146 1790921879.152668 read 0x018c42 thru 0x01961f (0x9de bytes) 000147 1790921879.182547 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd100ae31 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x393206d2 Mirror 4: 0x393206d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000148 1790921880.380079 trunc from 0x040000 to 0x00aff4 (0x3500b bytes) 000149 1790921880.504341 mapwrite 0x03c829 thru 0x03ffff (0x37d7 bytes) 000150 1790921880.573233 fallocate 0x0299a6 thru 0x0322cd (0x8928 bytes) 000151 1790921880.579492 trunc from 0x01ba97 to 0x027672 (0xbbdc bytes) 000152 1790921880.586207 trunc from 0x040000 to 0x03f302 (0xcfd bytes) 000153 1790921880.609933 fallocate 0x018ea0 thru 0x01eba3 (0x5d04 bytes) 000154 1790921880.623895 mapread 0x01808a thru 0x021c35 (0x9bac bytes) 000155 1790921880.660645 fallocate 0x01cbfb thru 0x01e68f (0x1a95 bytes) 000156 1790921880.673768 MIRROR_RESYNC 3 mirrors 000157 1790921881.654364 write 0x011c0d thru 0x011d98 (0x18c bytes) 000159 1790921881.696176 trunc from 0x001872 to 0x002bac (0x133b bytes) 000160 1790921881.703667 fallocate 0x02e740 thru 0x02feee (0x17af bytes) 000161 1790921881.710822 mapwrite 0x028c62 thru 0x038a57 (0xfdf6 bytes) 000162 1790921881.787789 read 0x02fe34 thru 0x03f302 (0xf4cf bytes) 000163 1790921881.795510 mapread 0x02db7a thru 0x033b74 (0x5ffb bytes) 000164 1790921881.798966 punch from 0x02ce8e to 0x03cdc6 (0xff39 bytes) 000165 1790921881.804376 write 0x03d849 thru 0x03ffff (0x27b7 bytes) 000166 1790921881.805514 mapread 0x00cdae thru 0x018738 (0xb98b bytes) 000167 1790921881.825662 read 0x03ce63 thru 0x03f2c7 (0x2465 bytes) 000168 1790921881.841177 MIRROR_RESYNC 3 mirrors 000170 1790921883.060033 mapwrite 0x016f65 thru 0x01b9a4 (0x4a40 bytes) 000171 1790921883.187349 mapwrite 0x0072be thru 0x007e1e (0xb61 bytes) 000172 1790921883.230780 mapwrite 0x02d3c9 thru 0x0324b2 (0x50ea bytes) 000173 1790921883.267232 write 0x03c99e thru 0x03d5c8 (0xc2b bytes) 000174 1790921883.281409 mapread 0x0009ce thru 0x007d2d (0x7360 bytes) 000175 1790921883.289090 write 0x000cf0 thru 0x002f66 (0x2277 bytes) 000177 1790921883.291297 read 0x00451f thru 0x00787d (0x335f bytes) 000178 1790921883.292726 read 0x03cdf8 thru 0x03ffff (0x3208 bytes) 000179 1790921883.304111 punch from 0x02b132 to 0x0338c8 (0x8797 bytes) 000180 1790921883.311599 mapwrite 0x0309b0 thru 0x03a1c6 (0x9817 bytes) 000181 1790921883.358753 mapwrite 0x02e3a4 thru 0x031361 (0x2fbe bytes) 000182 1790921883.386174 trunc from 0x011a1c to 0x021074 (0xf659 bytes) 000183 1790921883.392493 trunc from 0x040000 to 0x00361d (0x3c9e2 bytes) 000184 1790921883.419647 punch from 0x0023be to 0x00361d (0x1260 bytes) 000185 1790921883.426332 read 0x0006d5 thru 0x00361d (0x2f49 bytes) 000186 1790921883.434346 mapwrite 0x03b39d thru 0x03f3f1 (0x4055 bytes) 000187 1790921883.508666 mapread 0x005e9f thru 0x00aaca (0x4c2c bytes) 000188 1790921883.509998 trunc from 0x0124a9 to 0x02009e (0xdbf6 bytes) 000189 1790921883.517259 read 0x005e03 thru 0x00db84 (0x7d82 bytes) 000190 1790921883.518248 fallocate 0x03fecc thru 0x03ffff (0x134 bytes) 000191 1790921883.523767 punch from 0x03964e to 0x03ffff (0x69b2 bytes) 000192 1790921883.528722 read 0x0023d3 thru 0x011812 (0xf440 bytes) 000193 1790921883.530099 mapread 0x032100 thru 0x035a82 (0x3983 bytes) 000194 1790921883.532016 punch from 0x035160 to 0x036064 (0xf05 bytes) 000195 1790921883.536941 mapread 0x00cb6d thru 0x0104e7 (0x397b bytes) 000196 1790921883.538635 mapread 0x028892 thru 0x03050c (0x7c7b bytes) 000197 1790921883.540312 punch from 0x029d63 to 0x035109 (0xb3a7 bytes) 000198 1790921883.545938 trunc from 0x040000 to 0x016fe6 (0x29019 bytes) 000199 1790921883.564966 trunc from 0x0107cb to 0x016fe6 (0x681c bytes) 000200 1790921883.576577 fallocate 0x03ca14 thru 0x03ffff (0x35ec bytes) 000201 1790921883.592771 fallocate 0x029465 thru 0x02e852 (0x53ee bytes) 000202 1790921883.604413 mapread 0x02888e thru 0x02b647 (0x2dba bytes) 000204 1790921883.618306 fallocate 0x0267ea thru 0x02de05 (0x761c bytes) 000205 1790921883.623969 punch from 0x00543e to 0x00de04 (0x89c7 bytes) 000206 1790921883.634943 MIRROR_RESYNC 3 mirrors 000207 1790921884.612664 fallocate 0x00df59 thru 0x01374a (0x57f2 bytes) 000208 1790921884.682907 MIRROR_RESYNC 3 mirrors 000209 1790921885.523215 write 0x012667 thru 0x019193 (0x6b2d bytes) 000211 1790921885.578574 mapread 0x0140f7 thru 0x01ab33 (0x6a3d bytes) 000212 1790921885.588120 write 0x016a1e thru 0x021c9f (0xb282 bytes) 000213 1790921885.597187 trunc from 0x0003c7 to 0x00f558 (0xf192 bytes) 000214 1790921885.602632 punch from 0x025dda to 0x02e852 (0x8a79 bytes) 000215 1790921885.607319 fallocate 0x01e1ab thru 0x01ffc6 (0x1e1c bytes) 000216 1790921885.618982 mapwrite 0x016263 thru 0x01bf9e (0x5d3c bytes) 000217 1790921885.654322 read 0x01ed01 thru 0x023c6f (0x4f6f bytes) 000218 1790921885.667271 fallocate 0x021f85 thru 0x02f5dc (0xd658 bytes) 000219 1790921885.678069 fallocate 0x0137d4 thru 0x01e4e1 (0xad0e bytes) 000220 1790921885.685801 fallocate 0x03371b thru 0x03ffff (0xc8e5 bytes) 000221 1790921885.692822 mapwrite 0x00d125 thru 0x010db2 (0x3c8e bytes) 000222 1790921885.764361 fallocate 0x01f82c thru 0x025b31 (0x6306 bytes) 000223 1790921885.776809 trunc from 0x02f5dd to 0x027b0a (0x7ad2 bytes) 000224 1790921885.806841 write 0x03f691 thru 0x03ffff (0x96f bytes) 000225 1790921885.809532 mapwrite 0x000324 thru 0x00b771 (0xb44e bytes) 000226 1790921885.846730 read 0x036c97 thru 0x03ffff (0x9369 bytes) 000227 1790921885.866940 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xaa0344e (stale) Mirror 4: 0xaa0344e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xebf3a7e4 Mirror 5: 0xebf3a7e4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000228 1790921887.298808 read 0x0381fc thru 0x03fef0 (0x7cf5 bytes) 000229 1790921887.326490 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xaa0344e (stale) Mirror 4: 0xaa0344e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xaa0344e (stale) Mirror 4: 0xaa0344e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000230 1790921887.710378 trunc from 0x009ffb to 0x00dad0 (0x3ad6 bytes) 000231 1790921887.733053 fallocate 0x0265f8 thru 0x027b40 (0x1549 bytes) 000232 1790921887.740594 mapwrite 0x03a6ee thru 0x03e7ed (0x4100 bytes) 000234 1790921887.788926 fallocate 0x001ac1 thru 0x0112e2 (0xf822 bytes) 000235 1790921887.797517 mapwrite 0x01b258 thru 0x024539 (0x92e2 bytes) 000237 1790921887.857554 fallocate 0x02ae3a thru 0x02eed0 (0x4097 bytes) 000238 1790921887.865250 mapwrite 0x01a13c thru 0x029b2d (0xf9f2 bytes) 000239 1790921887.899738 mapwrite 0x02c779 thru 0x03a3a5 (0xdc2d bytes) 000240 1790921887.943402 mapwrite 0x00c5e9 thru 0x00cc16 (0x62e bytes) 000242 1790921887.996472 mapread 0x015eed thru 0x022567 (0xc67b bytes) 000244 1790921888.000079 mapwrite 0x00e6e6 thru 0x010cac (0x25c7 bytes) 000245 1790921888.023936 trunc from 0x01bc12 to 0x024256 (0x8645 bytes) 000246 1790921888.031347 read 0x03d982 thru 0x03ffff (0x267e bytes) 000247 1790921888.033266 mapwrite 0x020eb8 thru 0x024bed (0x3d36 bytes) 000248 1790921888.071075 trunc from 0x040000 to 0x020c0b (0x1f3f4 bytes) 000249 1790921888.110063 trunc from 0x020c0c to 0x0387a4 (0x17b99 bytes) 000251 1790921888.146271 punch from 0x01c7f8 to 0x0281c8 (0xb9d1 bytes) 000252 1790921888.153474 fallocate 0x03be13 thru 0x03ffff (0x41ed bytes) 000253 1790921888.160306 read 0x0167b9 thru 0x026340 (0xfb88 bytes) 000254 1790921888.178666 trunc from 0x01df63 to 0x0252fb (0x7399 bytes) 000255 1790921888.183772 trunc from 0x00c916 to 0x01749d (0xab88 bytes) 000256 1790921888.191612 trunc from 0x0387a5 to 0x03314f (0x5655 bytes) 000257 1790921888.211704 mapwrite 0x01e383 thru 0x0297d1 (0xb44f bytes) 000258 1790921888.288968 fallocate 0x01decb thru 0x024522 (0x6658 bytes) 000259 1790921888.301472 MIRROR_RESYNC 3 mirrors 000260 1790921889.576424 write 0x00fc75 thru 0x01b827 (0xbbb3 bytes) 000261 1790921889.621252 read 0x01fcac thru 0x0213fa (0x174f bytes) 000262 1790921889.628845 trunc from 0x033150 to 0x035fca (0x2e7b bytes) 000263 1790921889.656102 mapwrite 0x01b0ba thru 0x01e028 (0x2f6f bytes) 000264 1790921889.710502 mapwrite 0x02dc5a thru 0x035941 (0x7ce8 bytes) 000265 1790921889.745301 write 0x03f3b4 thru 0x03f3c1 (0xe bytes) 000266 1790921889.746804 trunc from 0x03f3c2 to 0x03d15f (0x2262 bytes) 000267 1790921889.764323 read 0x038bd0 thru 0x03bbb7 (0x2fe8 bytes) 000268 1790921889.793660 MIRROR_RESYNC 3 mirrors 000269 1790921891.029830 read 0x0309af thru 0x03d15f (0xc7b1 bytes) 000270 1790921891.062846 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa6:0x0]: Invalid argument (22) 000271 1790921891.456659 mapread 0x02d8aa thru 0x0391a2 (0xb8f9 bytes) 000272 1790921891.511835 read 0x02a7b3 thru 0x02cfd3 (0x2821 bytes) 000273 1790921891.514058 read 0x01db72 thru 0x023fc6 (0x6455 bytes) 000274 1790921891.527665 trunc from 0x03d160 to 0x02f3dd (0xdd82 bytes) 000275 1790921891.580394 mapread 0x01985f thru 0x027260 (0xda02 bytes) 000276 1790921891.693950 write 0x035664 thru 0x0368b9 (0x1256 bytes) 000277 1790921891.696226 trunc from 0x00fb70 to 0x010d4a (0x11db bytes) 000278 1790921891.706930 read 0x00501b thru 0x00cacf (0x7ab5 bytes) 000279 1790921891.729833 punch from 0x004e10 to 0x007aae (0x2c9f bytes) 000282 1790921891.743576 fallocate 0x037c54 thru 0x03ffff (0x83ac bytes) 000283 1790921891.753557 fallocate 0x0251a1 thru 0x027155 (0x1fb5 bytes) 000284 1790921891.763902 fallocate 0x018f39 thru 0x023d65 (0xae2d bytes) 000286 1790921891.781829 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x368ba): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x368ba): Mirror 4: 0x6f23c07c (stale) CRC-32 checksum value for chunk [0, 0x368ba): Mirror 1: 0x277991aa Mirror 5: 0x277991aa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x368ba: skipped 000287 1790921893.282139 fallocate 0x037601 thru 0x03cf15 (0x5915 bytes) 000288 1790921893.337362 mapwrite 0x031cc1 thru 0x039309 (0x7649 bytes) 000289 1790921893.404197 mapread 0x03cc0c thru 0x03cf15 (0x30a bytes) 000290 1790921893.415792 mapwrite 0x03c9f6 thru 0x03ffff (0x360a bytes) 000291 1790921893.499485 write 0x00404d thru 0x011a0d (0xd9c1 bytes) 000292 1790921893.522278 read 0x036771 thru 0x03ffff (0x988f bytes) 000294 1790921893.536685 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xf779856 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xf779856 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000295 1790921894.012726 mapread 0x0075d7 thru 0x0130ab (0xbad5 bytes) 000297 1790921894.043828 mapread 0x01ee39 thru 0x0272cc (0x8494 bytes) 000298 1790921894.062252 trunc from 0x013eb9 to 0x01fa1b (0xbb63 bytes) 000299 1790921894.067848 mapwrite 0x00a5b3 thru 0x015691 (0xb0df bytes) 000300 1790921894.104398 punch from 0x0070f9 to 0x00fede (0x8de6 bytes) 000301 1790921894.110566 trunc from 0x0295ac to 0x038629 (0xf07e bytes) 000302 1790921894.116896 fallocate 0x02fa30 thru 0x030dd7 (0x13a8 bytes) 000303 1790921894.122632 mapread 0x0235e4 thru 0x026aa9 (0x34c6 bytes) 000304 1790921894.124088 trunc from 0x040000 to 0x005184 (0x3ae7b bytes) 000305 1790921894.167086 trunc from 0x005185 to 0x010e93 (0xbd0f bytes) 000306 1790921894.198386 punch from 0x006848 to 0x010e93 (0xa64c bytes) 000307 1790921894.214818 fallocate 0x02be8d thru 0x0368c8 (0xaa3c bytes) 000309 1790921894.236843 trunc from 0x010e94 to 0x025615 (0x14782 bytes) 000310 1790921894.263086 punch from 0x01242f to 0x01de96 (0xba68 bytes) 000311 1790921894.272055 fallocate 0x02d699 thru 0x03d67b (0xffe3 bytes) 000312 1790921894.280410 write 0x017e14 thru 0x0187a4 (0x991 bytes) 000313 1790921894.291760 trunc from 0x01000f to 0x01afe3 (0xafd5 bytes) 000314 1790921894.304220 mapread 0x021a70 thru 0x025615 (0x3ba6 bytes) 000315 1790921894.314825 mapwrite 0x00a804 thru 0x01047a (0x5c77 bytes) 000316 1790921894.351183 read 0x000685 thru 0x002b5e (0x24da bytes) 000317 1790921894.358268 trunc from 0x025616 to 0x007526 (0x1e0ef bytes) 000318 1790921894.382954 mapwrite 0x038b80 thru 0x03e972 (0x5df3 bytes) 000319 1790921894.441233 mapwrite 0x01ce8b thru 0x02736d (0xa4e3 bytes) 000321 1790921894.485805 trunc from 0x03e973 to 0x01a37a (0x245f8 bytes) 000323 1790921894.512384 read 0x0071f5 thru 0x016c47 (0xfa53 bytes) 000324 1790921894.525355 trunc from 0x01a37b to 0x0142cc (0x60ae bytes) 000325 1790921894.540435 punch from 0x010087 to 0x013bd4 (0x3b4e bytes) 000326 1790921894.544646 write 0x0351da thru 0x03f54e (0xa375 bytes) 000327 1790921894.547158 trunc from 0x03f54f to 0x00d7c9 (0x31d85 bytes) 000329 1790921894.565912 punch from 0x00147e to 0x008a55 (0x75d8 bytes) 000330 1790921894.573160 trunc from 0x00d7ca to 0x02b339 (0x1db70 bytes) 000332 1790921894.592544 punch from 0x006951 to 0x01237c (0xba2c bytes) 000333 1790921894.598306 trunc from 0x02b33a to 0x025ebd (0x547c bytes) 000334 1790921894.616164 mapwrite 0x01c08d thru 0x02a4ff (0xe473 bytes) 000335 1790921894.665819 trunc from 0x02a500 to 0x007b16 (0x229e9 bytes) 000336 1790921894.686231 punch from 0x0045ad to 0x007b16 (0x356a bytes) 000337 1790921894.691352 write 0x03ef98 thru 0x03ffff (0x1068 bytes) 000338 1790921894.697806 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa6:0x0]: Invalid argument (22) 000339 1790921894.844656 fallocate 0x03034d thru 0x037ab6 (0x776a bytes) 000340 1790921894.850620 mapread 0x00073a thru 0x00bf31 (0xb7f8 bytes) 000341 1790921894.884686 punch from 0x03f602 to 0x03ffff (0x9fe bytes) 000342 1790921894.891736 read 0x007f10 thru 0x01769d (0xf78e bytes) 000343 1790921894.906177 mapread 0x01f39b thru 0x021f04 (0x2b6a bytes) 000344 1790921894.916913 fallocate 0x03fd6f thru 0x03ffff (0x291 bytes) 000345 1790921894.922410 mapwrite 0x03cb60 thru 0x03ffff (0x34a0 bytes) 000346 1790921894.951441 read 0x035fb4 thru 0x038e09 (0x2e56 bytes) 000347 1790921894.952084 trunc from 0x040000 to 0x006884 (0x3977b bytes) 000348 1790921894.976771 punch from 0x0058e1 to 0x006884 (0xfa4 bytes) 000349 1790921894.982763 fallocate 0x006d01 thru 0x00aae6 (0x3de6 bytes) 000350 1790921894.988505 mapwrite 0x005171 thru 0x00db30 (0x89c0 bytes) 000352 1790921895.066085 MIRROR_EXTEND to 2 mirrors 000353 1790921895.746424 mapwrite 0x024872 thru 0x032564 (0xdcf3 bytes) 000354 1790921895.893139 punch from 0x02f530 to 0x032564 (0x3035 bytes) 000355 1790921895.902710 mapread 0x024f2d thru 0x032564 (0xd638 bytes) 000356 1790921895.911650 trunc from 0x032565 to 0x01479e (0x1ddc6 bytes) 000357 1790921895.938661 trunc from 0x00138e to 0x0033ea (0x205d bytes) 000358 1790921895.945574 mapwrite 0x025744 thru 0x031c7d (0xc53a bytes) 000359 1790921896.016180 punch from 0x00965b to 0x015622 (0xbfc8 bytes) 000361 1790921896.024309 read 0x02f8d3 thru 0x031c7d (0x23ab bytes) 000362 1790921896.025120 fallocate 0x02a3de thru 0x030f9b (0x6bbe bytes) 000363 1790921896.035165 punch from 0x01a163 to 0x026695 (0xc533 bytes) 000364 1790921896.044707 write 0x03c2a7 thru 0x03ffff (0x3d59 bytes) 000365 1790921896.046323 trunc from 0x00fd7e to 0x012095 (0x2318 bytes) 000366 1790921896.064670 read 0x0163af thru 0x019e08 (0x3a5a bytes) 000367 1790921896.093758 trunc from 0x01f855 to 0x0296c5 (0x9e71 bytes) 000368 1790921896.140573 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa6:0x0]: Invalid argument (22) 000369 1790921896.357255 trunc from 0x040000 to 0x026798 (0x19867 bytes) 000370 1790921896.391235 trunc from 0x026799 to 0x0328df (0xc147 bytes) 000371 1790921896.423817 trunc from 0x00008d to 0x001dd6 (0x1d4a bytes) 000372 1790921896.428317 punch from 0x02306c to 0x02a0c0 (0x7055 bytes) 000373 1790921896.435089 write 0x0128bd thru 0x01dc63 (0xb3a7 bytes) 000376 1790921896.453931 mapread 0x015568 thru 0x023c17 (0xe6b0 bytes) 000378 1790921896.474366 mapread 0x0012a9 thru 0x0065cb (0x5323 bytes) 000379 1790921896.486765 mapread 0x01e841 thru 0x022b71 (0x4331 bytes) 000380 1790921896.487665 trunc from 0x0328e0 to 0x03cce4 (0xa405 bytes) 000381 1790921896.508385 trunc from 0x00f650 to 0x01c8ff (0xd2b0 bytes) 000382 1790921896.543708 write 0x00c5f8 thru 0x00f193 (0x2b9c bytes) 000383 1790921896.559113 punch from 0x01627b to 0x01e779 (0x84ff bytes) 000384 1790921896.583338 fallocate 0x015583 thru 0x01b7e6 (0x6264 bytes) 000385 1790921896.603369 fallocate 0x0394b7 thru 0x03cc0f (0x3759 bytes) 000386 1790921896.616195 trunc from 0x03cce5 to 0x0332d9 (0x9a0b bytes) 000387 1790921896.676436 fallocate 0x006903 thru 0x00d22f (0x692d bytes) 000388 1790921896.690839 read 0x00e2de thru 0x01bb7b (0xd89e bytes) 000389 1790921896.729746 write 0x006d16 thru 0x0148b3 (0xdb9e bytes) 000390 1790921896.736627 mapwrite 0x0299d7 thru 0x031085 (0x76af bytes) 000391 1790921896.881952 trunc from 0x0332da to 0x03a75e (0x7485 bytes) 000392 1790921896.919087 trunc from 0x03a75f to 0x00cce8 (0x2da76 bytes) 000393 1790921896.956030 trunc from 0x00cce9 to 0x014b0c (0x7e24 bytes) 000394 1790921896.979738 fallocate 0x033c19 thru 0x036f81 (0x3369 bytes) 000395 1790921896.986056 mapwrite 0x0136eb thru 0x01cbf6 (0x950c bytes) 000396 1790921897.048572 fallocate 0x0207f4 thru 0x02be53 (0xb660 bytes) 000397 1790921897.055072 trunc from 0x01871b to 0x021078 (0x895e bytes) 000399 1790921897.067578 fallocate 0x00ec8c thru 0x010247 (0x15bc bytes) 000400 1790921897.072196 punch from 0x00d91a to 0x018b03 (0xb1ea bytes) 000401 1790921897.079131 write 0x000262 thru 0x001181 (0xf20 bytes) 000402 1790921897.088641 mapread 0x004a56 thru 0x00d589 (0x8b34 bytes) 000403 1790921897.091054 mapread 0x01c250 thru 0x02b174 (0xef25 bytes) 000404 1790921897.108688 read 0x006d91 thru 0x00dec4 (0x7134 bytes) 000405 1790921897.109246 trunc from 0x018f18 to 0x020395 (0x747e bytes) 000406 1790921897.115541 punch from 0x024736 to 0x02be53 (0x771e bytes) 000407 1790921897.126234 fallocate 0x0164e8 thru 0x01d290 (0x6da9 bytes) 000408 1790921897.135833 read 0x00d091 thru 0x00d213 (0x183 bytes) 000409 1790921897.142762 trunc from 0x02be54 to 0x024739 (0x771a bytes) 000410 1790921897.157647 punch from 0x00fb8a to 0x01c38b (0xc802 bytes) 000411 1790921897.162253 read 0x0054ad thru 0x00a957 (0x54ab bytes) 000412 1790921897.162817 trunc from 0x00466e to 0x00c3cf (0x7d62 bytes) 000413 1790921897.169797 fallocate 0x0296b5 thru 0x037b3a (0xe486 bytes) 000414 1790921897.175648 read 0x01656e thru 0x01a0ca (0x3b5d bytes) 000415 1790921897.192386 trunc from 0x037b3b to 0x023ccd (0x13e6d bytes) 000416 1790921897.206580 fallocate 0x0003db thru 0x005d68 (0x598e bytes) 000417 1790921897.210535 write 0x02c0b4 thru 0x0334c0 (0x740d bytes) 000418 1790921897.211565 punch from 0x003c11 to 0x01182d (0xdc1d bytes) 000419 1790921897.216748 punch from 0x0182f0 to 0x018ebf (0xbd0 bytes) 000420 1790921897.220247 punch from 0x0035c6 to 0x00a950 (0x738b bytes) 000421 1790921897.223585 write 0x02316c thru 0x026500 (0x3395 bytes) 000422 1790921897.227394 write 0x00e401 thru 0x00fb4e (0x174e bytes) 000423 1790921897.235261 mapread 0x004680 thru 0x013d1f (0xf6a0 bytes) 000424 1790921897.285101 mapread 0x017b5c thru 0x01ccbb (0x5160 bytes) 000425 1790921897.290014 write 0x035239 thru 0x036832 (0x15fa bytes) 000426 1790921897.291427 mapread 0x032c5a thru 0x036832 (0x3bd9 bytes) 000427 1790921897.296312 trunc from 0x013d1f to 0x01c355 (0x8637 bytes) 000429 1790921897.303850 trunc from 0x025ea4 to 0x026052 (0x1af bytes) 000431 1790921897.328158 write 0x02ee9f thru 0x035a8a (0x6bec bytes) 000432 1790921897.330647 trunc from 0x03597e to 0x036832 (0xeb5 bytes) 000433 1790921897.368240 mapread 0x00789d thru 0x012686 (0xadea bytes) 000434 1790921897.370033 write 0x016e0f thru 0x01dd51 (0x6f43 bytes) 000435 1790921897.381542 mapread 0x01682e thru 0x022250 (0xba23 bytes) 000437 1790921897.384777 mapread 0x00b1f7 thru 0x00c069 (0xe73 bytes) 000438 1790921897.385937 trunc from 0x0118a1 to 0x020355 (0xeab5 bytes) 000439 1790921897.405151 write 0x00a36d thru 0x0163c9 (0xc05d bytes) 000440 1790921897.415009 punch from 0x020e9d to 0x02299c (0x1b00 bytes) 000441 1790921897.421116 write 0x00610e thru 0x00697f (0x872 bytes) 000443 1790921897.424392 trunc from 0x036833 to 0x013f31 (0x22901 bytes) 000444 1790921897.442625 write 0x000962 thru 0x003e89 (0x3528 bytes) 000446 1790921897.453399 fallocate 0x03496d thru 0x03e33b (0x99cf bytes) 000447 1790921897.459156 write 0x0361bb thru 0x037fff (0x1e45 bytes) 000448 1790921897.466795 punch from 0x01014c to 0x0149fc (0x48b1 bytes) 000449 1790921897.488737 trunc from 0x03e33c to 0x01bb20 (0x2281b bytes) 000450 1790921897.520724 trunc from 0x017d40 to 0x01bb20 (0x3de1 bytes) 000451 1790921897.526140 mapread 0x017c3a thru 0x01bb20 (0x3ee7 bytes) 000453 1790921897.546416 trunc from 0x00a9f2 to 0x00e9fa (0x4009 bytes) 000454 1790921897.550763 mapread 0x01715f thru 0x01bb20 (0x49c2 bytes) 000455 1790921897.552116 fallocate 0x00e35f thru 0x01577e (0x7420 bytes) 000457 1790921897.557611 fallocate 0x010cf5 thru 0x013d3b (0x3047 bytes) 000458 1790921897.561514 fallocate 0x01d934 thru 0x029dec (0xc4b9 bytes) 000460 1790921897.568607 read 0x023e42 thru 0x029dec (0x5fab bytes) 000461 1790921897.578176 mapread 0x019df0 thru 0x01c170 (0x2381 bytes) 000463 1790921897.579274 read 0x024e73 thru 0x029dec (0x4f7a bytes) 000464 1790921897.579706 read 0x007431 thru 0x0108aa (0x947a bytes) 000466 1790921897.582488 write 0x02c988 thru 0x02dc06 (0x127f bytes) 000467 1790921897.584647 mapwrite 0x0138c9 thru 0x022ebe (0xf5f6 bytes) 000468 1790921897.613609 write 0x00376c thru 0x0134a2 (0xfd37 bytes) 000469 1790921897.616598 fallocate 0x02bb85 thru 0x0381e0 (0xc65c bytes) 000471 1790921897.622581 read 0x0108c9 thru 0x01d32c (0xca64 bytes) 000472 1790921897.623141 write 0x025341 thru 0x033272 (0xdf32 bytes) 000473 1790921897.629112 trunc from 0x0381e1 to 0x00bf58 (0x2c288 bytes) 000474 1790921897.645136 mapwrite 0x02f728 thru 0x03d6cd (0xdfa6 bytes) 000475 1790921897.698697 fallocate 0x03f1ab thru 0x03ffff (0xe55 bytes) 000476 1790921897.706302 trunc from 0x0169fa to 0x01cee5 (0x64ec bytes) 000477 1790921897.723140 fallocate 0x0339a2 thru 0x03d42a (0x9a89 bytes) 000480 1790921897.743064 trunc from 0x040000 to 0x018244 (0x27dbb bytes) 000481 1790921897.773395 mapwrite 0x005e82 thru 0x00bb71 (0x5cf0 bytes) 000482 1790921897.798239 read 0x00beca thru 0x012919 (0x6a50 bytes) 000483 1790921897.813783 fallocate 0x035885 thru 0x0384ca (0x2c46 bytes) 000484 1790921897.820462 fallocate 0x0227c1 thru 0x03100f (0xe84f bytes) 000486 1790921897.833219 punch from 0x01b748 to 0x029737 (0xdff0 bytes) 000488 1790921897.848126 read 0x02b594 thru 0x02b7f3 (0x260 bytes) 000489 1790921897.863839 fallocate 0x01a0de thru 0x021a7f (0x79a2 bytes) 000490 1790921897.870516 trunc from 0x02f399 to 0x037a56 (0x86be bytes) 000492 1790921897.878222 punch from 0x023907 to 0x033028 (0xf722 bytes) 000493 1790921897.885245 read 0x015838 thru 0x01a5a5 (0x4d6e bytes) 000494 1790921897.886205 trunc from 0x01f79b to 0x02a742 (0xafa8 bytes) 000495 1790921897.897881 trunc from 0x00cd10 to 0x0157c7 (0x8ab8 bytes) 000496 1790921897.915177 write 0x025f16 thru 0x030c69 (0xad54 bytes) 000497 1790921897.933771 fallocate 0x015a0d thru 0x01d597 (0x7b8b bytes) 000498 1790921897.940230 mapread 0x0326b3 thru 0x0374c2 (0x4e10 bytes) 000499 1790921897.946656 trunc from 0x0384cb to 0x0177ba (0x20d10 bytes) 000500 1790921897.967663 fallocate 0x00d20d thru 0x01b081 (0xde75 bytes) 000502 1790921897.975194 punch from 0x01516c to 0x0177ba (0x264f bytes) 000503 1790921897.980672 trunc from 0x002bee to 0x00b609 (0x8a1c bytes) 000504 1790921897.986685 fallocate 0x039c6d thru 0x03ffff (0x6393 bytes) 000505 1790921897.991706 fallocate 0x0368a9 thru 0x03ffff (0x9757 bytes) 000506 1790921897.997756 punch from 0x00e487 to 0x015af5 (0x766f bytes) 000507 1790921898.003113 mapwrite 0x012e72 thru 0x01b0f7 (0x8286 bytes) 000508 1790921898.090106 mapread 0x03a1bb thru 0x03f8b1 (0x56f7 bytes) 000509 1790921898.091756 trunc from 0x040000 to 0x004dfb (0x3b204 bytes) 000510 1790921898.128536 punch from 0x000845 to 0x004dfb (0x45b7 bytes) 000511 1790921898.143550 trunc from 0x004dfc to 0x02a6de (0x258e3 bytes) 000512 1790921898.161766 mapread 0x023498 thru 0x02a6de (0x7247 bytes) 000513 1790921898.208451 mapread 0x00e42a thru 0x015dd1 (0x79a8 bytes) 000517 1790921898.255296 fallocate 0x001d15 thru 0x00bac7 (0x9db3 bytes) 000518 1790921898.262246 trunc from 0x02a6df to 0x014446 (0x16298 bytes) 000519 1790921898.290361 fallocate 0x00eca4 thru 0x016e2f (0x818c bytes) 000520 1790921898.297621 punch from 0x00c87f to 0x00fb0e (0x3290 bytes) 000521 1790921898.302780 mapread 0x00132c thru 0x006ebe (0x5b93 bytes) 000522 1790921898.380445 write 0x02d42e thru 0x0314f9 (0x40cc bytes) 000523 1790921898.381843 mapread 0x007e3f thru 0x011197 (0x9359 bytes) 000524 1790921898.435812 trunc from 0x0314fa to 0x00f1cd (0x2232c bytes) 000526 1790921898.458606 mapwrite 0x03fcb4 thru 0x03ffff (0x34c bytes) 000528 1790921898.514219 fallocate 0x023870 thru 0x0308c4 (0xd055 bytes) 000529 1790921898.530064 fallocate 0x03c114 thru 0x03ffff (0x3eec bytes) 000530 1790921898.543345 trunc from 0x03c710 to 0x03ffff (0x38f0 bytes) 000532 1790921898.550225 read 0x03b993 thru 0x03ffff (0x466d bytes) 000533 1790921898.572453 fallocate 0x00aa10 thru 0x011089 (0x667a bytes) 000534 1790921898.580239 mapread 0x021b28 thru 0x024c87 (0x3160 bytes) 000535 1790921898.582367 trunc from 0x040000 to 0x023b59 (0x1c4a6 bytes) 000536 1790921898.609848 mapwrite 0x01dce5 thru 0x01f011 (0x132d bytes) 000537 1790921898.631151 fallocate 0x01fdb1 thru 0x026c04 (0x6e54 bytes) 000538 1790921898.641560 trunc from 0x026c05 to 0x0051b5 (0x21a4f bytes) 000539 1790921898.670636 trunc from 0x0051b6 to 0x0376ef (0x3253a bytes) 000540 1790921898.697191 punch from 0x01cb0c to 0x0263d0 (0x98c5 bytes) 000541 1790921898.702517 trunc from 0x0376f0 to 0x03000c (0x76e3 bytes) 000543 1790921898.724299 mapwrite 0x023af6 thru 0x03355a (0xfa65 bytes) 000544 1790921898.901343 mapread 0x019521 thru 0x019c34 (0x714 bytes) 000545 1790921898.911551 write 0x019db4 thru 0x0209d8 (0x6c25 bytes) 000546 1790921898.923215 read 0x012c02 thru 0x01d21d (0xa61c bytes) 000547 1790921898.966827 mapwrite 0x0064da thru 0x00f7ab (0x92d2 bytes) 000549 1790921898.995679 read 0x02b42e thru 0x03355a (0x812d bytes) 000550 1790921899.005528 trunc from 0x03355b to 0x033c5d (0x703 bytes) 000552 1790921899.055905 write 0x02d10f thru 0x037bd1 (0xaac3 bytes) 000553 1790921899.062397 trunc from 0x03472b to 0x037bd1 (0x34a7 bytes) 000554 1790921899.109678 trunc from 0x037bd2 to 0x039b3a (0x1f69 bytes) 000555 1790921899.146776 punch from 0x00a03a to 0x00b404 (0x13cb bytes) 000557 1790921899.161692 trunc from 0x039b3b to 0x0389d5 (0x1165 bytes) 000558 1790921899.181743 mapread 0x0268e3 thru 0x02be44 (0x5562 bytes) 000560 1790921899.184436 write 0x0029e8 thru 0x0030c5 (0x6de bytes) 000561 1790921899.192509 MIRROR_EXTEND to 2 mirrors 000563 1790921900.576708 fallocate 0x01139d thru 0x0202b6 (0xef1a bytes) 000564 1790921900.638242 mapread 0x02a56d thru 0x035462 (0xaef6 bytes) 000565 1790921900.700409 read 0x0367df thru 0x0389d5 (0x21f7 bytes) 000566 1790921900.702049 write 0x014906 thru 0x021d38 (0xd433 bytes) 000567 1790921900.736347 read 0x02719b thru 0x02cb7d (0x59e3 bytes) 000568 1790921900.737767 read 0x030ad6 thru 0x0389d5 (0x7f00 bytes) 000570 1790921900.741363 trunc from 0x0389d6 to 0x01721b (0x217ba bytes) 000571 1790921900.794193 write 0x0196c4 thru 0x01da2d (0x436a bytes) 000572 1790921900.795825 mapwrite 0x0356c2 thru 0x036fbf (0x18fe bytes) 000573 1790921900.862523 mapread 0x02e762 thru 0x036fbf (0x885e bytes) 000574 1790921900.864525 trunc from 0x017156 to 0x02020c (0x90b7 bytes) 000575 1790921900.874293 MIRROR_RESYNC 2 mirrors 000576 1790921901.792307 fallocate 0x03d288 thru 0x03ffff (0x2d78 bytes) 000577 1790921901.844029 fallocate 0x014b54 thru 0x01e430 (0x98dd bytes) 000578 1790921901.859972 write 0x00cc64 thru 0x01a687 (0xda24 bytes) 000579 1790921901.874778 trunc from 0x015762 to 0x025525 (0xfdc4 bytes) 000580 1790921901.902411 trunc from 0x028ea6 to 0x0377a0 (0xe8fb bytes) 000581 1790921901.927762 mapwrite 0x038bf3 thru 0x03f59a (0x69a8 bytes) 000582 1790921902.043131 write 0x02d177 thru 0x03a5ef (0xd479 bytes) 000583 1790921902.051846 trunc from 0x03501c to 0x03ee0f (0x9df4 bytes) 000584 1790921902.072692 fallocate 0x00fb08 thru 0x01d5cd (0xdac6 bytes) 000586 1790921902.109366 MIRROR_RESYNC 2 mirrors 000587 1790921903.366826 punch from 0x038f07 to 0x03ffff (0x70f9 bytes) 000589 1790921903.420714 trunc from 0x040000 to 0x02e9a9 (0x11656 bytes) 000590 1790921903.471929 mapread 0x01c01a thru 0x022058 (0x603f bytes) 000591 1790921903.516591 mapwrite 0x036217 thru 0x03ffff (0x9de9 bytes) 000592 1790921903.606763 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x6a01a102 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbb20cc60 Mirror 3: 0xbb20cc60 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000593 1790921904.864769 mapread 0x00be80 thru 0x015d0f (0x9e90 bytes) 000594 1790921904.919184 mapwrite 0x002297 thru 0x0103ec (0xe156 bytes) 000597 1790921905.095613 write 0x028f1e thru 0x02eea0 (0x5f83 bytes) 000598 1790921905.126383 mapread 0x028721 thru 0x02b1e4 (0x2ac4 bytes) 000599 1790921905.127539 trunc from 0x040000 to 0x01546b (0x2ab94 bytes) 000600 1790921905.192489 fallocate 0x025c1a thru 0x02f0ce (0x94b5 bytes) 000601 1790921905.208936 punch from 0x027c86 to 0x02d386 (0x5701 bytes) 000602 1790921905.259956 MIRROR_RESYNC 3 mirrors 000603 1790921906.216619 MIRROR_EXTEND to 4 mirrors 000604 1790921907.369148 fallocate 0x0165ac thru 0x01a735 (0x418a bytes) 000605 1790921907.436086 punch from 0x011ce4 to 0x01ff77 (0xe294 bytes) 000606 1790921907.444644 punch from 0x015393 to 0x0214b6 (0xc124 bytes) 000607 1790921907.450710 fallocate 0x015136 thru 0x01593c (0x807 bytes) 000608 1790921907.456569 trunc from 0x02f0cf to 0x001fec (0x2d0e2 bytes) 000609 1790921907.487397 punch from 0x0017f0 to 0x001fec (0x7fd bytes) 000610 1790921907.496500 write 0x00b0f5 thru 0x013368 (0x8274 bytes) 000611 1790921907.499971 mapwrite 0x00bfd1 thru 0x0184c3 (0xc4f3 bytes) 000613 1790921907.563274 trunc from 0x00080d to 0x0033e8 (0x2bdc bytes) 000614 1790921907.580633 MIRROR_RESYNC 4 mirrors 000615 1790921908.899692 trunc from 0x0184c4 to 0x03c6c7 (0x24204 bytes) 000617 1790921908.950644 mapread 0x006a34 thru 0x014777 (0xdd44 bytes) 000618 1790921908.980407 mapread 0x019b32 thru 0x02619e (0xc66d bytes) 000619 1790921908.997980 punch from 0x02b892 to 0x03b815 (0xff84 bytes) 000620 1790921909.002493 fallocate 0x0171a2 thru 0x01ffeb (0x8e4a bytes) 000621 1790921909.008639 fallocate 0x026296 thru 0x034000 (0xdd6b bytes) 000622 1790921909.019658 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c6c8): Mirror 3: 0x27da7319 (stale) Mirror 4: 0x27da7319 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c6c8): Mirror 3: 0x27da7319 (stale) Mirror 4: 0x27da7319 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c6c8: skipped 000623 1790921909.640319 trunc from 0x018acd to 0x01daae (0x4fe2 bytes) 000624 1790921909.656664 fallocate 0x0072ad thru 0x00f38a (0x80de bytes) 000625 1790921909.664713 trunc from 0x034bb4 to 0x03c6c7 (0x7b14 bytes) 000626 1790921909.679746 trunc from 0x03c6c8 to 0x0334c8 (0x91ff bytes) 000627 1790921909.720481 mapwrite 0x03d0ed thru 0x03d6fe (0x612 bytes) 000628 1790921909.794317 fallocate 0x0175de thru 0x01a74f (0x3172 bytes) 000629 1790921909.807128 MIRROR_RESYNC 3 mirrors 000630 1790921911.470651 trunc from 0x03d6ff to 0x01cdff (0x208ff bytes) 000631 1790921911.584514 mapwrite 0x02ba11 thru 0x033a08 (0x7ff8 bytes) 000633 1790921911.722255 read 0x022c31 thru 0x029b8c (0x6f5c bytes) 000634 1790921911.750180 mapwrite 0x033ac5 thru 0x03bb30 (0x806c bytes) 000635 1790921911.887996 read 0x0382f1 thru 0x039903 (0x1613 bytes) 000636 1790921911.888770 mapread 0x00cb1b thru 0x014af0 (0x7fd6 bytes) 000637 1790921911.946973 mapread 0x022c58 thru 0x0325a0 (0xf949 bytes) 000638 1790921911.948192 write 0x0220dc thru 0x02fe08 (0xdd2d bytes) 000640 1790921911.968776 read 0x03a8f6 thru 0x03bb30 (0x123b bytes) 000641 1790921911.970455 punch from 0x032e7c to 0x039652 (0x67d7 bytes) 000643 1790921912.006317 read 0x0380ee thru 0x03bb30 (0x3a43 bytes) 000644 1790921912.024386 fallocate 0x0175fe thru 0x025abd (0xe4c0 bytes) 000645 1790921912.044372 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3bb31): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb31): Mirror 3: 0xb89bcc91 (stale) Mirror 4: 0xb89bcc91 (stale) CRC-32 checksum value for chunk [0, 0x3bb31): Mirror 1: 0x57a5ea8e Mirror 5: 0x57a5ea8e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb31: skipped 000646 1790921913.466417 mapread 0x023b4f thru 0x029874 (0x5d26 bytes) 000647 1790921913.549676 write 0x00e12c thru 0x00f5e4 (0x14b9 bytes) 000648 1790921913.630006 write 0x019704 thru 0x02864a (0xef47 bytes) 000649 1790921913.641279 mapwrite 0x005658 thru 0x01535d (0xfd06 bytes) 000650 1790921913.694413 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bb31): Mirror 4: 0xb89bcc91 (stale) Mirror 5: 0x57a5ea8e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb31): Mirror 4: 0xb89bcc91 (stale) Mirror 5: 0x57a5ea8e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb31: skipped 000651 1790921914.190236 trunc from 0x03bb31 to 0x0118bd (0x2a273 bytes) 000652 1790921914.226368 mapread 0x00c688 thru 0x0118bd (0x5236 bytes) 000653 1790921914.254695 punch from 0x00d5bf to 0x00e4c4 (0xf06 bytes) 000654 1790921914.260672 mapwrite 0x003ea3 thru 0x01101d (0xd17b bytes) 000655 1790921914.374578 fallocate 0x01fb1d thru 0x024f88 (0x546c bytes) 000656 1790921914.389342 write 0x0395a0 thru 0x03ffff (0x6a60 bytes) 000657 1790921914.399281 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8cb12c4c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8cb12c4c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000658 1790921914.765057 trunc from 0x00442f to 0x00b015 (0x6be7 bytes) 000659 1790921914.770066 punch from 0x036853 to 0x03ffff (0x97ad bytes) 000660 1790921914.776374 mapread 0x00232f thru 0x008c55 (0x6927 bytes) 000661 1790921914.801366 mapwrite 0x01cfc1 thru 0x02b9d4 (0xea14 bytes) 000662 1790921914.849346 punch from 0x01c02c to 0x02169e (0x5673 bytes) 000664 1790921914.857531 punch from 0x007f0e to 0x013af5 (0xbbe8 bytes) 000666 1790921914.864228 trunc from 0x008bf8 to 0x01408a (0xb493 bytes) 000667 1790921914.870969 write 0x013d9e thru 0x016ac7 (0x2d2a bytes) 000669 1790921914.886389 mapread 0x00d337 thru 0x0103f3 (0x30bd bytes) 000671 1790921914.908229 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8cb12c4c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x844f70b7 Mirror 5: 0x844f70b7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000672 1790921916.262684 write 0x01d60a thru 0x028628 (0xb01f bytes) 000673 1790921916.330923 trunc from 0x040000 to 0x0220d6 (0x1df29 bytes) 000674 1790921916.347706 mapread 0x00d5c9 thru 0x012835 (0x526d bytes) 000675 1790921916.366345 trunc from 0x0220d7 to 0x02d1f8 (0xb122 bytes) 000676 1790921916.386061 read 0x00cf87 thru 0x010274 (0x32ee bytes) 000677 1790921916.386755 mapread 0x013722 thru 0x01ba3e (0x831d bytes) 000678 1790921916.395939 trunc from 0x016c99 to 0x02028b (0x95f3 bytes) 000679 1790921916.416476 fallocate 0x0102e1 thru 0x01fac8 (0xf7e8 bytes) 000680 1790921916.424238 trunc from 0x02d1f9 to 0x0319f2 (0x47fa bytes) 000681 1790921916.441747 write 0x01a6de thru 0x02a240 (0xfb63 bytes) 000682 1790921916.453890 mapread 0x0088f8 thru 0x00b54f (0x2c58 bytes) 000683 1790921916.455422 trunc from 0x0319f3 to 0x01e171 (0x13881 bytes) 000684 1790921916.481713 MIRROR_RESYNC 3 mirrors 000685 1790921917.564254 punch from 0x00ea68 to 0x019694 (0xac2d bytes) 000686 1790921917.610641 trunc from 0x00e8ae to 0x00fbf8 (0x134b bytes) 000688 1790921917.618202 mapwrite 0x03738d thru 0x03ffff (0x8c73 bytes) 000689 1790921917.684317 trunc from 0x020cbe to 0x029c45 (0x8f88 bytes) 000690 1790921917.690961 trunc from 0x00cf68 to 0x01934d (0xc3e6 bytes) 000691 1790921917.696048 read 0x020bf6 thru 0x021ff6 (0x1401 bytes) 000692 1790921917.702556 write 0x012846 thru 0x01b69f (0x8e5a bytes) 000693 1790921917.716846 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x8bd49343 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x8bd49343 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000694 1790921918.043263 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x8bd49343 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4963e84e Mirror 6: 0x4963e84e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000697 1790921919.488275 fallocate 0x0046bd thru 0x00f564 (0xaea8 bytes) 000698 1790921919.573828 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x8bd49343 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x8bd49343 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000699 1790921919.909428 read 0x037b76 thru 0x03c5a7 (0x4a32 bytes) 000701 1790921919.921077 fallocate 0x003ccc thru 0x0047d3 (0xb08 bytes) 000702 1790921919.926736 fallocate 0x01e102 thru 0x027b1e (0x9a1d bytes) 000703 1790921919.941775 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x8bd49343 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4963e84e Mirror 6: 0x4963e84e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000704 1790921921.106417 MIRROR_RESYNC 3 mirrors 000705 1790921922.346227 fallocate 0x01e34e thru 0x026cb7 (0x896a bytes) 000706 1790921922.382528 trunc from 0x003649 to 0x012c0f (0xf5c7 bytes) 000707 1790921922.388665 write 0x02e4ac thru 0x03a842 (0xc397 bytes) 000708 1790921922.398176 trunc from 0x040000 to 0x02c97e (0x13681 bytes) 000709 1790921922.419122 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c97f): Mirror 6: 0x91969964 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c97f): Mirror 6: 0x91969964 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c97f: skipped 000710 1790921922.780804 punch from 0x013964 to 0x01ecc7 (0xb364 bytes) 000711 1790921922.799766 MIRROR_RESYNC 2 mirrors 000713 1790921923.997325 mapwrite 0x011a29 thru 0x01ef57 (0xd52f bytes) 000714 1790921924.233642 MIRROR_RESYNC 2 mirrors 000715 1790921925.135277 punch from 0x0015c3 to 0x00693a (0x5378 bytes) 000716 1790921925.189249 trunc from 0x02c97f to 0x031a40 (0x50c2 bytes) 000717 1790921925.230968 mapwrite 0x03965d thru 0x03ffff (0x69a3 bytes) 000718 1790921925.420972 fallocate 0x019e7d thru 0x02806c (0xe1f0 bytes) 000719 1790921925.435222 trunc from 0x040000 to 0x00817e (0x37e81 bytes) 000721 1790921925.461368 write 0x03c8f2 thru 0x03d49f (0xbae bytes) 000722 1790921925.464227 trunc from 0x00229b to 0x006806 (0x456c bytes) 000725 1790921925.472370 trunc from 0x03d4a0 to 0x038058 (0x5447 bytes) 000726 1790921925.502793 punch from 0x035495 to 0x038058 (0x2bc4 bytes) 000727 1790921925.507154 trunc from 0x00e7a2 to 0x00fc75 (0x14d4 bytes) 000728 1790921925.514082 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa6:0x0]: Invalid argument (22) 000730 1790921925.651636 mapread 0x02c6d4 thru 0x035178 (0x8aa5 bytes) 000731 1790921925.685939 trunc from 0x004be6 to 0x00b41a (0x6835 bytes) 000732 1790921925.693481 mapread 0x009037 thru 0x0148ec (0xb8b6 bytes) 000733 1790921925.735595 punch from 0x009aff to 0x014a80 (0xaf82 bytes) 000734 1790921925.740598 mapwrite 0x036cbc thru 0x03ffff (0x9344 bytes) 000735 1790921925.806080 trunc from 0x040000 to 0x019ad0 (0x2652f bytes) 000736 1790921925.827050 read 0x00470f thru 0x007574 (0x2e66 bytes) 000737 1790921925.827696 punch from 0x0138a4 to 0x019ad0 (0x622d bytes) 000739 1790921925.833914 trunc from 0x007121 to 0x013a55 (0xc935 bytes) 000740 1790921925.839236 mapwrite 0x017b1c thru 0x018d56 (0x123b bytes) 000741 1790921925.858963 trunc from 0x019ad1 to 0x02c921 (0x12e51 bytes) 000742 1790921925.876367 read 0x019c9a thru 0x01a473 (0x7da bytes) 000746 1790921925.888563 fallocate 0x022f16 thru 0x024c80 (0x1d6b bytes) 000747 1790921925.895534 mapread 0x02ba1e thru 0x02c921 (0xf04 bytes) 000749 1790921925.899431 fallocate 0x02eb8f thru 0x034da6 (0x6218 bytes) 000751 1790921925.907132 punch from 0x018d97 to 0x01de8e (0x50f8 bytes) 000752 1790921925.915083 mapwrite 0x01d5d8 thru 0x0200cc (0x2af5 bytes) 000753 1790921925.951164 fallocate 0x039782 thru 0x03d89f (0x411e bytes) 000754 1790921925.968231 MIRROR_EXTEND to 2 mirrors 000755 1790921926.816031 punch from 0x02a6dd to 0x02c921 (0x2245 bytes) 000756 1790921926.883420 read 0x01c4b5 thru 0x028353 (0xbe9f bytes) 000757 1790921926.922424 mapwrite 0x01d369 thru 0x01ed82 (0x1a1a bytes) 000758 1790921926.955030 mapread 0x007799 thru 0x00b722 (0x3f8a bytes) 000760 1790921926.984226 trunc from 0x01e22b to 0x01fe97 (0x1c6d bytes) 000761 1790921926.996480 trunc from 0x02c922 to 0x01196b (0x1afb6 bytes) 000762 1790921927.068587 mapread 0x0117eb thru 0x01196b (0x181 bytes) 000763 1790921927.069447 punch from 0x00426c to 0x00ab82 (0x6917 bytes) 000764 1790921927.086608 punch from 0x004544 to 0x004fbc (0xa79 bytes) 000765 1790921927.099920 trunc from 0x00be8b to 0x01196b (0x5ae1 bytes) 000766 1790921927.118170 write 0x028989 thru 0x032aed (0xa165 bytes) 000767 1790921927.128683 trunc from 0x032aee to 0x03caff (0xa012 bytes) 000768 1790921927.178273 read 0x0265df thru 0x0342d6 (0xdcf8 bytes) 000770 1790921927.207085 trunc from 0x004c88 to 0x012097 (0xd410 bytes) 000771 1790921927.221910 trunc from 0x03cb00 to 0x017da9 (0x24d56 bytes) 000774 1790921927.273885 mapwrite 0x0394b2 thru 0x03fdbb (0x690a bytes) 000776 1790921927.433466 trunc from 0x027d38 to 0x03326c (0xb535 bytes) 000777 1790921927.452464 write 0x039ec8 thru 0x03be23 (0x1f5c bytes) 000778 1790921927.453539 punch from 0x01f456 to 0x028979 (0x9524 bytes) 000779 1790921927.462974 fallocate 0x017d1f thru 0x01dd14 (0x5ff6 bytes) 000780 1790921927.474958 punch from 0x0072c9 to 0x00da78 (0x67b0 bytes) 000781 1790921927.486388 trunc from 0x034264 to 0x0362a7 (0x2044 bytes) 000782 1790921927.510294 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3fdbc): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fdbc): Mirror 2: 0x785b8fcd (stale) CRC-32 checksum value for chunk [0, 0x3fdbc): Mirror 1: 0xd98a833 Mirror 3: 0xd98a833 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fdbc: skipped 000783 1790921928.779263 punch from 0x02364b to 0x028db2 (0x5768 bytes) 000784 1790921928.827924 trunc from 0x03fdbc to 0x00ef88 (0x30e33 bytes) 000785 1790921928.851942 trunc from 0x004f99 to 0x00ef88 (0x9ff0 bytes) 000786 1790921928.865116 trunc from 0x005ebc to 0x00ef88 (0x90cd bytes) 000787 1790921928.874265 trunc from 0x004a93 to 0x00ac91 (0x61ff bytes) 000788 1790921928.881553 fallocate 0x01a8f4 thru 0x023c03 (0x9310 bytes) 000789 1790921928.886895 mapwrite 0x01ed95 thru 0x02d96d (0xebd9 bytes) 000790 1790921928.946827 fallocate 0x033f23 thru 0x03f191 (0xb26f bytes) 000791 1790921928.952502 fallocate 0x03384a thru 0x03f7a6 (0xbf5d bytes) 000792 1790921928.956378 read 0x005e1a thru 0x00ab10 (0x4cf7 bytes) 000793 1790921928.965083 mapwrite 0x00e7c6 thru 0x013d4e (0x5589 bytes) 000794 1790921929.003921 trunc from 0x03f7a7 to 0x01a924 (0x24e82 bytes) 000795 1790921929.058567 mapwrite 0x0071ad thru 0x01477d (0xd5d1 bytes) 000796 1790921929.107460 trunc from 0x0000a1 to 0x0077e3 (0x7743 bytes) 000797 1790921929.115742 trunc from 0x00d307 to 0x00f417 (0x2111 bytes) 000798 1790921929.127561 write 0x03529b thru 0x03db36 (0x889c bytes) 000800 1790921929.137535 trunc from 0x03db37 to 0x026e01 (0x16d35 bytes) 000801 1790921929.173196 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26e02): Mirror 2: 0xc1cb34fc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26e02): Mirror 2: 0xc1cb34fc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26e02: skipped 000802 1790921929.676912 punch from 0x0215c5 to 0x0249f0 (0x342c bytes) 000803 1790921929.693998 trunc from 0x01013c to 0x01fd9f (0xfc64 bytes) 000804 1790921929.707492 mapread 0x015095 thru 0x017e56 (0x2dc2 bytes) 000805 1790921929.741214 read 0x00463d thru 0x004ec4 (0x888 bytes) 000806 1790921929.749343 mapwrite 0x01f818 thru 0x02964f (0x9e38 bytes) 000807 1790921929.844039 trunc from 0x00caf0 to 0x01af10 (0xe421 bytes) 000808 1790921929.853305 mapread 0x020f97 thru 0x02964f (0x86b9 bytes) 000809 1790921929.854367 mapread 0x01f28e thru 0x022389 (0x30fc bytes) 000810 1790921929.855048 fallocate 0x00aeff thru 0x0124fb (0x75fd bytes) 000811 1790921929.863286 write 0x0206d1 thru 0x022b3c (0x246c bytes) 000812 1790921929.864499 mapwrite 0x002f85 thru 0x0062c4 (0x3340 bytes) 000813 1790921929.896689 fallocate 0x01dae8 thru 0x02c22f (0xe748 bytes) 000814 1790921929.903316 mapread 0x01e28a thru 0x0224bc (0x4233 bytes) 000815 1790921929.910576 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa6:0x0]: Invalid argument (22) 000816 1790921930.099902 punch from 0x016c9f to 0x01890b (0x1c6d bytes) 000817 1790921930.117100 mapread 0x011573 thru 0x015e72 (0x4900 bytes) 000818 1790921930.153432 mapwrite 0x025e59 thru 0x034e6f (0xf017 bytes) 000819 1790921930.259044 fallocate 0x02924c thru 0x02bb71 (0x2926 bytes) 000821 1790921930.266575 mapwrite 0x008282 thru 0x00b06b (0x2dea bytes) 000822 1790921930.314147 mapread 0x03418c thru 0x034c77 (0xaec bytes) 000823 1790921930.317541 write 0x02ee80 thru 0x033543 (0x46c4 bytes) 000824 1790921930.337488 MIRROR_EXTEND to 2 mirrors 000825 1790921931.350551 trunc from 0x034e70 to 0x027310 (0xdb5f bytes) 000826 1790921931.408397 mapread 0x00f7ed thru 0x01e72f (0xef43 bytes) 000827 1790921931.453304 write 0x0330b4 thru 0x038c5d (0x5baa bytes) 000828 1790921931.454484 punch from 0x0232c3 to 0x030c7b (0xd9b9 bytes) 000829 1790921931.464628 trunc from 0x0203e3 to 0x02057d (0x19b bytes) 000830 1790921931.472227 fallocate 0x03f2d4 thru 0x03ffff (0xd2c bytes) 000832 1790921931.478570 mapread 0x0043d5 thru 0x00fd11 (0xb93d bytes) 000833 1790921931.490716 write 0x033112 thru 0x03ffff (0xceee bytes) 000834 1790921931.494208 read 0x0120f0 thru 0x02062f (0xe540 bytes) 000835 1790921931.501382 write 0x025f2a thru 0x026e95 (0xf6c bytes) 000836 1790921931.520096 trunc from 0x011e8d to 0x01adfe (0x8f72 bytes) 000837 1790921931.530715 mapwrite 0x03ea3a thru 0x03ffff (0x15c6 bytes) 000839 1790921931.572531 write 0x01a641 thru 0x023184 (0x8b44 bytes) 000841 1790921931.590117 punch from 0x019de1 to 0x021b7a (0x7d9a bytes) 000843 1790921931.612025 fallocate 0x0024db thru 0x006ab2 (0x45d8 bytes) 000844 1790921931.616788 trunc from 0x040000 to 0x03b85a (0x47a5 bytes) 000845 1790921931.656505 mapread 0x0029ae thru 0x00c997 (0x9fea bytes) 000846 1790921931.658249 trunc from 0x0092f9 to 0x00e42b (0x5133 bytes) 000847 1790921931.674384 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa6:0x0]: Invalid argument (22) 000848 1790921931.806742 trunc from 0x03b85b to 0x0385df (0x327b bytes) 000849 1790921931.824344 write 0x003e47 thru 0x011baf (0xdd69 bytes) 000850 1790921931.832220 punch from 0x0025c5 to 0x009739 (0x7175 bytes) 000851 1790921931.846431 mapwrite 0x01ccac thru 0x020cbb (0x4010 bytes) 000852 1790921931.909753 trunc from 0x0385e0 to 0x0083af (0x30230 bytes) 000853 1790921931.949723 read 0x00455e thru 0x0083af (0x3e52 bytes) 000854 1790921931.962056 fallocate 0x029d71 thru 0x03321f (0x94af bytes) 000855 1790921931.973357 fallocate 0x03e1c0 thru 0x03ffff (0x1e40 bytes) 000856 1790921931.982316 mapread 0x01916a thru 0x027b72 (0xea09 bytes) 000857 1790921932.008762 punch from 0x02c597 to 0x035a6f (0x94d9 bytes) 000858 1790921932.015283 read 0x00ac2c thru 0x016df7 (0xc1cc bytes) 000859 1790921932.027076 fallocate 0x0174d8 thru 0x01c2af (0x4dd8 bytes) 000862 1790921932.033098 mapwrite 0x03301f thru 0x03ffff (0xcfe1 bytes) 000863 1790921932.052606 mapwrite 0x00d724 thru 0x015340 (0x7c1d bytes) 000865 1790921932.095539 write 0x006b89 thru 0x0167c4 (0xfc3c bytes) 000866 1790921932.107254 read 0x0249c2 thru 0x030c62 (0xc2a1 bytes) 000867 1790921932.108614 read 0x03f2f0 thru 0x03ffff (0xd10 bytes) 000869 1790921932.110076 trunc from 0x037d91 to 0x03ffff (0x826f bytes) 000870 1790921932.118377 fallocate 0x00b91b thru 0x0191dd (0xd8c3 bytes) 000872 1790921932.125480 trunc from 0x0060a7 to 0x01475f (0xe6b9 bytes) 000873 1790921932.144485 trunc from 0x0157c0 to 0x01faaa (0xa2eb bytes) 000874 1790921932.153766 trunc from 0x02b69a to 0x0311e1 (0x5b48 bytes) 000875 1790921932.162625 mapwrite 0x01858a thru 0x01cb78 (0x45ef bytes) 000876 1790921932.214752 mapread 0x011e9c thru 0x01c12e (0xa293 bytes) 000877 1790921932.215952 mapwrite 0x03a5a4 thru 0x03bb80 (0x15dd bytes) 000878 1790921932.232661 punch from 0x02466a to 0x02d0bb (0x8a52 bytes) 000879 1790921932.237987 punch from 0x03ec42 to 0x03ffff (0x13be bytes) 000880 1790921932.245442 write 0x02eede thru 0x03dd81 (0xeea4 bytes) 000881 1790921932.247019 fallocate 0x007898 thru 0x00dff8 (0x6761 bytes) 000882 1790921932.252274 read 0x007c3e thru 0x0177b4 (0xfb77 bytes) 000884 1790921932.254361 write 0x00e796 thru 0x01e077 (0xf8e2 bytes) 000885 1790921932.256373 trunc from 0x040000 to 0x01dd14 (0x222eb bytes) 000887 1790921932.278688 punch from 0x01938a to 0x01dd14 (0x498b bytes) 000891 1790921932.300483 trunc from 0x01dd15 to 0x02da9b (0xfd87 bytes) 000892 1790921932.314401 trunc from 0x02da9c to 0x0189fb (0x150a0 bytes) 000893 1790921932.331173 mapread 0x01271e thru 0x0163c5 (0x3ca8 bytes) 000894 1790921932.337234 MIRROR_EXTEND to 2 mirrors 000895 1790921932.967370 mapread 0x0135f3 thru 0x0189fb (0x5409 bytes) 000896 1790921933.031335 MIRROR_RESYNC 2 mirrors 000897 1790921933.513102 mapread 0x002486 thru 0x007480 (0x4ffb bytes) 000898 1790921933.638665 trunc from 0x008701 to 0x009d4a (0x164a bytes) 000900 1790921933.710738 read 0x0115d7 thru 0x0189fb (0x7425 bytes) 000902 1790921933.724338 trunc from 0x00e049 to 0x0189fb (0xa9b3 bytes) 000903 1790921933.730327 trunc from 0x00e20f to 0x0189fb (0xa7ed bytes) 000904 1790921933.738726 mapread 0x00f3fc thru 0x0189fb (0x9600 bytes) 000905 1790921933.782956 mapread 0x012f96 thru 0x0189fb (0x5a66 bytes) 000906 1790921933.784360 trunc from 0x0189fc to 0x008c2a (0xfdd1 bytes) 000907 1790921933.866258 mapread 0x0036ce thru 0x008c2a (0x555d bytes) 000908 1790921933.876778 punch from 0x005aec to 0x007224 (0x1739 bytes) 000909 1790921933.889106 read 0x007b02 thru 0x008c2a (0x1129 bytes) 000910 1790921933.903058 trunc from 0x003ca3 to 0x008c2a (0x4f88 bytes) 000911 1790921933.914493 write 0x02d088 thru 0x03981d (0xc796 bytes) 000912 1790921933.917223 read 0x00462d thru 0x01059f (0xbf73 bytes) 000913 1790921933.934715 write 0x02bccc thru 0x037ca8 (0xbfdd bytes) 000914 1790921933.947255 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3981e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3981e): Mirror 2: 0x799e43e6 (stale) CRC-32 checksum value for chunk [0, 0x3981e): Mirror 1: 0xfc21e142 Mirror 3: 0xfc21e142 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3981e: skipped 000915 1790921935.042396 MIRROR_RESYNC 3 mirrors 000916 1790921935.786807 trunc from 0x01bcef to 0x02950b (0xd81d bytes) 000917 1790921935.838498 write 0x01c3be thru 0x01e2ca (0x1f0d bytes) 000918 1790921935.850255 trunc from 0x0079ed to 0x010ab6 (0x90ca bytes) 000919 1790921935.856366 mapwrite 0x02f91a thru 0x03905a (0x9741 bytes) 000920 1790921935.931355 trunc from 0x03981e to 0x02e385 (0xb498 bytes) 000921 1790921935.961680 fallocate 0x01a125 thru 0x020285 (0x6161 bytes) 000922 1790921935.977309 trunc from 0x0126cb to 0x017623 (0x4f59 bytes) 000923 1790921935.990609 fallocate 0x01ce0b thru 0x029774 (0xc96a bytes) 000924 1790921936.003816 fallocate 0x03a78f thru 0x03ffff (0x5871 bytes) 000925 1790921936.027794 punch from 0x001f42 to 0x00ae10 (0x8ecf bytes) 000926 1790921936.038450 punch from 0x02cca9 to 0x02e385 (0x16dd bytes) 000927 1790921936.047484 read 0x01da5c thru 0x023b6a (0x610f bytes) 000928 1790921936.059507 trunc from 0x014013 to 0x01f8c5 (0xb8b3 bytes) 000929 1790921936.068012 write 0x0029cb thru 0x008969 (0x5f9f bytes) 000930 1790921936.089827 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e386): Mirror 3: 0x76cb3336 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e386): Mirror 3: 0x76cb3336 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e386: skipped 000931 1790921936.459522 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa6:0x0]: Invalid argument (22) 000932 1790921936.663809 mapread 0x0289c6 thru 0x02bf32 (0x356d bytes) 000933 1790921936.716254 write 0x01a81d thru 0x023f28 (0x970c bytes) 000934 1790921936.734808 trunc from 0x00ddfc to 0x019e02 (0xc007 bytes) 000935 1790921936.741701 write 0x007558 thru 0x00c940 (0x53e9 bytes) 000936 1790921936.753807 read 0x02b733 thru 0x02e385 (0x2c53 bytes) 000937 1790921936.754732 fallocate 0x033ffd thru 0x03e3af (0xa3b3 bytes) 000938 1790921936.760820 read 0x02316b thru 0x02c003 (0x8e99 bytes) 000939 1790921936.761940 fallocate 0x01d7a9 thru 0x021c4d (0x44a5 bytes) 000940 1790921936.768165 mapwrite 0x00f2b0 thru 0x017b64 (0x88b5 bytes) 000941 1790921936.831653 MIRROR_EXTEND to 2 mirrors 000942 1790921938.175306 mapwrite 0x010996 thru 0x01eff1 (0xe65c bytes) 000943 1790921938.466398 fallocate 0x0280a0 thru 0x02c67f (0x45e0 bytes) 000944 1790921938.477827 punch from 0x003d3c to 0x01344a (0xf70f bytes) 000945 1790921938.499927 write 0x02e2a2 thru 0x036f36 (0x8c95 bytes) 000946 1790921938.565179 MIRROR_RESYNC 2 mirrors 000947 1790921940.005711 mapread 0x036677 thru 0x03e3af (0x7d39 bytes) 000948 1790921940.021359 trunc from 0x03e3b0 to 0x013733 (0x2ac7c bytes) 000949 1790921940.053707 mapread 0x00bfcf thru 0x013733 (0x7765 bytes) 000950 1790921940.062152 mapread 0x00161c thru 0x0085e1 (0x6fc6 bytes) 000952 1790921940.067488 fallocate 0x0384af thru 0x03fbf3 (0x7745 bytes) 000953 1790921940.072420 mapread 0x010498 thru 0x013733 (0x329c bytes) 000954 1790921940.073398 fallocate 0x01d806 thru 0x0246ce (0x6ec9 bytes) 000955 1790921940.080996 MIRROR_RESYNC 2 mirrors 000956 1790921940.477658 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa6:0x0]: Invalid argument (22) 000957 1790921940.578765 write 0x003c3d thru 0x011eef (0xe2b3 bytes) 000958 1790921940.601744 punch from 0x013224 to 0x013733 (0x510 bytes) 000959 1790921940.606414 trunc from 0x003fef to 0x00e315 (0xa327 bytes) 000960 1790921940.616967 write 0x03019b thru 0x03bb92 (0xb9f8 bytes) 000961 1790921940.618470 punch from 0x01925a to 0x019e2b (0xbd2 bytes) 000962 1790921940.622671 read 0x00c743 thru 0x00e231 (0x1aef bytes) 000963 1790921940.632602 punch from 0x01337c to 0x020e78 (0xdafd bytes) 000964 1790921940.636632 fallocate 0x02d027 thru 0x035e91 (0x8e6b bytes) 000965 1790921940.640826 mapwrite 0x022461 thru 0x02a05e (0x7bfe bytes) 000967 1790921940.672429 punch from 0x01533e to 0x023fc5 (0xec88 bytes) 000968 1790921940.677109 trunc from 0x03bb93 to 0x01498e (0x27204 bytes) 000969 1790921940.705033 mapread 0x010d4d thru 0x01498e (0x3c42 bytes) 000970 1790921940.714338 write 0x03a0e0 thru 0x03ffff (0x5f20 bytes) 000971 1790921940.720579 MIRROR_EXTEND to 2 mirrors 000973 1790921941.581372 write 0x03a440 thru 0x03df60 (0x3b21 bytes) 000974 1790921941.623280 mapwrite 0x001cd2 thru 0x0092a1 (0x75d0 bytes) 000975 1790921941.661999 trunc from 0x040000 to 0x035dd2 (0xa22d bytes) 000976 1790921941.680614 read 0x00e405 thru 0x011720 (0x331c bytes) 000977 1790921941.688362 read 0x009e90 thru 0x0111df (0x7350 bytes) 000978 1790921941.695436 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x35dd3): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35dd3): Mirror 2: 0xc5ead1b (stale) CRC-32 checksum value for chunk [0, 0x35dd3): Mirror 1: 0x51c7982a Mirror 3: 0x51c7982a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35dd3: skipped 000979 1790921942.880572 punch from 0x0250ea to 0x033709 (0xe620 bytes) 000980 1790921942.946646 punch from 0x024ba2 to 0x029265 (0x46c4 bytes) 000981 1790921942.956798 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35dd3): Mirror 3: 0x51c7982a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35dd3): Mirror 3: 0x51c7982a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35dd3: skipped 000982 1790921943.342702 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa6:0x0]: Invalid argument (22) 000984 1790921943.522474 MIRROR_EXTEND to 2 mirrors 000985 1790921944.425363 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa6:0x0]: Invalid argument (22) 000986 1790921944.593225 mapwrite 0x02e3bc thru 0x02f515 (0x115a bytes) 000987 1790921944.662808 fallocate 0x010105 thru 0x01c036 (0xbf32 bytes) 000988 1790921944.667313 trunc from 0x035dd3 to 0x022970 (0x13462 bytes) 000989 1790921944.692331 mapwrite 0x005a13 thru 0x00d955 (0x7f43 bytes) 000990 1790921944.754718 fallocate 0x02e7cf thru 0x032eab (0x46dd bytes) 000992 1790921944.761022 mapwrite 0x00b858 thru 0x0158f0 (0xa099 bytes) 000993 1790921944.795155 punch from 0x014932 to 0x0196ba (0x4d89 bytes) 000994 1790921944.801966 trunc from 0x032eac to 0x0226cb (0x107e0 bytes) 000996 1790921944.821474 trunc from 0x0199ab to 0x01adb7 (0x140d bytes) 000999 1790921944.835281 MIRROR_EXTEND to 2 mirrors 001000 1790921945.504159 read 0x021b8f thru 0x0226cb (0xb3d bytes) All operations completed A-OK! PASS 70a (94s) == sanity-flr test 71: check mirror extend parent fid ==== 02:19:13 (1790921953) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x16b:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (12s) == sanity-flr test 72: check append on FLR file ========== 02:19:26 (1790921966) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x154:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.234612 s, 8.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x154:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x155:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116392 s, 9.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x154:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x155:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x135:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x154:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x155:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x135:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x156:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 02:19:36 (1790921976) Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre2 Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 4814848 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6424736 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8978432 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7713600 .. Extending file size to 8656608 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9018880 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61912 61914 61916 61917 61918 done 192.168.202.116@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg216-client.virtnet /mnt/lustre2 (opts:) 192.168.202.116@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg216-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 02:20:58 (1790922058) Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre2 Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3221888 .. Extending file size to 1563584 .. failed Extending file size to 7335008 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6131072 .. Extending file size to 8167744 .. Extending file size to 8162976 .. Extending file size to 8564896 .. Extending file size to 9032288 .. Extending file size to 9057280 .. Extending file size to 8972128 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9256064 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 63338 63339 63340 63343 63344 failed 192.168.202.116@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg216-client.virtnet /mnt/lustre2 (opts:) 192.168.202.116@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg216-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 02:22:17 (1790922137) Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2288 1269016 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2032 1269272 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24568 3582452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18848 3588172 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43416 7170624 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2288 1269016 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2032 1269272 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24568 3582452 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18848 3588172 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43416 7170624 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.164906 s, 12.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.257291 s, 8.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x4bf19fce (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.116@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg216-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 02:22:40 (1790922160) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0975164 s, 21.5 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 02:22:48 (1790922168) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xe7:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14b:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14b:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14c:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x16b:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 02:22:56 (1790922176) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.142014 s, 14.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1c5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1c6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 02:23:05 (1790922185) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0942277 s, 22.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x16c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x16c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x16d:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 02:23:14 (1790922194) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.156103 s, 13.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x16e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x16e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x16f:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 02:23:49 (1790922229) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg216-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104433 s, 10.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1c8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1c7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1c8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1c7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c7:0x0] } PASS 204d (53s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 02:24:42 (1790922282) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg216-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.105988 s, 19.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1c9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1c9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (51s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 02:25:33 (1790922333) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg216-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg216-server: Pool lustre.test_204f created Adding targets to pool oleg216-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg216-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.173257 s, 12.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x150:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x170:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x150:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x170:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg216-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg216-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg216-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 02:26:53 (1790922413) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 02:27:02 (1790922422) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 02:27:12 (1790922432) oleg216-server: Pool lustre.test_206 created Adding targets to pool oleg216-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg216-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg216-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg216-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg216-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 02:27:41 (1790922461) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112032 s, 9.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319003 s, 32.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xd0d9730a Mirror 3: 0xd0d9730a PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 02:27:54 (1790922474) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 02:27:59 (1790922479) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 02:28:03 (1790922483) Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0789412 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x21f0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1d80000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1d80000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1d80000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x16e0000 Read this (4 bytes): BBBB 192.168.202.116@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg216-client.virtnet /mnt/lustre2 (opts:) PASS 209a (40s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 02:28:43 (1790922523) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0289446 s, 283 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 02:28:50 (1790922530) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939072 s, 11.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x106 lmm_fid: [0x200000405:0x106:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x106 lmm_fid: [0x200000405:0x106:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x17b:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-10-02 02:28:52.000000000 -0400 Modify: 2026-10-02 02:28:52.000000000 -0400 Change: 2026-10-02 02:28:53.000000000 -0400 Birth: 2026-10-02 02:28:51.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 02:29:00 (1790922540) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826911 s, 12.7 MB/s oleg216-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg216-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg216-server: O/280000401/d28/17c: File not found by ext2_lookup PASS 210b (32s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 02:29:32 (1790922572) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.269913 s, 152 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0203248 s, 202 kB/s size after second write -rw-r--r-- 1 root root 4096 Oct 2 02:29 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10b:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Oct 2 02:29 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 02:29:41 (1790922581) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (11s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 02:29:51 (1790922591) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 2016 sec =========== 02:30:02 (1790922602) === sanity-flr: start cleanup 02:30:04 (1790922604) === === sanity-flr: finish cleanup 02:30:14 (1790922614) ===