-----============= acceptance-small: sanity-flr ============----- Fri Sep 11 08:46:28 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg131-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 08:46:51 (1789130811) === oleg131-client.virtnet: executing check_config_client /mnt/lustre oleg131-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg131-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8be4899ad800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8be4899ad800.idle_timeout=debug disable quota as required oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 Setting lod.lustre-MDT0000-mdtlov.enable_compr_rotational from 0 to 1 Waiting 90s for '1' Updated after 2s: want '1' got '1' Setting lod.lustre-MDT0001-mdtlov.enable_compr_rotational from 0 to 1 Waiting 90s for '1' Updated after 2s: want '1' got '1' === sanity-flr: finish setup 08:47:59 (1789130879) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8248] == sanity-flr test 0a: lfs mirror create with -N option == 08:48:03 (1789130883) PASS 0a (23s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 08:48:26 (1789130906) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 08:48:30 (1789130910) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 08:48:34 (1789130914) PASS 0d (33s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 08:49:07 (1789130947) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 08:49:12 (1789130952) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 08:49:16 (1789130956) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8be4899ad800.stats=clear osc.lustre-OST0001-osc-ffff8be4899ad800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 08:49:25 (1789130965) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 08:49:38 (1789130978) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 08:49:49 (1789130989) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 08:49:53 (1789130993) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 08:50:03 (1789131003) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (12s) == sanity-flr test 2: create components from existing files ========================================================== 08:50:15 (1789131015) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 08:50:24 (1789131024) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 08:50:33 (1789131033) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 08:50:42 (1789131042) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 08:50:55 (1789131055) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 08:51:02 (1789131062) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.745043 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864655 s, 12.1 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 08:51:12 (1789131072) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.314153 s, 26.7 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 08:51:22 (1789131082) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 08:51:34 (1789131094) Stopping /mnt/lustre-ost1 (opts:) on oleg131-server oleg131-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8be4899ad800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8be4899ad800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg131-server: oleg131-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg131-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg131-server oleg131-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8be4899ad800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8be4899ad800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg131-server: oleg131-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg131-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (83s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 08:52:57 (1789131177) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.345293 s, 24.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg131-server oleg131-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8be4899ad800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8be4899ad800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg131-server: oleg131-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg131-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (52s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 08:53:49 (1789131229) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg131-server oleg131-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8be4899ad800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8be4899ad800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg131-server: oleg131-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg131-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg131-server oleg131-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8be4899ad800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8be4899ad800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg131-server: oleg131-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg131-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (92s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 08:55:22 (1789131322) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.01805 s, 13.0 MB/s -rw-r--r-- 1 root root 52428800 Sep 11 08:55 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.982s user 0m0.002s sys 0m0.182s Stopping /mnt/lustre-ost1 (opts:) on oleg131-server oleg131-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8be4899ad800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8be4899ad800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.478s user 0m0.002s sys 0m0.204s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg131-server: oleg131-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg131-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg131-server oleg131-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8be4899ad800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8be4899ad800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.283s user 0m0.001s sys 0m0.182s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg131-server: oleg131-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg131-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg131-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8be4899ad800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8be4899ad800.ost_server_uuid in FULL state after 0 sec PASS 33b (108s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 08:57:10 (1789131430) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 08:57:13 (1789131433) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 08:57:17 (1789131437) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 08:57:21 (1789131441) PASS 35 (11s) == sanity-flr test 36a: write to mirrored files ========== 08:57:32 (1789131452) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 1.07524 s, 11.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (19s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 08:57:51 (1789131471) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885813 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484856 s, 21.6 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 08:58:00 (1789131480) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 08:58:09 (1789131489) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00122704 s, 835 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00195071 s, 525 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00132575 s, 772 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0015895 s, 644 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 08:58:21 (1789131501) create test files with size 15741169 .. e28ddc333c3f0dae696255a6fa0832df - 6ba57f97928943e739401ffcac1f7895 - 9bc5076fd3a1ecfe9f120d6119b74d41 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg131-client.virtnet /mnt/lustre (opts:) Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 08:58:48 (1789131528) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 08:59:05 (1789131545) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 08:59:13 (1789131553) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600605 s, 17.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xba:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435467 s, 24.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xba:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 08:59:24 (1789131564) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.112718 s, 18.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.329948 s, 12.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.204129 s, 10.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 08:59:43 (1789131583) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 08:59:47 (1789131587) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 08:59:51 (1789131591) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554581 s, 18.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (10s) == sanity-flr test 43c: read prefer uncompressed mirror == 09:00:01 (1789131601) llite.lustre-ffff8be4855f0000.enable_compression=1 striped dir -i1 -c2 -H crush /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (9s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 09:00:10 (1789131610) llite.lustre-ffff8be4855f0000.enable_compression=1 striped dir -i1 -c2 -H crush /mnt/lustre/d43d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634069 s, 16.5 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286479 s, 36.6 MB/s **verify components PASS 43d (10s) == sanity-flr test 44a: lfs mirror split check =========== 09:00:20 (1789131620) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.328131 s, 9.6 MB/s PASS 44a (12s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:00:32 (1789131632) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc8:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:00:43 (1789131643) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.299721 s, 35.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x9:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg131-client.virtnet /mnt/lustre (opts:) Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:00:55 (1789131655) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8be491362000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.259012 s, 40.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg131-client.virtnet /mnt/lustre (opts:) Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8be492b5a000.xattr_cache=1 PASS 44e (18s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:01:13 (1789131673) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (40s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:01:53 (1789131713) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:02:34 (1789131754) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (5s) == sanity-flr test 48: Verify snapshot mirror ============ 09:02:39 (1789131759) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.323123 s, 9.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.114256 s, 27.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x38cdea3f (nosync,stale) Mirror 2: 0x38cdea3f (nosync,stale) ** verify mirror 2 doesn't change original checksum: e004d2365019d1edd861d5ebfaf1dec6 - mirror 1 checksum: 510858b74f74fa51f0be701c13c4341a - mirror 2 checksum: e004d2365019d1edd861d5ebfaf1dec6 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-11T09:02:43-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-11T09:02:43-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: e004d2365019d1edd861d5ebfaf1dec6 - mirror 1 checksum: 510858b74f74fa51f0be701c13c4341a - mirror 2 checksum: 510858b74f74fa51f0be701c13c4341a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-11T09:02:43-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-11T09:02:43-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } PASS 48 (15s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 09:02:54 (1789131774) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553346 s, 18.9 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 314368.. 315391: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (11s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:03:05 (1789131785) Failing mds1 on oleg131-server Stopping /mnt/lustre-mds1 (opts:) on oleg131-server 09:03:17 (1789131797) shut down facet: mds1 facet_host: oleg131-server facet_failover_host: oleg131-server Failover mds1 to oleg131-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0000 09:03:53 (1789131833) targets are mounted 09:03:53 (1789131833) facet_failover done oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506065 s, 20.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } PASS 50A (68s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:04:13 (1789131853) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0500775 s, 409 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0233464 s, 877 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (12s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:04:26 (1789131866) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.102312 s, 200 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0138729 s, 1.5 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (15s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:04:40 (1789131880) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00686528 s, 2.4 MB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:04:51 (1789131891) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0687379 s, 298 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (12s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:05:02 (1789131902) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0103534 s, 396 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0434163 s, 94.3 kB/s PASS 60a (11s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:05:13 (1789131913) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:05:39 (1789131939) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:06:10 (1789131970) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x47:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 09:06:36 (1789131996) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (28s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 09:07:04 (1789132024) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 09:07:13 (1789132033) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 59172 mirror split pid 59173 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 59172 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 59173 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 09:08:23 (1789132103) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 2106 000001 1789132106.625504 write 0x031d94 thru 0x03b794 (0x9a01 bytes) 000002 1789132106.668775 mapwrite 0x031aeb thru 0x038f4e (0x7464 bytes) 000003 1789132106.728799 read 0x01e271 thru 0x024886 (0x6616 bytes) fallocating to largest ever: 0x36b9e 000004 1789132106.747259 fallocate 0x02f84a thru 0x036b9d (0x7354 bytes) 000006 1789132106.771455 punch from 0x01b734 to 0x026d42 (0xb60f bytes) 000007 1789132106.813024 MIRROR_EXTEND to 2 mirrors 000008 1789132108.422777 trunc from 0x00d358 to 0x019020 (0xbcc9 bytes) 000009 1789132108.505816 mapread 0x014aed thru 0x0249d6 (0xfeea bytes) 000010 1789132108.567943 punch from 0x020082 to 0x02ab31 (0xaab0 bytes) 000011 1789132108.584325 write 0x01e773 thru 0x02cbbb (0xe449 bytes) 000012 1789132108.602751 fallocate 0x009f80 thru 0x014684 (0xa705 bytes) 000013 1789132108.621496 mapread 0x009262 thru 0x0146a3 (0xb442 bytes) 000014 1789132108.652139 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) 000015 1789132109.095747 read 0x000619 thru 0x002129 (0x1b11 bytes) 000016 1789132109.108271 fallocate 0x019127 thru 0x0269df (0xd8b9 bytes) 000018 1789132109.138342 read 0x003fd6 thru 0x008c93 (0x4cbe bytes) 000019 1789132109.211734 trunc from 0x03b795 to 0x0012bd (0x3a4d7 bytes) 000020 1789132109.256876 write 0x031924 thru 0x03ffff (0xe6dc bytes) 000022 1789132109.288485 mapread 0x02fac9 thru 0x03dfcf (0xe507 bytes) 000024 1789132109.326630 mapwrite 0x03eb0a thru 0x03ffff (0x14f6 bytes) 000026 1789132109.381403 write 0x017b84 thru 0x0214d0 (0x994d bytes) 000027 1789132109.436000 read 0x009386 thru 0x00ea21 (0x569c bytes) 000029 1789132109.469649 read 0x007897 thru 0x00a56d (0x2cd7 bytes) 000030 1789132109.470289 fallocate 0x019605 thru 0x025c73 (0xc66f bytes) 000031 1789132109.488293 read 0x03c06b thru 0x03ffff (0x3f95 bytes) 000034 1789132109.496873 trunc from 0x040000 to 0x00aa54 (0x355ab bytes) 000035 1789132109.534394 mapread 0x005240 thru 0x00aa54 (0x5815 bytes) 000036 1789132109.558522 read 0x006415 thru 0x00aa54 (0x4640 bytes) 000037 1789132109.559644 fallocate 0x02f152 thru 0x0357c5 (0x6674 bytes) fallocating to largest ever: 0x40000 000039 1789132109.572117 fallocate 0x03b512 thru 0x03ffff (0x4aee bytes) 000040 1789132109.590969 read 0x002f57 thru 0x0128a1 (0xf94b bytes) 000041 1789132109.626949 read 0x007a03 thru 0x012dc3 (0xb3c1 bytes) 000042 1789132109.628925 mapread 0x029bc6 thru 0x02b488 (0x18c3 bytes) 000043 1789132109.630410 fallocate 0x03355f thru 0x03ffff (0xcaa1 bytes) 000044 1789132109.655799 fallocate 0x00cf56 thru 0x01aa85 (0xdb30 bytes) 000045 1789132109.666931 mapwrite 0x004f38 thru 0x00e9d5 (0x9a9e bytes) 000046 1789132109.739783 trunc from 0x018738 to 0x024346 (0xbc0f bytes) 000047 1789132109.760989 fallocate 0x02e7b3 thru 0x02f499 (0xce7 bytes) 000048 1789132109.778565 trunc from 0x01eeee to 0x01f0a7 (0x1ba bytes) 000049 1789132109.792384 trunc from 0x00a1bb to 0x00c658 (0x249e bytes) 000050 1789132109.805539 trunc from 0x006ac6 to 0x01483c (0xdd77 bytes) 000051 1789132109.814717 punch from 0x0002f3 to 0x006d0a (0x6a18 bytes) 000052 1789132109.840158 MIRROR_EXTEND to 2 mirrors 000053 1789132110.973947 punch from 0x0276e5 to 0x033f24 (0xc840 bytes) 000054 1789132111.033757 fallocate 0x01afb0 thru 0x023a11 (0x8a62 bytes) 000055 1789132111.053523 trunc from 0x01172b to 0x013c4d (0x2523 bytes) 000056 1789132111.074787 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe20eea22 Mirror 3: 0xe20eea22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000058 1789132112.769063 mapwrite 0x02b3f3 thru 0x0381a2 (0xcdb0 bytes) 000059 1789132112.886461 trunc from 0x040000 to 0x034a75 (0xb58a bytes) 000060 1789132112.917055 read 0x016a9d thru 0x021555 (0xaab9 bytes) 000061 1789132112.928916 fallocate 0x000ad0 thru 0x007c5a (0x718b bytes) 000062 1789132112.934342 write 0x0085c4 thru 0x0120bf (0x9afc bytes) 000063 1789132112.944102 fallocate 0x00132a thru 0x006f93 (0x5c6a bytes) 000064 1789132112.948781 punch from 0x01d77d to 0x020f1a (0x379e bytes) 000065 1789132112.953490 read 0x02016a thru 0x02880c (0x86a3 bytes) 000066 1789132112.963616 write 0x00edb9 thru 0x016b66 (0x7dae bytes) 000067 1789132112.965049 mapread 0x00c927 thru 0x015872 (0x8f4c bytes) 000068 1789132112.966744 trunc from 0x034a76 to 0x015895 (0x1f1e0 bytes) 000069 1789132112.985640 write 0x01ac07 thru 0x025490 (0xa88a bytes) 000070 1789132112.987309 mapread 0x000138 thru 0x00f20b (0xf0d4 bytes) 000071 1789132112.989417 mapread 0x025221 thru 0x025490 (0x270 bytes) 000072 1789132112.990499 read 0x004776 thru 0x00dcee (0x9579 bytes) 000073 1789132112.991113 write 0x03f484 thru 0x03ffff (0xb7c bytes) 000074 1789132112.997144 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000075 1789132113.396744 trunc from 0x040000 to 0x02468b (0x1b974 bytes) 000076 1789132113.417818 fallocate 0x01ca60 thru 0x028e9a (0xc43b bytes) 000078 1789132113.425127 mapwrite 0x0032f5 thru 0x00813b (0x4e47 bytes) 000079 1789132113.467525 trunc from 0x012b0c to 0x01e726 (0xbc1b bytes) 000080 1789132113.480676 MIRROR_RESYNC 2 mirrors 000081 1789132114.658217 read 0x02401d thru 0x028e9a (0x4e7e bytes) 000082 1789132114.692788 mapwrite 0x03b6f7 thru 0x03dec7 (0x27d1 bytes) 000083 1789132114.917517 trunc from 0x03dec8 to 0x00513d (0x38d8a bytes) 000084 1789132114.978416 MIRROR_RESYNC 2 mirrors 000085 1789132116.024033 fallocate 0x02ea5c thru 0x0349bd (0x5f62 bytes) 000086 1789132116.178509 mapwrite 0x034e2d thru 0x03bf34 (0x7108 bytes) 000087 1789132116.449079 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) 000088 1789132116.840322 read 0x03a00e thru 0x03b034 (0x1027 bytes) 000089 1789132116.856766 mapwrite 0x002d98 thru 0x00c8d2 (0x9b3b bytes) 000091 1789132117.031663 punch from 0x01ac52 to 0x01baa7 (0xe56 bytes) 000092 1789132117.042435 fallocate 0x019bed thru 0x028af3 (0xef07 bytes) 000093 1789132117.078102 trunc from 0x011daf to 0x017324 (0x5576 bytes) 000094 1789132117.103411 write 0x034dcf thru 0x03ee99 (0xa0cb bytes) 000096 1789132117.134720 punch from 0x032eb3 to 0x03e56b (0xb6b9 bytes) 000098 1789132117.190809 trunc from 0x00b336 to 0x018756 (0xd421 bytes) 000099 1789132117.209485 punch from 0x03be63 to 0x03ee99 (0x3037 bytes) 000100 1789132117.236384 mapread 0x02284b thru 0x023701 (0xeb7 bytes) 000101 1789132117.283670 fallocate 0x019df3 thru 0x022054 (0x8262 bytes) 000102 1789132117.298684 fallocate 0x0118c0 thru 0x01fe19 (0xe55a bytes) 000104 1789132117.328913 write 0x01e9c6 thru 0x027d16 (0x9351 bytes) 000105 1789132117.340743 fallocate 0x01c383 thru 0x029dff (0xda7d bytes) 000106 1789132117.347293 mapread 0x00d5ce thru 0x01cec1 (0xf8f4 bytes) 000107 1789132117.384295 mapwrite 0x039542 thru 0x03ffff (0x6abe bytes) 000108 1789132117.463753 fallocate 0x00dafd thru 0x017950 (0x9e54 bytes) 000109 1789132117.481399 MIRROR_EXTEND to 2 mirrors 000111 1789132118.411711 mapwrite 0x02d179 thru 0x03c221 (0xf0a9 bytes) 000113 1789132118.567112 mapread 0x032d3b thru 0x03f00e (0xc2d4 bytes) 000114 1789132118.587534 MIRROR_RESYNC 2 mirrors 000115 1789132120.044167 punch from 0x012c65 to 0x017f39 (0x52d5 bytes) 000116 1789132120.168452 MIRROR_RESYNC 2 mirrors 000118 1789132121.280161 mapread 0x026a39 thru 0x033117 (0xc6df bytes) 000119 1789132121.311980 MIRROR_EXTEND to 3 mirrors 000120 1789132122.352118 punch from 0x03e15f to 0x03ffff (0x1ea1 bytes) 000121 1789132122.413958 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb01f2b64 (stale) Mirror 3: 0xb01f2b64 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x36961830 Mirror 4: 0x36961830 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000122 1789132123.891619 mapread 0x025d0a thru 0x02b827 (0x5b1e bytes) 000123 1789132123.910917 fallocate 0x0318f9 thru 0x03ffff (0xe707 bytes) 000124 1789132123.984404 write 0x015fc3 thru 0x024b22 (0xeb60 bytes) 000125 1789132124.001714 punch from 0x02f700 to 0x03be0b (0xc70c bytes) 000126 1789132124.011070 mapwrite 0x02a7e9 thru 0x02cf14 (0x272c bytes) 000128 1789132124.077309 trunc from 0x040000 to 0x0281db (0x17e24 bytes) 000129 1789132124.117428 fallocate 0x021039 thru 0x030d78 (0xfd40 bytes) 000130 1789132124.121494 punch from 0x0237f8 to 0x0281db (0x49e4 bytes) 000132 1789132124.138208 mapwrite 0x0185c8 thru 0x01fc35 (0x766e bytes) 000133 1789132124.175234 fallocate 0x021add thru 0x02321e (0x1742 bytes) 000134 1789132124.192058 trunc from 0x01c332 to 0x02630d (0x9fdc bytes) 000135 1789132124.197966 read 0x017a90 thru 0x018c71 (0x11e2 bytes) 000136 1789132124.199657 write 0x03d0a3 thru 0x03fa8e (0x29ec bytes) 000138 1789132124.201713 mapread 0x01a5cd thru 0x028aa6 (0xe4da bytes) 000139 1789132124.217535 punch from 0x00c071 to 0x01b386 (0xf316 bytes) 000141 1789132124.222657 mapread 0x0251c7 thru 0x032715 (0xd54f bytes) 000142 1789132124.230945 trunc from 0x0303bc to 0x03a66c (0xa2b1 bytes) 000143 1789132124.239877 mapread 0x03da5b thru 0x03fa8e (0x2034 bytes) 000144 1789132124.242336 mapread 0x001efa thru 0x011dca (0xfed1 bytes) 000145 1789132124.274339 trunc from 0x03fa8f to 0x010fdb (0x2eab3 bytes) 000146 1789132124.292859 mapwrite 0x01413c thru 0x01c580 (0x8445 bytes) 000147 1789132124.335615 trunc from 0x002e90 to 0x00a87b (0x79ec bytes) 000150 1789132124.341255 mapwrite 0x02261d thru 0x025a3b (0x341f bytes) 000151 1789132124.365218 mapwrite 0x03f34e thru 0x03ffff (0xcb2 bytes) 000152 1789132124.401596 punch from 0x007909 to 0x00ccad (0x53a5 bytes) 000153 1789132124.406923 trunc from 0x022151 to 0x031c5d (0xfb0d bytes) 000154 1789132124.415979 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb01f2b64 (stale) Mirror 3: 0xb01f2b64 (stale) Mirror 4: 0x36961830 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x160b18d Mirror 5: 0x160b18d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000155 1789132125.181771 trunc from 0x040000 to 0x020c35 (0x1f3ca bytes) 000156 1789132125.269594 trunc from 0x020c36 to 0x0277c5 (0x6b90 bytes) 000157 1789132125.292314 fallocate 0x004005 thru 0x00bd20 (0x7d1c bytes) 000158 1789132125.301020 write 0x020f19 thru 0x021227 (0x30f bytes) 000159 1789132125.314053 fallocate 0x02ac8a thru 0x02bcbb (0x1032 bytes) 000161 1789132125.337410 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x277c6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x277c6): Mirror 2: 0x6375a0af (stale) Mirror 3: 0x6375a0af (stale) Mirror 4: 0x6375a0af (stale) Mirror 5: 0xf43b2aca (stale) CRC-32 checksum value for chunk [0, 0x277c6): Mirror 1: 0x23fe3dec Mirror 6: 0x23fe3dec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x277c6: skipped 000162 1789132127.353533 trunc from 0x0277c6 to 0x01eac7 (0x8cfe bytes) 000163 1789132127.463554 mapread 0x00e422 thru 0x0167fc (0x83db bytes) 000164 1789132127.516089 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1eac8): Mirror 2: 0xf6c4abd6 (stale) Mirror 3: 0xf6c4abd6 (stale) Mirror 4: 0xf6c4abd6 (stale) Mirror 5: 0xe26defa8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1eac8): Mirror 2: 0xf6c4abd6 (stale) Mirror 3: 0xf6c4abd6 (stale) Mirror 4: 0xf6c4abd6 (stale) Mirror 5: 0xe26defa8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1eac8: skipped 000165 1789132127.974743 punch from 0x01e745 to 0x01eac7 (0x383 bytes) 000166 1789132127.984035 mapread 0x00db29 thru 0x013c51 (0x6129 bytes) 000167 1789132128.005431 write 0x032461 thru 0x0335d2 (0x1172 bytes) 000168 1789132128.011164 trunc from 0x0335d3 to 0x03b65b (0x8089 bytes) 000169 1789132128.031830 write 0x023e48 thru 0x02936e (0x5527 bytes) 000170 1789132128.045794 trunc from 0x03b65c to 0x00db8f (0x2dacc bytes) 000171 1789132128.070261 mapread 0x004f14 thru 0x00a563 (0x5650 bytes) 000172 1789132128.084665 mapread 0x003dd2 thru 0x00cff9 (0x9228 bytes) 000173 1789132128.086458 fallocate 0x02ddfe thru 0x03afc9 (0xd1cc bytes) 000174 1789132128.092623 mapwrite 0x016ee8 thru 0x01fe0a (0x8f23 bytes) 000176 1789132128.173607 read 0x009b0d thru 0x0189c2 (0xeeb6 bytes) 000177 1789132128.187368 trunc from 0x03afca to 0x0335eb (0x79de bytes) 000178 1789132128.230411 mapwrite 0x023350 thru 0x031414 (0xe0c5 bytes) 000179 1789132128.301774 trunc from 0x01986f to 0x0255da (0xbd6c bytes) 000180 1789132128.311518 write 0x03af2e thru 0x03ffff (0x50d2 bytes) 000181 1789132128.317860 MIRROR_RESYNC 5 mirrors 000182 1789132130.732983 punch from 0x037ac4 to 0x03fbaa (0x80e7 bytes) 000183 1789132130.795398 mapwrite 0x039734 thru 0x03ec09 (0x54d6 bytes) 000184 1789132130.834865 punch from 0x007ad3 to 0x00c44a (0x4978 bytes) 000185 1789132130.838301 mapread 0x03c099 thru 0x03ffff (0x3f67 bytes) 000186 1789132130.839347 fallocate 0x012946 thru 0x019a01 (0x70bc bytes) 000188 1789132130.844419 read 0x007e54 thru 0x017956 (0xfb03 bytes) 000189 1789132130.854356 trunc from 0x040000 to 0x005c69 (0x3a396 bytes) 000190 1789132130.877286 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x5c6a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5c6a): Mirror 2: 0x41d78df4 (stale) Mirror 3: 0x41d78df4 (stale) Mirror 4: 0x41d78df4 (stale) Mirror 5: 0x41d78df4 (stale) CRC-32 checksum value for chunk [0, 0x5c6a): Mirror 1: 0x41d78df4 Mirror 6: 0x41d78df4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5c6a: skipped 000192 1789132132.231924 mapread 0x0015c4 thru 0x005c69 (0x46a6 bytes) 000193 1789132132.273145 write 0x000298 thru 0x00dd0e (0xda77 bytes) 000194 1789132132.327977 mapread 0x006f94 thru 0x00dd0e (0x6d7b bytes) 000195 1789132132.329498 mapwrite 0x010dba thru 0x01ff08 (0xf14f bytes) 000197 1789132132.411332 mapread 0x017d41 thru 0x01ff08 (0x81c8 bytes) 000198 1789132132.413249 fallocate 0x030ed9 thru 0x03714a (0x6272 bytes) 000201 1789132132.425167 write 0x0017ad thru 0x0072fb (0x5b4f bytes) 000202 1789132132.434434 fallocate 0x03c935 thru 0x03ffff (0x36cb bytes) 000203 1789132132.441899 trunc from 0x031983 to 0x03714a (0x57c8 bytes) 000204 1789132132.448133 punch from 0x013075 to 0x016d62 (0x3cee bytes) 000205 1789132132.456163 mapread 0x01b52b thru 0x01bdc8 (0x89e bytes) 000206 1789132132.459363 mapwrite 0x023f89 thru 0x02954a (0x55c2 bytes) 000207 1789132132.504068 MIRROR_RESYNC 6 mirrors 000208 1789132134.906038 trunc from 0x0037aa to 0x00b648 (0x7e9f bytes) 000210 1789132134.948455 fallocate 0x01f264 thru 0x01f784 (0x521 bytes) 000211 1789132134.959325 read 0x016c16 thru 0x017af6 (0xee1 bytes) 000212 1789132134.966640 punch from 0x00d1a8 to 0x010eb4 (0x3d0d bytes) 000213 1789132134.971808 trunc from 0x02376a to 0x03098b (0xd222 bytes) 000215 1789132134.978729 write 0x027893 thru 0x02e39f (0x6b0d bytes) 000216 1789132134.989667 read 0x017bf4 thru 0x01da7e (0x5e8b bytes) 000217 1789132135.000890 mapread 0x00c217 thru 0x00d189 (0xf73 bytes) 000218 1789132135.002405 fallocate 0x0368aa thru 0x0398a5 (0x2ffc bytes) 000219 1789132135.007922 trunc from 0x0398a6 to 0x03405e (0x5847 bytes) 000220 1789132135.031914 read 0x032507 thru 0x03405e (0x1b58 bytes) 000221 1789132135.033288 mapread 0x00750b thru 0x008214 (0xd0a bytes) 000222 1789132135.034919 mapwrite 0x014fbd thru 0x01b359 (0x639d bytes) 000224 1789132135.069058 trunc from 0x03405f to 0x018b1f (0x1b53f bytes) 000225 1789132135.122635 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x18b20): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18b20): Mirror 2: 0xbd2ce0af (stale) Mirror 3: 0xbd2ce0af (stale) Mirror 4: 0xbd2ce0af (stale) Mirror 5: 0xbd2ce0af (stale) Mirror 6: 0xbd2ce0af (stale) CRC-32 checksum value for chunk [0, 0x18b20): Mirror 1: 0x4361f7c3 Mirror 7: 0x4361f7c3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18b20: skipped 000226 1789132136.520030 trunc from 0x018b20 to 0x002f0a (0x15c15 bytes) 000227 1789132136.581993 mapwrite 0x03ac12 thru 0x03ffff (0x53ee bytes) 000228 1789132136.641321 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x22cc8dae (stale) Mirror 3: 0x22cc8dae (stale) Mirror 5: 0x22cc8dae (stale) Mirror 6: 0x22cc8dae (stale) Mirror 7: 0x723fec08 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x22cc8dae (stale) Mirror 3: 0x22cc8dae (stale) Mirror 5: 0x22cc8dae (stale) Mirror 6: 0x22cc8dae (stale) Mirror 7: 0x723fec08 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000229 1789132137.189602 trunc from 0x00e2fd to 0x01ac64 (0xc968 bytes) 000230 1789132137.212277 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x22cc8dae (stale) Mirror 3: 0x22cc8dae (stale) Mirror 5: 0x22cc8dae (stale) Mirror 6: 0x22cc8dae (stale) Mirror 7: 0x723fec08 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf61148a4 Mirror 8: 0xf61148a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000231 1789132138.854282 punch from 0x03fd62 to 0x03ffff (0x29e bytes) 000232 1789132138.923907 trunc from 0x040000 to 0x010e2b (0x2f1d4 bytes) 000233 1789132138.984066 punch from 0x00244e to 0x008984 (0x6537 bytes) 000234 1789132139.005428 trunc from 0x010e2c to 0x000a2b (0x10400 bytes) 000235 1789132139.058316 mapwrite 0x029dc6 thru 0x0325d1 (0x880c bytes) 000236 1789132139.229325 mapwrite 0x0008b5 thru 0x003aed (0x3239 bytes) 000237 1789132139.280808 mapread 0x0256a6 thru 0x029462 (0x3dbd bytes) 000239 1789132139.284514 read 0x0305b8 thru 0x03226f (0x1cb8 bytes) 000240 1789132139.286621 mapread 0x02b6c6 thru 0x02dfc6 (0x2901 bytes) 000241 1789132139.287809 mapread 0x02e396 thru 0x0325d1 (0x423c bytes) 000242 1789132139.306226 read 0x007563 thru 0x00bd00 (0x479e bytes) 000243 1789132139.309617 mapread 0x02973f thru 0x0325d1 (0x8e93 bytes) 000244 1789132139.314914 punch from 0x005dd5 to 0x010082 (0xa2ae bytes) 000245 1789132139.334960 fallocate 0x03cee4 thru 0x03ffff (0x311c bytes) 000246 1789132139.351982 write 0x032c5d thru 0x03f7b2 (0xcb56 bytes) 000247 1789132139.390129 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x22cc8dae (stale) Mirror 3: 0x22cc8dae (stale) Mirror 6: 0x22cc8dae (stale) Mirror 7: 0x723fec08 (stale) Mirror 8: 0xf61148a4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x22cc8dae (stale) Mirror 3: 0x22cc8dae (stale) Mirror 6: 0x22cc8dae (stale) Mirror 7: 0x723fec08 (stale) Mirror 8: 0xf61148a4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000248 1789132140.144679 trunc from 0x040000 to 0x03bd1c (0x42e3 bytes) 000249 1789132140.171113 trunc from 0x028989 to 0x02d4fd (0x4b75 bytes) 000250 1789132140.178046 trunc from 0x023345 to 0x026348 (0x3004 bytes) 000251 1789132140.191392 MIRROR_RESYNC 6 mirrors 000253 1789132142.161105 write 0x004f86 thru 0x0091e6 (0x4261 bytes) 000254 1789132142.287886 punch from 0x0062e6 to 0x00ab44 (0x485f bytes) 000255 1789132142.312832 fallocate 0x03f3c5 thru 0x03ffff (0xc3b bytes) 000256 1789132142.317899 trunc from 0x01d4ab to 0x025926 (0x847c bytes) 000257 1789132142.326964 fallocate 0x017f19 thru 0x024684 (0xc76c bytes) 000258 1789132142.331915 fallocate 0x00d671 thru 0x010854 (0x31e4 bytes) 000259 1789132142.340156 trunc from 0x040000 to 0x035194 (0xae6b bytes) 000260 1789132142.370262 mapwrite 0x03481a thru 0x03b48c (0x6c73 bytes) 000261 1789132142.428748 trunc from 0x0165c2 to 0x0240c2 (0xdb01 bytes) 000262 1789132142.439284 trunc from 0x03b48d to 0x037fb0 (0x34dc bytes) 000263 1789132142.478692 mapread 0x026c09 thru 0x02b439 (0x4831 bytes) 000264 1789132142.493240 mapread 0x000c6b thru 0x00939b (0x8731 bytes) 000265 1789132142.505116 fallocate 0x028b21 thru 0x0318f0 (0x8dd0 bytes) 000266 1789132142.513304 punch from 0x007b5c to 0x011090 (0x9535 bytes) 000267 1789132142.523166 mapread 0x020d0e thru 0x02d8f1 (0xcbe4 bytes) 000268 1789132142.528928 punch from 0x032fbd to 0x037fb0 (0x4ff4 bytes) 000269 1789132142.542448 mapwrite 0x00a4cb thru 0x00cbf1 (0x2727 bytes) 000270 1789132142.578773 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37fb1): Mirror 2: 0xd21b4d76 (stale) Mirror 6: 0xd21b4d76 (stale) Mirror 7: 0xd21b4d76 (stale) Mirror 8: 0xd21b4d76 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37fb1): Mirror 2: 0xd21b4d76 (stale) Mirror 6: 0xd21b4d76 (stale) Mirror 7: 0xd21b4d76 (stale) Mirror 8: 0xd21b4d76 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37fb1: skipped 000271 1789132143.104814 mapread 0x0160c9 thru 0x0247f9 (0xe731 bytes) 000272 1789132143.143593 write 0x00df33 thru 0x014c34 (0x6d02 bytes) 000273 1789132143.157067 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x37fb1): Mirror 2: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37fb1): Mirror 2: 0xd21b4d76 (stale) Mirror 6: 0xd21b4d76 (stale) Mirror 7: 0xd21b4d76 (stale) Mirror 8: 0xd21b4d76 (stale) CRC-32 checksum value for chunk [0, 0x37fb1): Mirror 1: 0x69867827 Mirror 9: 0x69867827 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37fb1: skipped 000274 1789132144.424161 trunc from 0x037fb1 to 0x01c6ae (0x1b902 bytes) 000275 1789132144.499786 punch from 0x010618 to 0x017fac (0x7995 bytes) 000276 1789132144.505233 fallocate 0x039c14 thru 0x03ffff (0x63ec bytes) 000277 1789132144.516709 MIRROR_RESYNC 6 mirrors 000279 1789132146.433004 trunc from 0x00321b to 0x00d870 (0xa656 bytes) 000281 1789132146.518712 trunc from 0x01c6af to 0x02456c (0x7ebe bytes) 000282 1789132146.559957 trunc from 0x02456d to 0x011198 (0x133d4 bytes) 000283 1789132146.576678 trunc from 0x011199 to 0x007c09 (0x958f bytes) 000284 1789132146.596649 mapread 0x004ad9 thru 0x007c09 (0x3131 bytes) 000286 1789132146.625519 fallocate 0x01b34b thru 0x024ab4 (0x976a bytes) 000287 1789132146.631516 mapread 0x00e292 thru 0x0195ec (0xb35b bytes) 000288 1789132146.649460 punch from 0x011a8c to 0x01a34f (0x88c4 bytes) 000289 1789132146.654896 trunc from 0x024ab5 to 0x005c28 (0x1ee8c bytes) 000290 1789132146.686939 trunc from 0x005c29 to 0x00b5ca (0x59a2 bytes) 000291 1789132146.706080 punch from 0x0071db to 0x00b5ca (0x43f0 bytes) 000292 1789132146.710823 punch from 0x004557 to 0x00b5ca (0x7074 bytes) 000293 1789132146.714946 trunc from 0x006a81 to 0x00b5ca (0x4b4a bytes) 000294 1789132146.720787 trunc from 0x000378 to 0x00b5ca (0xb253 bytes) 000295 1789132146.726005 fallocate 0x02741f thru 0x027901 (0x4e3 bytes) 000296 1789132146.729928 read 0x004cdc thru 0x0098b3 (0x4bd8 bytes) 000297 1789132146.740222 trunc from 0x00b5cb to 0x0041e6 (0x73e4 bytes) 000298 1789132146.761678 punch from 0x00178d to 0x0041e6 (0x2a5a bytes) 000299 1789132146.769073 write 0x01e412 thru 0x02ab2d (0xc71c bytes) 000300 1789132146.779743 trunc from 0x021ad1 to 0x02ab2d (0x905d bytes) 000301 1789132146.799117 mapwrite 0x00bf0d thru 0x018c87 (0xcd7b bytes) 000302 1789132146.883121 mapread 0x004e9c thru 0x00efe2 (0xa147 bytes) 000303 1789132146.901021 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ab2e): Mirror 2: 0xff0e6b44 (stale) Mirror 6: 0xff0e6b44 (stale) Mirror 7: 0xff0e6b44 (stale) Mirror 9: 0xff0e6b44 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ab2e): Mirror 2: 0xff0e6b44 (stale) Mirror 6: 0xff0e6b44 (stale) Mirror 7: 0xff0e6b44 (stale) Mirror 9: 0xff0e6b44 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ab2e: skipped 000304 1789132147.683132 write 0x00cbb0 thru 0x011b0f (0x4f60 bytes) 000305 1789132147.728011 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ab2e): Mirror 6: 0xff0e6b44 (stale) Mirror 7: 0xff0e6b44 (stale) Mirror 9: 0xff0e6b44 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ab2e): Mirror 6: 0xff0e6b44 (stale) Mirror 7: 0xff0e6b44 (stale) Mirror 9: 0xff0e6b44 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ab2e: skipped 000306 1789132148.518574 trunc from 0x00a266 to 0x00dd68 (0x3b03 bytes) 000307 1789132148.528852 trunc from 0x02ab2e to 0x005131 (0x259fc bytes) 000308 1789132148.587874 punch from 0x004e6e to 0x005131 (0x2c4 bytes) 000309 1789132148.598646 mapread 0x0026e3 thru 0x005131 (0x2a4f bytes) 000310 1789132148.650825 read 0x001307 thru 0x005131 (0x3e2b bytes) 000311 1789132148.651578 write 0x01a3ae thru 0x025bd8 (0xb82b bytes) 000312 1789132148.656016 punch from 0x01aff0 to 0x025bd8 (0xabe9 bytes) 000313 1789132148.686937 read 0x00003f thru 0x007721 (0x76e3 bytes) 000314 1789132148.716758 trunc from 0x025bd9 to 0x03d98d (0x17db5 bytes) 000315 1789132148.766581 fallocate 0x02d0ac thru 0x03c68f (0xf5e4 bytes) 000316 1789132148.774425 write 0x023de3 thru 0x02e03b (0xa259 bytes) 000317 1789132148.813319 fallocate 0x011e53 thru 0x01620b (0x43b9 bytes) 000318 1789132148.823219 mapwrite 0x03f8fd thru 0x03ffff (0x703 bytes) 000320 1789132148.936535 write 0x01eedf thru 0x02de13 (0xef35 bytes) 000321 1789132148.938637 punch from 0x00b8cb to 0x00e4b3 (0x2be9 bytes) 000322 1789132148.954727 trunc from 0x040000 to 0x01b15f (0x24ea0 bytes) 000323 1789132148.975573 trunc from 0x00a39c to 0x0156c9 (0xb32e bytes) 000324 1789132148.985989 punch from 0x016739 to 0x0199bf (0x3287 bytes) 000325 1789132149.005637 fallocate 0x003773 thru 0x010232 (0xcac0 bytes) 000326 1789132149.027964 read 0x019f7c thru 0x01b15f (0x11e4 bytes) 000328 1789132149.069492 write 0x00b08c thru 0x010a56 (0x59cb bytes) 000329 1789132149.070800 punch from 0x00337c to 0x00f573 (0xc1f8 bytes) 000330 1789132149.105090 trunc from 0x013f31 to 0x018c4f (0x4d1f bytes) 000331 1789132149.120137 trunc from 0x0143f8 to 0x01b15f (0x6d68 bytes) 000332 1789132149.131362 punch from 0x0134c0 to 0x015ccb (0x280c bytes) 000333 1789132149.140475 write 0x006b39 thru 0x00d5cd (0x6a95 bytes) 000334 1789132149.166023 fallocate 0x01242e thru 0x018011 (0x5be4 bytes) 000335 1789132149.179916 mapwrite 0x032166 thru 0x034e06 (0x2ca1 bytes) 000337 1789132149.305153 mapread 0x0273b3 thru 0x02e4a2 (0x70f0 bytes) 000338 1789132149.306039 fallocate 0x00e2d7 thru 0x01af10 (0xcc3a bytes) 000339 1789132149.312876 mapread 0x008d08 thru 0x011fac (0x92a5 bytes) 000340 1789132149.342836 punch from 0x002a84 to 0x010d26 (0xe2a3 bytes) 000341 1789132149.365310 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34e07): Mirror 7: 0x356afedb (stale) Mirror 9: 0x356afedb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34e07): Mirror 7: 0x356afedb (stale) Mirror 9: 0x356afedb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34e07: skipped 000342 1789132149.922138 trunc from 0x034e07 to 0x02b8cb (0x953b bytes) 000343 1789132149.947145 fallocate 0x038e18 thru 0x03ffff (0x71e8 bytes) 000344 1789132149.953816 trunc from 0x0304f9 to 0x03f2cb (0xedd3 bytes) 000345 1789132149.960436 mapread 0x02ea53 thru 0x03c247 (0xd7f5 bytes) 000347 1789132150.015938 fallocate 0x00594d thru 0x012f55 (0xd609 bytes) 000348 1789132150.021645 mapwrite 0x01e930 thru 0x02bc02 (0xd2d3 bytes) 000350 1789132150.068349 trunc from 0x040000 to 0x006d89 (0x39276 bytes) 000351 1789132150.093547 fallocate 0x016a6e thru 0x025d54 (0xf2e7 bytes) 000352 1789132150.100956 read 0x0123ff thru 0x018d21 (0x6923 bytes) 000353 1789132150.112642 punch from 0x0026a0 to 0x006031 (0x3992 bytes) 000354 1789132150.118258 fallocate 0x007eaa thru 0x015e60 (0xdfb7 bytes) 000355 1789132150.125689 write 0x003d8e thru 0x01335a (0xf5cd bytes) 000356 1789132150.134459 trunc from 0x021c47 to 0x0241b9 (0x2573 bytes) 000357 1789132150.139479 fallocate 0x039c54 thru 0x03dbda (0x3f87 bytes) 000358 1789132150.155089 MIRROR_RESYNC 3 mirrors 000359 1789132151.125033 trunc from 0x01b2fc to 0x02460b (0x9310 bytes) 000360 1789132151.165094 punch from 0x00ff74 to 0x01b6e6 (0xb773 bytes) 000361 1789132151.170943 write 0x003cbb thru 0x00d455 (0x979b bytes) 000362 1789132151.181120 mapread 0x00972d thru 0x01809b (0xe96f bytes) 000363 1789132151.202135 trunc from 0x03b9b3 to 0x03dbda (0x2228 bytes) 000364 1789132151.207395 trunc from 0x03dbdb to 0x0045a5 (0x39635 bytes) 000365 1789132151.248907 trunc from 0x000afa to 0x0045a5 (0x3aac bytes) 000366 1789132151.260802 mapread 0x003839 thru 0x0045a5 (0xd6d bytes) 000368 1789132151.273877 punch from 0x0044eb to 0x0045a5 (0xbb bytes) 000369 1789132151.282160 trunc from 0x0045a6 to 0x01e519 (0x19f74 bytes) 000370 1789132151.302652 punch from 0x019c8a to 0x01e519 (0x4890 bytes) 000371 1789132151.307944 mapread 0x0154a7 thru 0x01e519 (0x9073 bytes) 000373 1789132151.324928 write 0x039698 thru 0x03ffff (0x6968 bytes) 000374 1789132151.327099 trunc from 0x02ebcc to 0x03c569 (0xd99e bytes) 000375 1789132151.352041 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x54750cc (stale) Mirror 9: 0x54750cc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfca53392 Mirror 10: 0xfca53392 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000377 1789132152.421015 mapread 0x03e7f5 thru 0x03ffff (0x180b bytes) 000378 1789132152.445516 mapwrite 0x0373ed thru 0x03ffff (0x8c13 bytes) 000379 1789132152.589723 fallocate 0x039df8 thru 0x03cde6 (0x2fef bytes) 000380 1789132152.614898 trunc from 0x040000 to 0x027c2f (0x183d0 bytes) 000381 1789132152.720903 read 0x027573 thru 0x027c2f (0x6bd bytes) 000382 1789132152.729000 mapwrite 0x03767f thru 0x03ffff (0x8981 bytes) 000383 1789132152.796409 write 0x03f1c4 thru 0x03ffff (0xe3c bytes) 000384 1789132152.797402 read 0x01a426 thru 0x01f6b1 (0x528c bytes) 000386 1789132152.814895 mapread 0x00d144 thru 0x00fd89 (0x2c46 bytes) 000387 1789132152.816931 mapwrite 0x004ab6 thru 0x00f487 (0xa9d2 bytes) 000388 1789132152.870368 write 0x036347 thru 0x03e361 (0x801b bytes) 000389 1789132152.874478 trunc from 0x040000 to 0x02d398 (0x12c67 bytes) 000390 1789132152.912379 trunc from 0x02d399 to 0x0256f4 (0x7ca4 bytes) 000391 1789132152.941542 trunc from 0x01fbe0 to 0x0256f4 (0x5b15 bytes) 000392 1789132152.949502 mapread 0x00d05d thru 0x01a88d (0xd831 bytes) 000393 1789132152.952502 read 0x001f76 thru 0x00731d (0x53a8 bytes) 000394 1789132152.954164 trunc from 0x01b24c to 0x01e50b (0x32c0 bytes) 000395 1789132152.966107 trunc from 0x0256f5 to 0x027ead (0x27b9 bytes) 000396 1789132152.994429 mapread 0x01eedc thru 0x027ead (0x8fd2 bytes) 000397 1789132153.013473 mapwrite 0x03198a thru 0x03690e (0x4f85 bytes) 000398 1789132153.085923 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3690f): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3690f): Mirror 7: 0xb0f618bc (stale) Mirror 9: 0xb0f618bc (stale) Mirror 10: 0x3acef7c (stale) CRC-32 checksum value for chunk [0, 0x3690f): Mirror 1: 0xc1f1396f Mirror 11: 0xc1f1396f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3690f: skipped 000399 1789132154.143305 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3690f): Mirror 9: 0xb0f618bc (stale) Mirror 10: 0x3acef7c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3690f): Mirror 9: 0xb0f618bc (stale) Mirror 10: 0x3acef7c (stale) CRC-32 checksum value for chunk [0, 0x3690f): Mirror 1: 0xc1f1396f Mirror 11: 0xc1f1396f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3690f: skipped 000400 1789132154.732660 trunc from 0x03690f to 0x027920 (0xefee bytes) 000401 1789132154.844971 punch from 0x00c043 to 0x015cdb (0x9c99 bytes) 000402 1789132154.852057 read 0x00fbf4 thru 0x01ecc2 (0xf0cf bytes) 000403 1789132154.865837 trunc from 0x00b2a8 to 0x0170b9 (0xbe12 bytes) 000404 1789132154.878891 fallocate 0x0248e0 thru 0x02ce7a (0x859b bytes) 000405 1789132154.886565 fallocate 0x02551e thru 0x030c3e (0xb721 bytes) 000406 1789132154.892887 mapwrite 0x035333 thru 0x03ffff (0xaccd bytes) 000407 1789132154.969946 trunc from 0x008fe6 to 0x010b14 (0x7b2f bytes) 000408 1789132154.978617 fallocate 0x027b52 thru 0x035044 (0xd4f3 bytes) 000410 1789132154.985952 mapwrite 0x0047fc thru 0x00712e (0x2933 bytes) 000411 1789132155.012126 trunc from 0x006296 to 0x00b7b1 (0x551c bytes) 000412 1789132155.016969 write 0x018a1f thru 0x018f95 (0x577 bytes) 000413 1789132155.018104 trunc from 0x040000 to 0x02f7de (0x10821 bytes) 000414 1789132155.037577 MIRROR_RESYNC 4 mirrors 000415 1789132156.334390 mapwrite 0x02054a thru 0x02a054 (0x9b0b bytes) 000417 1789132156.446702 mapwrite 0x0120df thru 0x01ecde (0xcc00 bytes) 000418 1789132156.478852 mapwrite 0x02c02f thru 0x03033c (0x430e bytes) 000420 1789132156.529127 mapread 0x01a9cb thru 0x021c65 (0x729b bytes) 000421 1789132156.530343 punch from 0x005473 to 0x00b298 (0x5e26 bytes) 000422 1789132156.536727 write 0x02e652 thru 0x0367a1 (0x8150 bytes) 000423 1789132156.539782 punch from 0x00155c to 0x00b338 (0x9ddd bytes) 000424 1789132156.549407 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x367a2): Mirror 10: 0x49ea8ef1 (stale) Mirror 11: 0x49ea8ef1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x367a2): Mirror 10: 0x49ea8ef1 (stale) Mirror 11: 0x49ea8ef1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x367a2: skipped 000425 1789132156.907994 fallocate 0x00fc07 thru 0x017db4 (0x81ae bytes) 000426 1789132156.919931 fallocate 0x029227 thru 0x038dea (0xfbc4 bytes) 000427 1789132156.935513 write 0x00fda9 thru 0x01b83b (0xba93 bytes) 000428 1789132156.962536 mapwrite 0x0103ca thru 0x01fae2 (0xf719 bytes) 000429 1789132157.035121 trunc from 0x0367a2 to 0x023813 (0x12f8e bytes) 000430 1789132157.069952 write 0x037ed8 thru 0x038878 (0x9a1 bytes) 000431 1789132157.072484 trunc from 0x038879 to 0x003779 (0x350ff bytes) 000432 1789132157.105333 MIRROR_RESYNC 3 mirrors 000433 1789132157.780828 MIRROR_RESYNC 3 mirrors 000434 1789132158.129846 trunc from 0x00377a to 0x022192 (0x1ea19 bytes) 000435 1789132158.197936 MIRROR_RESYNC 3 mirrors 000436 1789132158.978676 read 0x00a85d thru 0x012a61 (0x8205 bytes) 000437 1789132158.995163 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) 000438 1789132159.493393 trunc from 0x00371a to 0x004929 (0x1210 bytes) 000439 1789132159.577708 trunc from 0x022193 to 0x000fa6 (0x211ec bytes) 000440 1789132159.655790 mapwrite 0x02622b thru 0x02da81 (0x7857 bytes) 000441 1789132159.795696 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2da82): Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2da82): Mirror 11: 0xc43795c3 (stale) CRC-32 checksum value for chunk [0, 0x2da82): Mirror 1: 0xa6ba80e3 Mirror 12: 0xa6ba80e3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2da82: skipped 000442 1789132161.020468 MIRROR_RESYNC 3 mirrors 000443 1789132161.694668 mapread 0x01fd4d thru 0x026fb0 (0x7264 bytes) 000444 1789132161.729703 mapwrite 0x03bd25 thru 0x03f9f4 (0x3cd0 bytes) 000446 1789132161.800471 punch from 0x02162a to 0x02b5e3 (0x9fba bytes) 000447 1789132161.806643 punch from 0x01b3d6 to 0x01ef98 (0x3bc3 bytes) 000448 1789132161.810890 trunc from 0x0007a2 to 0x002e0a (0x2669 bytes) 000449 1789132161.816871 fallocate 0x033c80 thru 0x03b85d (0x7bde bytes) 000450 1789132161.822494 fallocate 0x03733c thru 0x03ffff (0x8cc4 bytes) 000452 1789132161.827899 write 0x01803c thru 0x01e0c2 (0x6087 bytes) 000453 1789132161.836571 trunc from 0x03f9f5 to 0x0372d8 (0x871c bytes) 000454 1789132161.853416 read 0x00365b thru 0x00ac41 (0x75e7 bytes) 000455 1789132161.859392 punch from 0x030532 to 0x0310b9 (0xb88 bytes) 000456 1789132161.865764 punch from 0x0022be to 0x00a332 (0x8075 bytes) 000457 1789132161.881144 MIRROR_RESYNC 3 mirrors 000458 1789132163.075144 MIRROR_RESYNC 3 mirrors 000459 1789132163.440698 read 0x00cf86 thru 0x00fb59 (0x2bd4 bytes) 000460 1789132163.447122 read 0x00e0a7 thru 0x01cb60 (0xeaba bytes) 000461 1789132163.459108 mapread 0x0204cd thru 0x028ba0 (0x86d4 bytes) 000462 1789132163.460009 mapwrite 0x01c866 thru 0x022251 (0x59ec bytes) 000463 1789132163.578267 mapwrite 0x030cc8 thru 0x03713c (0x6475 bytes) 000464 1789132163.655211 fallocate 0x021da6 thru 0x02be95 (0xa0f0 bytes) 000465 1789132163.662836 mapread 0x02d669 thru 0x0372d8 (0x9c70 bytes) 000466 1789132163.712474 trunc from 0x003621 to 0x00dd83 (0xa763 bytes) 000468 1789132163.719896 read 0x029235 thru 0x030501 (0x72cd bytes) 000469 1789132163.745226 mapread 0x00ec0d thru 0x01c608 (0xd9fc bytes) 000470 1789132163.746394 trunc from 0x0372d9 to 0x00d233 (0x2a0a5 bytes) 000471 1789132163.798637 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd234): Mirror 12: 0x6ce6393e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd234): Mirror 12: 0x6ce6393e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd234: skipped 000472 1789132164.116038 fallocate 0x00a350 thru 0x0141a0 (0x9e51 bytes) 000473 1789132164.121236 mapread 0x00340b thru 0x005d12 (0x2908 bytes) 000474 1789132164.139914 mapwrite 0x014c31 thru 0x0247a0 (0xfb70 bytes) 000475 1789132164.227988 mapread 0x0187cd thru 0x01d6fe (0x4f32 bytes) 000476 1789132164.230783 trunc from 0x0247a1 to 0x01eee5 (0x58bb bytes) 000477 1789132164.254524 fallocate 0x03c51e thru 0x03ffff (0x3ae2 bytes) 000478 1789132164.263128 fallocate 0x02a489 thru 0x02de71 (0x39e9 bytes) 000479 1789132164.272711 write 0x011900 thru 0x01d3d7 (0xbad8 bytes) 000480 1789132164.277708 trunc from 0x040000 to 0x027300 (0x18cff bytes) 000481 1789132164.322470 mapread 0x016312 thru 0x01c773 (0x6462 bytes) 000482 1789132164.326431 mapwrite 0x00acf1 thru 0x00bfd4 (0x12e4 bytes) 000483 1789132164.380670 write 0x028fd9 thru 0x033d31 (0xad59 bytes) 000484 1789132164.384761 read 0x01b4e6 thru 0x0291da (0xdcf5 bytes) 000485 1789132164.400286 trunc from 0x033d32 to 0x02979b (0xa596 bytes) 000486 1789132164.435622 punch from 0x009187 to 0x00b92d (0x27a7 bytes) 000487 1789132164.440904 trunc from 0x010bfa to 0x01de54 (0xd25b bytes) 000488 1789132164.447998 mapread 0x0286f9 thru 0x02979b (0x10a3 bytes) 000489 1789132164.449274 read 0x01fae7 thru 0x024b31 (0x504b bytes) 000490 1789132164.450609 fallocate 0x02f60c thru 0x03111a (0x1b0f bytes) 000491 1789132164.461471 trunc from 0x0141fe to 0x01ec3b (0xaa3e bytes) 000492 1789132164.472764 read 0x01212e thru 0x017199 (0x506c bytes) 000493 1789132164.483943 trunc from 0x02979c to 0x02cc40 (0x34a5 bytes) 000494 1789132164.530231 fallocate 0x0311b1 thru 0x03ffff (0xee4f bytes) 000495 1789132164.542641 trunc from 0x02cc41 to 0x02b037 (0x1c09 bytes) 000496 1789132164.574927 trunc from 0x00bcab to 0x00ca90 (0xde6 bytes) 000497 1789132164.581575 trunc from 0x02b038 to 0x001805 (0x29832 bytes) 000499 1789132164.608937 mapwrite 0x01c8c7 thru 0x0254d9 (0x8c13 bytes) 000500 1789132164.748854 write 0x004394 thru 0x00b35c (0x6fc9 bytes) 000501 1789132164.760943 read 0x0012cd thru 0x008d64 (0x7a98 bytes) 000502 1789132164.773658 read 0x012138 thru 0x0181e8 (0x60b1 bytes) 000503 1789132164.774462 write 0x00db4d thru 0x018e2d (0xb2e1 bytes) 000504 1789132164.776070 trunc from 0x0254da to 0x0051b5 (0x20324 bytes) 000505 1789132164.800527 mapread 0x004908 thru 0x0051b5 (0x8ae bytes) 000506 1789132164.801565 mapwrite 0x009b49 thru 0x0177ee (0xdca6 bytes) 000507 1789132164.858765 write 0x03dd89 thru 0x03ffff (0x2277 bytes) 000508 1789132164.860438 fallocate 0x0335ab thru 0x035533 (0x1f89 bytes) 000509 1789132164.867234 mapwrite 0x03ef9c thru 0x03ffff (0x1064 bytes) 000512 1789132164.898679 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xfeab8e78 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9948880b Mirror 13: 0x9948880b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000513 1789132165.982799 trunc from 0x00b644 to 0x017985 (0xc342 bytes) 000514 1789132166.094437 write 0x0389a1 thru 0x039937 (0xf97 bytes) 000515 1789132166.134772 punch from 0x033fa5 to 0x038ac9 (0x4b25 bytes) 000516 1789132166.163395 write 0x02609c thru 0x02efb5 (0x8f1a bytes) 000517 1789132166.240349 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xfeab8e78 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xfeab8e78 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000518 1789132166.592252 mapread 0x034b67 thru 0x03ffff (0xb499 bytes) 000520 1789132166.623745 trunc from 0x030568 to 0x038990 (0x8429 bytes) 000521 1789132166.629585 mapwrite 0x034549 thru 0x03ffff (0xbab7 bytes) 000523 1789132166.680028 punch from 0x01af61 to 0x02a38b (0xf42b bytes) 000524 1789132166.685559 mapread 0x005e0e thru 0x013cbc (0xdeaf bytes) 000525 1789132166.726446 punch from 0x01453d to 0x016152 (0x1c16 bytes) 000526 1789132166.734247 trunc from 0x009889 to 0x01112f (0x78a7 bytes) 000527 1789132166.763514 trunc from 0x040000 to 0x00fcc9 (0x30336 bytes) 000528 1789132166.866873 MIRROR_RESYNC 2 mirrors 000529 1789132167.920119 punch from 0x00efd7 to 0x00fcc9 (0xcf3 bytes) 000530 1789132168.020822 mapwrite 0x014520 thru 0x018958 (0x4439 bytes) 000531 1789132168.173923 trunc from 0x018959 to 0x02aa7b (0x12123 bytes) 000532 1789132168.210295 mapwrite 0x03de0e thru 0x03ffff (0x21f2 bytes) 000533 1789132168.299483 fallocate 0x018ce2 thru 0x021837 (0x8b56 bytes) 000534 1789132168.305470 mapread 0x01df43 thru 0x02dd0a (0xfdc8 bytes) 000535 1789132168.350457 trunc from 0x040000 to 0x0384fc (0x7b03 bytes) 000536 1789132168.375312 read 0x023a4f thru 0x024669 (0xc1b bytes) 000537 1789132168.376864 read 0x002eb2 thru 0x0115ac (0xe6fb bytes) 000538 1789132168.392700 mapwrite 0x0165ef thru 0x017cc6 (0x16d8 bytes) 000539 1789132168.407138 mapwrite 0x032bd1 thru 0x03ffff (0xd42f bytes) 000540 1789132168.490963 punch from 0x004a51 to 0x011cfd (0xd2ad bytes) 000541 1789132168.503615 trunc from 0x008525 to 0x011db3 (0x988f bytes) 000542 1789132168.513857 write 0x004695 thru 0x00eaba (0xa426 bytes) 000543 1789132168.532729 write 0x0089b6 thru 0x014787 (0xbdd2 bytes) 000544 1789132168.537615 fallocate 0x00d317 thru 0x012ffc (0x5ce6 bytes) 000545 1789132168.548461 read 0x03f7a7 thru 0x03ffff (0x859 bytes) 000546 1789132168.552868 write 0x02c9b5 thru 0x032a58 (0x60a4 bytes) 000547 1789132168.555075 punch from 0x03a6ec to 0x03ffff (0x5914 bytes) 000548 1789132168.564378 read 0x024ce3 thru 0x029299 (0x45b7 bytes) 000549 1789132168.566584 fallocate 0x02daca thru 0x03a835 (0xcd6c bytes) 000550 1789132168.578580 read 0x03c243 thru 0x03ffff (0x3dbd bytes) 000552 1789132168.601145 mapread 0x001b97 thru 0x007eda (0x6344 bytes) 000553 1789132168.605366 mapread 0x02d6d8 thru 0x0321bd (0x4ae6 bytes) 000554 1789132168.614336 write 0x009b93 thru 0x010285 (0x66f3 bytes) 000555 1789132168.615786 fallocate 0x011e27 thru 0x019a1f (0x7bf9 bytes) 000556 1789132168.627767 trunc from 0x040000 to 0x00ca13 (0x335ec bytes) 000557 1789132168.683839 write 0x0196bb thru 0x023d52 (0xa698 bytes) 000558 1789132168.689827 punch from 0x02049c to 0x020706 (0x26b bytes) 000559 1789132168.715251 MIRROR_RESYNC 2 mirrors 000560 1789132169.639922 fallocate 0x00428f thru 0x00aed8 (0x6c4a bytes) 000561 1789132169.685872 MIRROR_RESYNC 2 mirrors 000562 1789132170.687367 trunc from 0x023d53 to 0x010d76 (0x12fdc bytes) 000563 1789132170.741801 mapread 0x00a818 thru 0x00ca8d (0x2276 bytes) 000564 1789132170.777656 mapread 0x00f162 thru 0x010d76 (0x1c15 bytes) 000565 1789132170.786816 mapread 0x004e3a thru 0x010188 (0xb34f bytes) 000566 1789132170.805371 fallocate 0x007a7d thru 0x01086a (0x8dee bytes) 000567 1789132170.811745 fallocate 0x024230 thru 0x0245c0 (0x391 bytes) 000568 1789132170.822542 fallocate 0x00ea40 thru 0x01497e (0x5f3f bytes) 000569 1789132170.837094 mapread 0x006afc thru 0x010d76 (0xa27b bytes) 000570 1789132170.841988 write 0x01c85c thru 0x01f682 (0x2e27 bytes) 000571 1789132170.846100 trunc from 0x01f683 to 0x01bfa6 (0x36dc bytes) 000572 1789132170.888724 read 0x0179af thru 0x01964d (0x1c9f bytes) 000574 1789132170.897273 write 0x00ae1f thru 0x012df0 (0x7fd2 bytes) 000575 1789132170.910873 trunc from 0x009b4e to 0x0146a2 (0xab55 bytes) 000577 1789132170.945084 read 0x00ccb6 thru 0x0145ee (0x7939 bytes) 000578 1789132170.994325 mapread 0x012e77 thru 0x01a70c (0x7896 bytes) 000579 1789132170.995364 mapwrite 0x03134f thru 0x03b77a (0xa42c bytes) 000580 1789132171.117671 trunc from 0x03b77b to 0x013943 (0x27e37 bytes) 000581 1789132171.155904 punch from 0x0087b8 to 0x00c437 (0x3c80 bytes) 000582 1789132171.169280 mapread 0x010cf5 thru 0x0116a3 (0x9af bytes) 000583 1789132171.172626 mapwrite 0x009031 thru 0x00c95c (0x392c bytes) 000584 1789132171.279104 mapwrite 0x004171 thru 0x011831 (0xd6c1 bytes) 000585 1789132171.333098 trunc from 0x00d844 to 0x013943 (0x6100 bytes) 000586 1789132171.346143 mapread 0x00486c thru 0x010f6b (0xc700 bytes) 000587 1789132171.419387 mapread 0x0109be thru 0x013943 (0x2f86 bytes) 000588 1789132171.471310 mapwrite 0x0305b9 thru 0x039e8c (0x98d4 bytes) 000589 1789132171.652247 mapwrite 0x0292f7 thru 0x036967 (0xd671 bytes) 000590 1789132171.782755 mapwrite 0x0178ef thru 0x01e528 (0x6c3a bytes) 000591 1789132171.926100 fallocate 0x01d513 thru 0x022f29 (0x5a17 bytes) 000592 1789132171.947860 mapwrite 0x02e898 thru 0x03e59e (0xfd07 bytes) 000593 1789132172.064330 punch from 0x01e6f8 to 0x02007c (0x1985 bytes) 000594 1789132172.073479 fallocate 0x033390 thru 0x03ce86 (0x9af7 bytes) 000595 1789132172.085764 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) 000596 1789132172.434636 trunc from 0x03e59f to 0x03f2d3 (0xd35 bytes) 000597 1789132172.464026 write 0x012732 thru 0x01b46a (0x8d39 bytes) 000598 1789132172.478273 read 0x01c690 thru 0x023876 (0x71e7 bytes) 000599 1789132172.491142 write 0x00cc40 thru 0x018bd6 (0xbf97 bytes) 000600 1789132172.502366 write 0x00355d thru 0x012ed6 (0xf97a bytes) 000601 1789132172.513487 punch from 0x03f119 to 0x03f2d3 (0x1bb bytes) 000602 1789132172.519447 trunc from 0x03f2d4 to 0x00a66b (0x34c68 bytes) 000603 1789132172.541503 write 0x013431 thru 0x022a84 (0xf654 bytes) 000604 1789132172.545151 write 0x02c96b thru 0x0358b3 (0x8f49 bytes) 000605 1789132172.546868 trunc from 0x02b8a4 to 0x02ddec (0x2549 bytes) 000606 1789132172.566340 fallocate 0x028023 thru 0x034d25 (0xcd03 bytes) 000607 1789132172.571800 mapwrite 0x01a108 thru 0x029b1b (0xfa14 bytes) 000608 1789132172.615671 trunc from 0x0358b4 to 0x017e5e (0x1da55 bytes) 000609 1789132172.636763 punch from 0x000838 to 0x002ef7 (0x26c0 bytes) 000610 1789132172.644343 trunc from 0x017e5f to 0x03212f (0x1a2d1 bytes) 000611 1789132172.660329 mapread 0x023553 thru 0x025083 (0x1b31 bytes) 000612 1789132172.673971 write 0x01bddb thru 0x023202 (0x7428 bytes) 000613 1789132172.680632 fallocate 0x00e382 thru 0x016ee8 (0x8b67 bytes) 000614 1789132172.686353 write 0x001ad4 thru 0x00d14b (0xb678 bytes) 000615 1789132172.698982 punch from 0x016562 to 0x01978e (0x322d bytes) 000616 1789132172.705663 trunc from 0x032130 to 0x03df60 (0xbe31 bytes) 000618 1789132172.733488 write 0x03f23c thru 0x03ffff (0xdc4 bytes) 000621 1789132172.740162 mapread 0x019478 thru 0x023f2a (0xaab3 bytes) 000622 1789132172.756663 read 0x01a5f3 thru 0x0202e8 (0x5cf6 bytes) 000623 1789132172.757735 trunc from 0x02be11 to 0x03a375 (0xe565 bytes) 000625 1789132172.769133 trunc from 0x011c02 to 0x01c75d (0xab5c bytes) 000626 1789132172.797015 trunc from 0x040000 to 0x0353bd (0xac42 bytes) 000627 1789132172.817165 punch from 0x0084b9 to 0x00bc74 (0x37bc bytes) 000628 1789132172.827799 write 0x03bac8 thru 0x03ffff (0x4538 bytes) 000629 1789132172.832609 mapread 0x01010e thru 0x019667 (0x955a bytes) 000631 1789132172.850573 mapread 0x004570 thru 0x013fe9 (0xfa7a bytes) 000632 1789132172.852489 write 0x01e733 thru 0x02858b (0x9e59 bytes) 000633 1789132172.854294 read 0x030e43 thru 0x037f38 (0x70f6 bytes) 000634 1789132172.854929 read 0x010788 thru 0x01b3df (0xac58 bytes) 000635 1789132172.855558 punch from 0x0072e9 to 0x00a846 (0x355e bytes) 000636 1789132172.866304 MIRROR_EXTEND to 2 mirrors 000637 1789132174.155530 trunc from 0x040000 to 0x018fb9 (0x27046 bytes) 000638 1789132174.231749 write 0x00715d thru 0x007d18 (0xbbc bytes) 000639 1789132174.237619 read 0x00f0ae thru 0x018ed4 (0x9e27 bytes) 000640 1789132174.247531 fallocate 0x00f57f thru 0x0179f0 (0x8472 bytes) 000641 1789132174.256152 punch from 0x0009a4 to 0x00d01d (0xc67a bytes) 000642 1789132174.267611 trunc from 0x0119d2 to 0x018fb9 (0x75e8 bytes) 000643 1789132174.272523 trunc from 0x018fba to 0x00761d (0x1199c bytes) 000644 1789132174.305024 mapread 0x001939 thru 0x00221a (0x8e2 bytes) 000645 1789132174.321006 write 0x008596 thru 0x00954c (0xfb7 bytes) 000646 1789132174.327469 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) 000647 1789132174.472349 punch from 0x002265 to 0x006aa0 (0x483c bytes) 000648 1789132174.478485 trunc from 0x00954d to 0x01b900 (0x123b4 bytes) 000649 1789132174.499694 write 0x031f4a thru 0x034ea4 (0x2f5b bytes) 000654 1789132174.505642 mapwrite 0x02ecb0 thru 0x035a5c (0x6dad bytes) 000655 1789132174.566519 mapwrite 0x029f51 thru 0x039969 (0xfa19 bytes) 000656 1789132174.617615 write 0x01e7d1 thru 0x021598 (0x2dc8 bytes) 000657 1789132174.628719 punch from 0x0089c6 to 0x00d376 (0x49b1 bytes) 000658 1789132174.635504 fallocate 0x002706 thru 0x00767c (0x4f77 bytes) 000659 1789132174.641760 punch from 0x01b31d to 0x02037e (0x5062 bytes) 000660 1789132174.655091 punch from 0x033de2 to 0x039969 (0x5b88 bytes) 000661 1789132174.661362 write 0x013055 thru 0x01652c (0x34d8 bytes) 000662 1789132174.671753 fallocate 0x00f756 thru 0x01013d (0x9e8 bytes) 000663 1789132174.677868 write 0x029b22 thru 0x030387 (0x6866 bytes) 000664 1789132174.680082 trunc from 0x03996a to 0x010f8f (0x289da bytes) 000665 1789132174.699720 trunc from 0x00b30b to 0x00d0a9 (0x1d9f bytes) 000666 1789132174.705284 mapwrite 0x028ef9 thru 0x032f63 (0xa06b bytes) 000667 1789132174.792104 write 0x032dd8 thru 0x033d38 (0xf61 bytes) 000668 1789132174.793879 punch from 0x00492f to 0x013ea2 (0xf574 bytes) 000670 1789132174.806708 trunc from 0x00f325 to 0x018dce (0x9aaa bytes) 000671 1789132174.818726 mapread 0x016bf3 thru 0x0208b3 (0x9cc1 bytes) 000672 1789132174.839306 fallocate 0x029eed thru 0x02fe1e (0x5f32 bytes) 000673 1789132174.846021 trunc from 0x033d39 to 0x03754b (0x3813 bytes) 000674 1789132174.863255 read 0x008015 thru 0x00ea1b (0x6a07 bytes) 000676 1789132174.875551 mapread 0x020161 thru 0x022aff (0x299f bytes) 000677 1789132174.878919 trunc from 0x015467 to 0x01f197 (0x9d31 bytes) 000678 1789132174.887748 read 0x01cb64 thru 0x01e457 (0x18f4 bytes) 000679 1789132174.899995 trunc from 0x03754c to 0x02e460 (0x90eb bytes) 000680 1789132174.939571 write 0x03cf9e thru 0x03ffff (0x3062 bytes) 000681 1789132174.941077 write 0x01c483 thru 0x027961 (0xb4df bytes) 000683 1789132174.946544 fallocate 0x016e0a thru 0x01c155 (0x534c bytes) 000686 1789132174.956721 mapwrite 0x03fcf7 thru 0x03ffff (0x309 bytes) 000687 1789132174.997204 trunc from 0x006359 to 0x00eaf4 (0x879c bytes) 000688 1789132175.007081 write 0x00ecac thru 0x01ac2c (0xbf81 bytes) 000689 1789132175.019346 read 0x028b23 thru 0x02d952 (0x4e30 bytes) 000690 1789132175.020025 read 0x03d751 thru 0x03ffff (0x28af bytes) 000691 1789132175.020606 punch from 0x00b7f2 to 0x017b21 (0xc330 bytes) 000692 1789132175.037364 read 0x03e266 thru 0x03ffff (0x1d9a bytes) 000693 1789132175.039132 read 0x02cbc4 thru 0x0342ad (0x76ea bytes) 000694 1789132175.054754 read 0x003184 thru 0x00b63a (0x84b7 bytes) 000695 1789132175.056541 trunc from 0x021fad to 0x023c8a (0x1cde bytes) 000696 1789132175.063272 read 0x0316cf thru 0x0342aa (0x2bdc bytes) 000697 1789132175.064242 mapwrite 0x02361e thru 0x02e7f6 (0xb1d9 bytes) 000698 1789132175.097029 punch from 0x03f35e to 0x03ffff (0xca2 bytes) 000699 1789132175.103045 trunc from 0x0312a3 to 0x03c269 (0xafc7 bytes) 000701 1789132175.110601 mapwrite 0x0090c8 thru 0x009974 (0x8ad bytes) 000702 1789132175.125034 punch from 0x00ca9d to 0x0165e9 (0x9b4d bytes) 000703 1789132175.131044 fallocate 0x02c370 thru 0x03034b (0x3fdc bytes) 000704 1789132175.136715 read 0x03b8e3 thru 0x03ffff (0x471d bytes) 000705 1789132175.154479 read 0x018310 thru 0x01857c (0x26d bytes) 000706 1789132175.155580 trunc from 0x040000 to 0x01476d (0x2b892 bytes) 000707 1789132175.179766 fallocate 0x0114da thru 0x0141a9 (0x2cd0 bytes) 000708 1789132175.194956 write 0x00e230 thru 0x013a17 (0x57e8 bytes) 000709 1789132175.197713 mapwrite 0x0164b9 thru 0x018ed9 (0x2a21 bytes) 000710 1789132175.258498 write 0x0346a7 thru 0x03f824 (0xb17e bytes) 000711 1789132175.260346 mapread 0x01d65b thru 0x029c0f (0xc5b5 bytes) 000712 1789132175.278030 punch from 0x01b31d to 0x0234b8 (0x819c bytes) 000713 1789132175.286158 fallocate 0x00334f thru 0x005a75 (0x2727 bytes) 000714 1789132175.295431 trunc from 0x02c72e to 0x039478 (0xcd4b bytes) 000715 1789132175.325624 read 0x004cba thru 0x007ad8 (0x2e1f bytes) 000716 1789132175.333213 mapwrite 0x03f60b thru 0x03ffff (0x9f5 bytes) 000717 1789132175.385039 punch from 0x019c7e to 0x0275e3 (0xd966 bytes) 000718 1789132175.397584 read 0x02ae02 thru 0x03093b (0x5b3a bytes) 000719 1789132175.411524 read 0x02a9b9 thru 0x038bd3 (0xe21b bytes) 000721 1789132175.412866 trunc from 0x040000 to 0x02c9d9 (0x13626 bytes) 000722 1789132175.440504 read 0x0264c4 thru 0x02c9d9 (0x6516 bytes) 000723 1789132175.443372 trunc from 0x004f77 to 0x01272a (0xd7b4 bytes) 000724 1789132175.449610 write 0x0070f7 thru 0x011c6a (0xab74 bytes) 000726 1789132175.463627 fallocate 0x00b60a thru 0x01049d (0x4e94 bytes) 000727 1789132175.469795 mapwrite 0x01c416 thru 0x021ccd (0x58b8 bytes) 000728 1789132175.492743 punch from 0x02c1db to 0x02c9d9 (0x7ff bytes) 000729 1789132175.499324 punch from 0x00a2a9 to 0x016203 (0xbf5b bytes) 000731 1789132175.506734 read 0x027382 thru 0x02b952 (0x45d1 bytes) 000733 1789132175.508432 trunc from 0x02c9da to 0x00419b (0x2883e bytes) 000734 1789132175.530515 mapwrite 0x0216d3 thru 0x021a79 (0x3a7 bytes) 000735 1789132175.589235 trunc from 0x021a7a to 0x029f13 (0x849a bytes) 000737 1789132175.620359 write 0x01e4a2 thru 0x023535 (0x5094 bytes) 000738 1789132175.636540 mapread 0x028aef thru 0x029f13 (0x1425 bytes) 000739 1789132175.637749 punch from 0x00eba6 to 0x018428 (0x9883 bytes) 000741 1789132175.649035 fallocate 0x012833 thru 0x02223b (0xfa09 bytes) 000742 1789132175.660788 mapread 0x01f2e9 thru 0x0219db (0x26f3 bytes) 000743 1789132175.671227 fallocate 0x006245 thru 0x0132ec (0xd0a8 bytes) 000744 1789132175.683590 write 0x00526e thru 0x00be54 (0x6be7 bytes) 000745 1789132175.701719 punch from 0x00d12c to 0x01bfe4 (0xeeb9 bytes) 000747 1789132175.721799 mapread 0x026f1e thru 0x029f13 (0x2ff6 bytes) 000748 1789132175.726273 fallocate 0x000add thru 0x007b6c (0x7090 bytes) 000749 1789132175.745398 trunc from 0x01ec67 to 0x029f13 (0xb2ad bytes) 000750 1789132175.800500 read 0x01804e thru 0x026ac3 (0xea76 bytes) 000751 1789132175.823769 mapread 0x01561d thru 0x015d31 (0x715 bytes) 000752 1789132175.831074 MIRROR_EXTEND to 2 mirrors 000753 1789132176.841382 trunc from 0x01677c to 0x025ae7 (0xf36c bytes) 000754 1789132176.953742 trunc from 0x029f14 to 0x00c708 (0x1d80b bytes) 000755 1789132177.022506 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xc709): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc709): Mirror 2: 0x5553c5c (stale) CRC-32 checksum value for chunk [0, 0xc709): Mirror 1: 0x5553c5c Mirror 3: 0x5553c5c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc709: skipped 000756 1789132178.485130 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) 000757 1789132178.893934 read 0x00476f thru 0x00c708 (0x7f9a bytes) 000758 1789132178.909119 write 0x02cce9 thru 0x0360b4 (0x93cc bytes) 000759 1789132178.976408 fallocate 0x0108f2 thru 0x01cab2 (0xc1c1 bytes) 000760 1789132178.982947 mapread 0x02f6b6 thru 0x02f8da (0x225 bytes) 000761 1789132178.985865 trunc from 0x0074e8 to 0x00eb70 (0x7689 bytes) 000762 1789132178.993085 trunc from 0x0360b5 to 0x03d973 (0x78bf bytes) 000763 1789132179.028249 fallocate 0x008298 thru 0x00e357 (0x60c0 bytes) 000764 1789132179.035281 read 0x011dd0 thru 0x018c64 (0x6e95 bytes) 000765 1789132179.068578 MIRROR_RESYNC 2 mirrors 000766 1789132179.897570 write 0x02c491 thru 0x038723 (0xc293 bytes) 000768 1789132179.956994 punch from 0x0111ac to 0x0141e9 (0x303e bytes) 000769 1789132179.963522 mapread 0x01fccb thru 0x02a260 (0xa596 bytes) 000770 1789132179.995311 trunc from 0x03d974 to 0x032696 (0xb2dd bytes) 000771 1789132180.020817 mapread 0x000512 thru 0x004e75 (0x4964 bytes) 000772 1789132180.032552 punch from 0x026c5a to 0x028705 (0x1aac bytes) 000773 1789132180.038108 mapread 0x016881 thru 0x01c061 (0x57e1 bytes) 000774 1789132180.049008 trunc from 0x02a72e to 0x02e4f1 (0x3dc4 bytes) 000775 1789132180.066022 trunc from 0x032697 to 0x03bf14 (0x987e bytes) 000776 1789132180.083479 fallocate 0x035efe thru 0x036a63 (0xb66 bytes) 000778 1789132180.091027 punch from 0x018223 to 0x0254a1 (0xd27f bytes) 000779 1789132180.107047 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) 000780 1789132180.307065 mapwrite 0x02daf2 thru 0x0306b4 (0x2bc3 bytes) 000781 1789132180.355659 punch from 0x01183a to 0x018048 (0x680f bytes) 000782 1789132180.366471 fallocate 0x00a4c8 thru 0x00c095 (0x1bce bytes) 000783 1789132180.382939 mapwrite 0x01f7c4 thru 0x02212c (0x2969 bytes) 000784 1789132180.456861 mapwrite 0x010e05 thru 0x013cec (0x2ee8 bytes) 000785 1789132180.534912 trunc from 0x03bf15 to 0x03dfc9 (0x20b5 bytes) 000786 1789132180.577902 trunc from 0x014bb5 to 0x01a0eb (0x5537 bytes) 000788 1789132180.586540 mapread 0x0214a3 thru 0x0245df (0x313d bytes) 000789 1789132180.587582 fallocate 0x03b196 thru 0x03f23f (0x40aa bytes) 000791 1789132180.602079 write 0x02c122 thru 0x033729 (0x7608 bytes) 000792 1789132180.625130 MIRROR_EXTEND to 2 mirrors 000793 1789132181.889958 mapread 0x015bdf thru 0x019df0 (0x4212 bytes) 000794 1789132181.983332 MIRROR_RESYNC 2 mirrors 000796 1789132182.613099 mapwrite 0x0255dd thru 0x026ca6 (0x16ca bytes) 000797 1789132182.736771 trunc from 0x038dc3 to 0x03f23f (0x647d bytes) 000798 1789132182.744162 write 0x021993 thru 0x03134f (0xf9bd bytes) 000799 1789132182.790877 mapwrite 0x00b6e7 thru 0x00d3af (0x1cc9 bytes) 000800 1789132182.835039 mapwrite 0x0254c6 thru 0x0312a7 (0xbde2 bytes) 000801 1789132182.864504 trunc from 0x0352e6 to 0x03f23f (0x9f5a bytes) 000802 1789132182.871469 trunc from 0x007ca3 to 0x01502f (0xd38d bytes) 000803 1789132182.877529 mapwrite 0x0350ff thru 0x03695c (0x185e bytes) 000804 1789132182.902295 trunc from 0x03f240 to 0x018444 (0x26dfb bytes) 000805 1789132182.930498 mapwrite 0x031a64 thru 0x035552 (0x3aef bytes) 000806 1789132182.989466 punch from 0x0033e7 to 0x008daa (0x59c4 bytes) 000807 1789132182.995342 mapwrite 0x01ccd6 thru 0x01cfa4 (0x2cf bytes) 000808 1789132183.020665 mapwrite 0x03add9 thru 0x03ffff (0x5227 bytes) 000809 1789132183.093532 mapwrite 0x02f765 thru 0x02fe1f (0x6bb bytes) 000810 1789132183.119744 fallocate 0x01f6a9 thru 0x02f680 (0xffd8 bytes) 000811 1789132183.127470 punch from 0x0088e5 to 0x00eeae (0x65ca bytes) 000813 1789132183.136533 trunc from 0x040000 to 0x037e23 (0x81dc bytes) 000814 1789132183.160735 punch from 0x0303ea to 0x0372ec (0x6f03 bytes) 000816 1789132183.169134 write 0x032d52 thru 0x03ffff (0xd2ae bytes) 000817 1789132183.178221 write 0x025990 thru 0x028587 (0x2bf8 bytes) 000818 1789132183.188271 mapwrite 0x025a41 thru 0x03566e (0xfc2e bytes) 000819 1789132183.217425 trunc from 0x040000 to 0x0352ed (0xad12 bytes) 000821 1789132183.246245 mapread 0x034a67 thru 0x0352ed (0x887 bytes) 000822 1789132183.254435 trunc from 0x0352ee to 0x03433a (0xfb3 bytes) 000823 1789132183.279218 trunc from 0x03433b to 0x014766 (0x1fbd4 bytes) 000824 1789132183.343230 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) 000826 1789132183.593772 fallocate 0x0278f3 thru 0x02af1f (0x362d bytes) 000827 1789132183.606206 fallocate 0x020d54 thru 0x030298 (0xf545 bytes) 000828 1789132183.613467 read 0x005f69 thru 0x014d35 (0xedcd bytes) 000829 1789132183.632205 trunc from 0x00bcb7 to 0x01aa21 (0xed6b bytes) 000830 1789132183.640384 mapwrite 0x028c3f thru 0x02c4b5 (0x3877 bytes) 000831 1789132183.734325 write 0x018fdd thru 0x0232be (0xa2e2 bytes) 000832 1789132183.752493 write 0x02322f thru 0x0281ef (0x4fc1 bytes) 000833 1789132183.753448 trunc from 0x00fc85 to 0x01bac3 (0xbe3f bytes) 000834 1789132183.774103 read 0x026682 thru 0x02c4b5 (0x5e34 bytes) 000835 1789132183.791645 MIRROR_EXTEND to 2 mirrors 000836 1789132184.668290 trunc from 0x02c4b6 to 0x00ff99 (0x1c51c bytes) 000837 1789132184.749524 mapread 0x007029 thru 0x00ff99 (0x8f71 bytes) 000838 1789132184.778939 mapwrite 0x01c81d thru 0x028c8d (0xc471 bytes) 000839 1789132184.855294 trunc from 0x028c8e to 0x0251db (0x3ab2 bytes) 000840 1789132184.890420 mapread 0x00eabf thru 0x018092 (0x95d4 bytes) 000841 1789132184.905447 fallocate 0x0271b0 thru 0x02de8f (0x6ce0 bytes) 000842 1789132184.917320 mapread 0x003480 thru 0x004b90 (0x1711 bytes) 000843 1789132184.919840 write 0x004153 thru 0x00ac8f (0x6b3d bytes) 000844 1789132184.923375 trunc from 0x02de90 to 0x028efb (0x4f94 bytes) 000845 1789132184.959089 mapread 0x024f75 thru 0x028efb (0x3f87 bytes) 000846 1789132184.973476 read 0x00e67f thru 0x016d54 (0x86d6 bytes) 000847 1789132184.974038 trunc from 0x028efc to 0x03fb1b (0x16c20 bytes) 000848 1789132185.007215 read 0x01c197 thru 0x01ec55 (0x2abf bytes) 000849 1789132185.009089 read 0x002bcb thru 0x01019c (0xd5d2 bytes) 000850 1789132185.012528 trunc from 0x00531a to 0x0092d4 (0x3fbb bytes) 000851 1789132185.035889 punch from 0x010b75 to 0x0182b5 (0x7741 bytes) 000852 1789132185.048442 trunc from 0x03fb1c to 0x009608 (0x36513 bytes) 000853 1789132185.072249 punch from 0x007a8d to 0x009608 (0x1b7c bytes) 000854 1789132185.078006 trunc from 0x009609 to 0x024778 (0x1b170 bytes) 000855 1789132185.090685 mapread 0x010233 thru 0x014896 (0x4664 bytes) 000857 1789132185.108119 fallocate 0x0141c4 thru 0x01b22f (0x706c bytes) 000858 1789132185.114833 fallocate 0x000475 thru 0x004bef (0x477b bytes) 000859 1789132185.123932 fallocate 0x028c38 thru 0x03176b (0x8b34 bytes) 000860 1789132185.133295 punch from 0x01d131 to 0x02b8ea (0xe7ba bytes) 000861 1789132185.139623 mapwrite 0x0101b8 thru 0x0124a5 (0x22ee bytes) 000862 1789132185.154230 read 0x005792 thru 0x0074d6 (0x1d45 bytes) 000863 1789132185.161122 MIRROR_RESYNC 2 mirrors 000864 1789132186.322956 MIRROR_EXTEND to 3 mirrors 000865 1789132187.434663 trunc from 0x016c46 to 0x01af7f (0x433a bytes) 000866 1789132187.475643 fallocate 0x0012a6 thru 0x003b62 (0x28bd bytes) 000867 1789132187.481969 write 0x028e2f thru 0x02aa53 (0x1c25 bytes) 000871 1789132187.503795 trunc from 0x0030f0 to 0x00a1ff (0x7110 bytes) 000872 1789132187.511735 mapwrite 0x00b150 thru 0x013f45 (0x8df6 bytes) 000873 1789132187.584130 fallocate 0x000497 thru 0x00c11f (0xbc89 bytes) 000874 1789132187.596371 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3176c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3176c): Mirror 2: 0xd509b593 (stale) Mirror 3: 0xd509b593 (stale) CRC-32 checksum value for chunk [0, 0x3176c): Mirror 1: 0x5d20a2ab Mirror 4: 0x5d20a2ab lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3176c: skipped 000875 1789132189.168680 read 0x02b4d3 thru 0x02cede (0x1a0c bytes) 000876 1789132189.174549 mapwrite 0x00f5a2 thru 0x012c96 (0x36f5 bytes) 000877 1789132189.344123 trunc from 0x03176c to 0x026368 (0xb403 bytes) 000878 1789132189.374032 fallocate 0x02562d thru 0x02f711 (0xa0e5 bytes) 000879 1789132189.405491 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f712): Mirror 2: 0x612f132d (stale) Mirror 4: 0xd5792f3b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f712): Mirror 2: 0x612f132d (stale) Mirror 4: 0xd5792f3b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f712: skipped 000880 1789132190.098466 write 0x03a01a thru 0x03ec07 (0x4bee bytes) 000881 1789132190.107061 fallocate 0x015b4f thru 0x01d1e0 (0x7692 bytes) 000882 1789132190.112016 read 0x039ded thru 0x03ec07 (0x4e1b bytes) 000883 1789132190.119196 read 0x025a66 thru 0x0353e9 (0xf984 bytes) 000884 1789132190.133928 punch from 0x00c14a to 0x00c554 (0x40b bytes) 000886 1789132190.143723 trunc from 0x03ec08 to 0x019b1b (0x250ec bytes) 000888 1789132190.172627 trunc from 0x00d5a6 to 0x00d5f9 (0x54 bytes) 000889 1789132190.181224 trunc from 0x019b1c to 0x003c5a (0x15ec1 bytes) 000890 1789132190.206999 punch from 0x001d3b to 0x003c5a (0x1f20 bytes) 000891 1789132190.213834 trunc from 0x001e0a to 0x003c5a (0x1e51 bytes) 000892 1789132190.229567 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3c5b): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c5b): Mirror 2: 0xdc2968e8 (stale) Mirror 4: 0xdc2968e8 (stale) CRC-32 checksum value for chunk [0, 0x3c5b): Mirror 1: 0xdc2968e8 Mirror 5: 0xdc2968e8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c5b: skipped 000893 1789132191.273650 fallocate 0x017d13 thru 0x019530 (0x181e bytes) 000895 1789132191.370233 trunc from 0x002bee to 0x00618b (0x359e bytes) 000896 1789132191.397262 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19531): Mirror 4: 0x54309fd6 (stale) Mirror 5: 0xe746015a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19531): Mirror 4: 0x54309fd6 (stale) Mirror 5: 0xe746015a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19531: skipped 000897 1789132191.855126 mapwrite 0x037087 thru 0x03d2db (0x6255 bytes) 000899 1789132191.991305 punch from 0x022e88 to 0x027408 (0x4581 bytes) 000900 1789132192.003200 punch from 0x00aeb9 to 0x00e204 (0x334c bytes) 000901 1789132192.022843 trunc from 0x005a96 to 0x00fd94 (0xa2ff bytes) 000902 1789132192.039756 trunc from 0x032f55 to 0x0369f9 (0x3aa5 bytes) 000904 1789132192.071286 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d2dc): Mirror 4: 0xa98f6a8a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d2dc): Mirror 4: 0xa98f6a8a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d2dc: skipped 000906 1789132192.620106 mapread 0x0393e6 thru 0x03d2db (0x3ef6 bytes) 000907 1789132192.654963 mapwrite 0x021382 thru 0x02f74d (0xe3cc bytes) 000908 1789132192.722761 read 0x037422 thru 0x03a2d7 (0x2eb6 bytes) 000909 1789132192.723454 punch from 0x023d7e to 0x031ad0 (0xdd53 bytes) 000910 1789132192.735762 punch from 0x013999 to 0x017413 (0x3a7b bytes) 000911 1789132192.746069 trunc from 0x00c248 to 0x010e49 (0x4c02 bytes) 000912 1789132192.752703 trunc from 0x03d2dc to 0x0065fd (0x36cde bytes) 000913 1789132192.778457 trunc from 0x0065fe to 0x03d6e6 (0x370e9 bytes) 000914 1789132192.794667 trunc from 0x03d6e7 to 0x00fd09 (0x2d9dd bytes) 000915 1789132192.814221 mapwrite 0x02f59d thru 0x032e86 (0x38ea bytes) 000916 1789132192.868792 trunc from 0x00c86c to 0x011c66 (0x53fb bytes) 000917 1789132192.881396 write 0x0158f4 thru 0x01d83f (0x7f4c bytes) 000919 1789132192.898773 write 0x024353 thru 0x02c38a (0x8038 bytes) 000920 1789132192.907011 trunc from 0x00b490 to 0x017054 (0xbbc5 bytes) 000921 1789132192.937837 read 0x001e04 thru 0x002eb1 (0x10ae bytes) 000922 1789132192.970273 mapwrite 0x0202ef thru 0x02332f (0x3041 bytes) 000923 1789132193.040834 fallocate 0x03ab40 thru 0x03ffff (0x54c0 bytes) 000924 1789132193.053241 mapwrite 0x035c67 thru 0x03b75f (0x5af9 bytes) 000925 1789132193.168329 trunc from 0x02b4e2 to 0x034151 (0x8c70 bytes) 000926 1789132193.177105 trunc from 0x03b760 to 0x020462 (0x1b2fd bytes) 000927 1789132193.237003 MIRROR_RESYNC 2 mirrors 000928 1789132194.334946 trunc from 0x020463 to 0x024df5 (0x4993 bytes) 000931 1789132194.387066 read 0x01155c thru 0x013db4 (0x2859 bytes) 000932 1789132194.398104 MIRROR_RESYNC 2 mirrors 000934 1789132195.043086 trunc from 0x0147de to 0x0230ba (0xe8dd bytes) 000935 1789132195.079163 mapread 0x01c3c6 thru 0x024df5 (0x8a30 bytes) 000936 1789132195.102984 read 0x024d0e thru 0x024df5 (0xe8 bytes) 000937 1789132195.104332 trunc from 0x024df6 to 0x00fbcf (0x15226 bytes) 000938 1789132195.152480 fallocate 0x013c2b thru 0x020931 (0xcd07 bytes) 000939 1789132195.158624 fallocate 0x0125a7 thru 0x018c86 (0x66e0 bytes) 000940 1789132195.164381 trunc from 0x012615 to 0x018c86 (0x6672 bytes) 000941 1789132195.169261 read 0x00793c thru 0x007e95 (0x55a bytes) 000942 1789132195.181298 mapread 0x00985e thru 0x00e298 (0x4a3b bytes) 000943 1789132195.183010 punch from 0x0132bf to 0x018c86 (0x59c8 bytes) 000944 1789132195.187957 trunc from 0x0112dc to 0x018c86 (0x79ab bytes) 000945 1789132195.193088 mapwrite 0x00d318 thru 0x00db7e (0x867 bytes) 000946 1789132195.213060 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) 000947 1789132195.407400 fallocate 0x0336e0 thru 0x03ffff (0xc920 bytes) 000948 1789132195.411970 trunc from 0x00b785 to 0x01a2e5 (0xeb61 bytes) 000949 1789132195.418412 punch from 0x029800 to 0x033bb3 (0xa3b4 bytes) 000950 1789132195.430009 mapread 0x02ed9c thru 0x036446 (0x76ab bytes) 000951 1789132195.468881 MIRROR_EXTEND to 2 mirrors 000953 1789132196.140369 trunc from 0x040000 to 0x02f7a7 (0x10858 bytes) 000954 1789132196.274523 write 0x0003df thru 0x007c51 (0x7873 bytes) 000955 1789132196.286128 trunc from 0x02f7a8 to 0x01acef (0x14ab8 bytes) 000957 1789132196.309993 fallocate 0x039614 thru 0x03e7b8 (0x51a5 bytes) 000958 1789132196.316093 mapwrite 0x00c6de thru 0x0136e2 (0x7005 bytes) 000959 1789132196.366061 mapread 0x004cc8 thru 0x013aa7 (0xede0 bytes) 000960 1789132196.369083 fallocate 0x0042a6 thru 0x00f972 (0xb6cd bytes) 000961 1789132196.374923 trunc from 0x01acf0 to 0x018f0b (0x1de4 bytes) 000962 1789132196.391371 read 0x005d9f thru 0x0088a5 (0x2b07 bytes) 000963 1789132196.397782 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x18f0c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18f0c): Mirror 2: 0x724e012 (stale) CRC-32 checksum value for chunk [0, 0x18f0c): Mirror 1: 0x3ad8b914 Mirror 3: 0x3ad8b914 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18f0c: skipped 000964 1789132197.591157 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x18f0c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18f0c): Mirror 2: 0x724e012 (stale) CRC-32 checksum value for chunk [0, 0x18f0c): Mirror 1: 0x3ad8b914 Mirror 3: 0x3ad8b914 Mirror 4: 0x3ad8b914 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18f0c: skipped 000965 1789132199.144906 punch from 0x00475c to 0x00fedb (0xb780 bytes) 000966 1789132199.205079 punch from 0x006799 to 0x00a2b2 (0x3b1a bytes) 000967 1789132199.222278 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18f0c): Mirror 3: 0x3ad8b914 (stale) Mirror 4: 0x3ad8b914 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18f0c): Mirror 3: 0x3ad8b914 (stale) Mirror 4: 0x3ad8b914 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18f0c: skipped 000968 1789132199.795985 mapread 0x00e643 thru 0x013cc0 (0x567e bytes) 000969 1789132199.844949 mapread 0x0061b1 thru 0x00ac14 (0x4a64 bytes) 000970 1789132199.874089 trunc from 0x018cff to 0x018f0b (0x20d bytes) 000971 1789132199.881841 fallocate 0x0125eb thru 0x012e71 (0x887 bytes) 000972 1789132199.888959 fallocate 0x03e505 thru 0x03ffff (0x1afb bytes) 000974 1789132199.908922 punch from 0x00996c to 0x018f0b (0xf5a0 bytes) 000975 1789132199.931258 write 0x0341c9 thru 0x038918 (0x4750 bytes) 000976 1789132199.951041 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x38919): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38919): Mirror 3: 0xaa39fd8d (stale) Mirror 4: 0xaa39fd8d (stale) CRC-32 checksum value for chunk [0, 0x38919): Mirror 1: 0x1aad74d3 Mirror 5: 0x1aad74d3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38919: skipped 000977 1789132201.146959 trunc from 0x038919 to 0x017a9f (0x20e79 bytes) 000978 1789132201.208014 mapwrite 0x02e224 thru 0x0349c2 (0x679f bytes) 000980 1789132201.300240 trunc from 0x0349c3 to 0x005f48 (0x2ea7a bytes) 000981 1789132201.329352 mapwrite 0x036489 thru 0x03ffff (0x9b77 bytes) 000982 1789132201.393702 write 0x034b83 thru 0x03ffff (0xb47d bytes) 000983 1789132201.396535 mapread 0x019919 thru 0x023b28 (0xa210 bytes) 000984 1789132201.420425 punch from 0x025cae to 0x02f282 (0x95d5 bytes) 000985 1789132201.436972 mapread 0x01f4d9 thru 0x02dedb (0xea03 bytes) 000987 1789132201.450878 trunc from 0x0083e2 to 0x013249 (0xae68 bytes) 000988 1789132201.456908 trunc from 0x040000 to 0x0141a7 (0x2be58 bytes) 000989 1789132201.494922 write 0x019d5c thru 0x01ca33 (0x2cd8 bytes) 000990 1789132201.496783 read 0x00418b thru 0x00c474 (0x82ea bytes) 000991 1789132201.514141 trunc from 0x015f79 to 0x0176f2 (0x177a bytes) 000992 1789132201.521953 read 0x01789d thru 0x01ca33 (0x5197 bytes) 000993 1789132201.535156 read 0x001318 thru 0x006168 (0x4e51 bytes) 000994 1789132201.536061 fallocate 0x009bc5 thru 0x017e85 (0xe2c1 bytes) 000995 1789132201.540613 write 0x01a03b thru 0x0207c1 (0x6787 bytes) 000996 1789132201.546839 trunc from 0x0207c2 to 0x01ea37 (0x1d8a bytes) 000997 1789132201.564773 mapwrite 0x00305f thru 0x00f935 (0xc8d7 bytes) 000998 1789132201.598984 punch from 0x0037fc to 0x00c122 (0x8927 bytes) 000999 1789132201.616613 MIRROR_RESYNC 4 mirrors 001000 1789132202.982748 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) All operations completed A-OK! PASS 70a (108s) == sanity-flr test 71: check mirror extend parent fid ==== 09:10:10 (1789132210) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x20:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (13s) == sanity-flr test 72: check append on FLR file ========== 09:10:23 (1789132223) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ce:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.184696 s, 11.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ce:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155978 s, 6.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2cf:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ce:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2cf:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ce:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d4:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (13s) == sanity-flr test 200a: stress test ===================== 09:10:36 (1789132236) Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre2 Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 3750176 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6064416 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7033792 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8673440 .. Extending file size to 7949024 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8934848 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9112640 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 62823 62825 62827 62828 62829 done 192.168.201.131@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg131-client.virtnet /mnt/lustre2 (opts:) 192.168.201.131@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg131-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (85s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:12:01 (1789132321) Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre2 Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 1723776 .. Extending file size to 1911328 .. failed Extending file size to 7633152 .. Extending file size to 3129184 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8944512 .. Extending file size to 7070688 .. Extending file size to 7510848 .. Extending file size to 8151392 .. Extending file size to 9025504 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8767168 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 64737 64738 64739 64743 64744 done 192.168.201.131@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg131-client.virtnet /mnt/lustre2 (opts:) 192.168.201.131@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg131-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (85s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:13:26 (1789132406) Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2580 1268724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1864 1269440 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18688 3588332 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24296 3582724 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42984 7171056 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2580 1268724 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1864 1269440 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18688 3588332 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24296 3582724 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42984 7171056 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.154308 s, 13.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.285987 s, 7.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x455324b6 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.201.131@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg131-client.virtnet /mnt/lustre2 (opts:) PASS 200c (22s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:13:50 (1789132430) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.127212 s, 16.5 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 09:13:59 (1789132439) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d5:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2eb:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2eb:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ec:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ef:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:14:08 (1789132448) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.113705 s, 18.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:14:16 (1789132456) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.138443 s, 15.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x30:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x30:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x31:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:14:26 (1789132466) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.168699 s, 12.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ed:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ed:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f1:0x0] } PASS 204c (37s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:15:03 (1789132503) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg131-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134083 s, 7.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ee:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ee:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ef:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:16:00 (1789132560) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg131-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.151006 s, 13.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (57s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:16:56 (1789132616) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg131-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg131-server: Pool lustre.test_204f created Adding targets to pool oleg131-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg131-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.226132 s, 9.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg131-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg131-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg131-server: Pool lustre.test_204f destroyed PASS 204f (85s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:18:21 (1789132701) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:18:31 (1789132711) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:18:41 (1789132721) oleg131-server: Pool lustre.test_206 created Adding targets to pool oleg131-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg131-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg131-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg131-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg131-server: Pool lustre.test_206 destroyed PASS 206 (35s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:19:17 (1789132757) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14561 s, 7.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293911 s, 35.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x87119eb8 Mirror 3: 0x87119eb8 PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:19:31 (1789132771) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:19:35 (1789132775) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:19:39 (1789132779) Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.110426 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1ce0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1900000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1900000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1900000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1a40000 Read this (4 bytes): BBBB 192.168.201.131@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg131-client.virtnet /mnt/lustre2 (opts:) PASS 209a (44s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:20:23 (1789132823) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0307546 s, 266 kB/s the very first read read with warm cache PASS 209b (10s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:20:33 (1789132833) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701206 s, 15.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1f6 lmm_fid: [0x200000405:0x1f6:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1f6 lmm_fid: [0x200000405:0x1f6:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ff:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611638 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 09:20:35.000000000 -0400 Modify: 2026-09-11 09:20:35.000000000 -0400 Change: 2026-09-11 09:20:38.000000000 -0400 Birth: 2026-09-11 09:20:36.000000000 -0400 PASS 210a (12s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:20:46 (1789132846) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044139 s, 23.8 MB/s oleg131-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg131-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg131-server: O/280000401/d29/2fd: File not found by ext2_lookup PASS 210b (23s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:21:08 (1789132868) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.342954 s, 119 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0389516 s, 105 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 11 09:21 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1fb:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 11 09:21 /mnt/lustre/f211.sanity-flr PASS 211 (11s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 09:21:19 (1789132879) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 09:21:29 (1789132889) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 2110 sec =========== 09:21:40 (1789132900) osc.lustre-OST0000-osc-ffff8be492b5a000.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 osc.lustre-OST0001-osc-ffff8be492b5a000.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 obdfilter.lustre-OST0000.stats_compr= snapshot_time 1789132904.738790447 secs.nsecs start_time 1789131359.705581815 secs.nsecs elapsed_time 1545.033208632 secs.nsecs uncompressed_objects 2423 samples [reqs] obdfilter.lustre-OST0001.stats_compr= snapshot_time 1789132904.738839839 secs.nsecs start_time 1789131399.864342671 secs.nsecs elapsed_time 1504.874497168 secs.nsecs uncompressed_objects 2485 samples [reqs] === sanity-flr: start cleanup 09:21:44 (1789132904) === === sanity-flr: finish cleanup 09:21:54 (1789132914) ===