-----============= acceptance-small: sanity-flr ============----- Tue Sep 8 08:10:15 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg442-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 08:10:35 (1788869435) === oleg442-client.virtnet: executing check_config_client /mnt/lustre oleg442-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg442-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa036d1346000.idle_timeout=debug osc.lustre-OST0001-osc-ffffa036d1346000.idle_timeout=debug disable quota as required oleg442-server: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 08:11:24 (1788869484) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8210] == sanity-flr test 0a: lfs mirror create with -N option == 08:11:26 (1788869486) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 08:11:45 (1788869505) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 08:11:49 (1788869509) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 08:11:53 (1788869513) PASS 0d (27s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 08:12:20 (1788869540) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 08:12:24 (1788869544) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 08:12:29 (1788869549) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffffa036d1346000.stats=clear osc.lustre-OST0001-osc-ffffa036d1346000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 08:12:38 (1788869558) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 08:12:50 (1788869570) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 08:13:00 (1788869580) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 08:13:04 (1788869584) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 08:13:13 (1788869593) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 08:13:22 (1788869602) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 08:13:31 (1788869611) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 08:13:40 (1788869620) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 08:13:48 (1788869628) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 08:14:00 (1788869640) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 08:14:07 (1788869647) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.486384 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100369 s, 10.4 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 08:14:16 (1788869656) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0812186 s, 25.8 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 08:14:25 (1788869665) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 08:14:37 (1788869677) Stopping /mnt/lustre-ost1 (opts:) on oleg442-server oleg442-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa036d1346000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa036d1346000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg442-server: oleg442-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg442-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg442-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg442-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg442-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg442-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg442-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg442-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg442-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg442-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg442-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg442-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg442-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg442-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg442-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg442-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg442-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg442-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg442-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg442-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg442-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg442-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg442-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg442-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg442-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg442-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg442-server oleg442-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa036d1346000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa036d1346000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg442-server: oleg442-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg442-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (145s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 08:17:02 (1788869822) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.978892 s, 11.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg442-server oleg442-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa036d1346000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa036d1346000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg442-server: oleg442-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg442-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (52s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 08:17:55 (1788869875) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg442-server oleg442-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa036d1346000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa036d1346000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg442-server: oleg442-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg442-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg442-server oleg442-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa036d1346000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa036d1346000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg442-server: oleg442-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg442-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (89s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 08:19:25 (1788869965) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.27198 s, 16.0 MB/s -rw-r--r-- 1 root root 52428800 Sep 8 08:19 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.161s user 0m0.001s sys 0m0.176s Stopping /mnt/lustre-ost1 (opts:) on oleg442-server oleg442-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa036d1346000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa036d1346000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.256s user 0m0.001s sys 0m0.181s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg442-server: oleg442-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg442-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg442-server oleg442-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa036d1346000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa036d1346000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.378s user 0m0.000s sys 0m0.191s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg442-server: oleg442-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg442-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg442-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa036d1346000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa036d1346000.ost_server_uuid in FULL state after 0 sec PASS 33b (101s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 08:21:06 (1788870066) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 08:21:10 (1788870070) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 08:21:13 (1788870073) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 08:21:17 (1788870077) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 08:21:25 (1788870085) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.45147 s, 27.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 08:21:37 (1788870097) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580106 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244932 s, 42.8 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 08:21:46 (1788870106) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 08:21:54 (1788870114) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00182768 s, 560 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00129711 s, 789 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00365191 s, 280 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0022543 s, 454 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 08:22:04 (1788870124) create test files with size 15737643 .. 8ce4e03ba94bd1d4e061be24bc8540ee - 9a116a4b0e88f77af24d5d22518a8573 - de7a3f0461e021ab66b1fc0beac8d7d5 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg442-client.virtnet /mnt/lustre (opts:) Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 08:22:29 (1788870149) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 08:22:43 (1788870163) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 08:22:51 (1788870171) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444992 s, 23.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263991 s, 39.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 08:23:02 (1788870182) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0642118 s, 32.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.389955 s, 10.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.222269 s, 9.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 08:23:19 (1788870199) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 08:23:24 (1788870204) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 08:23:28 (1788870208) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768895 s, 13.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 08:23:37 (1788870217) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.233151 s, 13.5 MB/s PASS 44a (9s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 08:23:46 (1788870226) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc8:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcd:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcd:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 08:23:56 (1788870236) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.342819 s, 30.6 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg442-client.virtnet /mnt/lustre (opts:) Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 08:24:08 (1788870248) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffffa036d1088800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.23642 s, 44.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg442-client.virtnet /mnt/lustre (opts:) Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffffa036c5d2b000.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 08:24:24 (1788870264) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (36s) == sanity-flr test 46: Verify setstripe --copy option ==== 08:25:00 (1788870300) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (35s) == sanity-flr test 47: Verify mirror obj alloc =========== 08:25:35 (1788870335) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 08:25:39 (1788870339) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.373161 s, 8.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.192248 s, 16.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xcde955dd (nosync,stale) Mirror 2: 0xcde955dd (nosync,stale) ** verify mirror 2 doesn't change original checksum: 1b5b9ec31df39d5a42b006c90b931fe9 - mirror 1 checksum: 37f9aff4fc89497ce99ddfcbff6b897f - mirror 2 checksum: 1b5b9ec31df39d5a42b006c90b931fe9 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-08T08:25:43-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-08T08:25:43-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 1b5b9ec31df39d5a42b006c90b931fe9 - mirror 1 checksum: 37f9aff4fc89497ce99ddfcbff6b897f - mirror 2 checksum: 37f9aff4fc89497ce99ddfcbff6b897f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-08T08:25:43-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-08T08:25:43-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 08:25:50 (1788870350) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346771 s, 30.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 08:25:57 (1788870357) Failing mds1 on oleg442-server Stopping /mnt/lustre-mds1 (opts:) on oleg442-server 08:26:10 (1788870370) shut down facet: mds1 facet_host: oleg442-server facet_failover_host: oleg442-server Failover mds1 to oleg442-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg442-server: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-MDT0000 08:26:42 (1788870402) targets are mounted 08:26:42 (1788870402) facet_failover done oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270808 s, 38.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } PASS 50A (65s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 08:27:03 (1788870423) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0658854 s, 311 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0318868 s, 642 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (14s) == sanity-flr test 50b: mirror rsync handles sparseness == 08:27:16 (1788870436) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0724342 s, 283 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0128773 s, 1.6 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (12s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 08:27:28 (1788870448) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00909269 s, 1.8 MB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 08:27:38 (1788870458) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0785581 s, 261 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (10s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 08:27:48 (1788870468) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0155997 s, 263 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0525992 s, 77.9 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 08:27:57 (1788870477) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (28s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 08:28:25 (1788870505) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 08:28:54 (1788870534) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 08:29:18 (1788870558) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 08:29:43 (1788870583) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 08:29:52 (1788870592) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57570 mirror split pid 57571 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7769: 57570 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 57571 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 08:31:00 (1788870660) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 662 truncating to largest ever: 0x12fe4 000003 1788870662.642713 trunc from 00000000 to 0x012fe3 (0x12fe4 bytes) 000004 1788870662.671109 mapwrite 0x0333c4 thru 0x03ca70 (0x96ad bytes) 000005 1788870662.740626 mapread 0x02518e thru 0x033226 (0xe099 bytes) 000006 1788870662.754646 write 0x01b734 thru 0x023a88 (0x8355 bytes) 000007 1788870662.771777 punch from 0x013bfd to 0x021ba9 (0xdfad bytes) 000008 1788870662.798786 mapwrite 0x01052c thru 0x012991 (0x2466 bytes) 000010 1788870662.831774 punch from 0x0130eb to 0x0144ed (0x1403 bytes) 000011 1788870662.838798 trunc from 0x01d419 to 0x02a049 (0xcc31 bytes) zero_range to largest ever: 0x1a1a5 000012 1788870662.844110 trunc from 0x00dfec to 0x01a1a4 (0xc1b9 bytes) fallocating to largest ever: 0x2d07a 000013 1788870662.853216 fallocate 0x0209a6 thru 0x02d079 (0xc6d4 bytes) 000014 1788870662.859349 mapwrite 0x00a7fe thru 0x0175c3 (0xcdc6 bytes) 000015 1788870662.906442 trunc from 0x017ec0 to 0x02014a (0x828b bytes) 000016 1788870662.913098 mapwrite 0x00d59b thru 0x018e73 (0xb8d9 bytes) 000017 1788870662.963408 punch from 0x018839 to 0x01e56a (0x5d32 bytes) 000018 1788870662.970179 mapread 0x02cd21 thru 0x0333b3 (0x6693 bytes) zero_range to largest ever: 0x35971 000019 1788870662.972284 trunc from 0x0331bf to 0x035970 (0x27b2 bytes) 000020 1788870662.978434 mapwrite 0x026433 thru 0x02d8ff (0x74cd bytes) 000021 1788870663.016110 mapwrite 0x0123f4 thru 0x01aff2 (0x8bff bytes) 000022 1788870663.055063 trunc from 0x02646a to 0x02794b (0x14e2 bytes) 000023 1788870663.062971 trunc from 0x005064 to 0x010424 (0xb3c1 bytes) 000025 1788870663.084327 MIRROR_EXTEND to 2 mirrors 000026 1788870663.800853 mapwrite 0x02c397 thru 0x03b189 (0xedf3 bytes) 000027 1788870663.903184 trunc from 0x03ca71 to 0x02066c (0x1c404 bytes) 000028 1788870663.952503 read 0x01be24 thru 0x02066c (0x4849 bytes) 000029 1788870663.961912 write 0x029122 thru 0x036877 (0xd756 bytes) 000030 1788870663.964480 punch from 0x00ad62 to 0x00fdb2 (0x5051 bytes) 000031 1788870663.969386 mapread 0x00a8c3 thru 0x00b67c (0xdba bytes) 000032 1788870663.978282 read 0x00e20c thru 0x012378 (0x416d bytes) 000033 1788870663.983229 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36878): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36878): Mirror 2: 0x94c61936 (stale) CRC-32 checksum value for chunk [0, 0x36878): Mirror 1: 0x54fd8d4b Mirror 3: 0x54fd8d4b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36878: skipped 000034 1788870664.870617 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36878): Mirror 2: 0x94c61936 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36878): Mirror 2: 0x94c61936 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36878: skipped 000035 1788870665.196176 read 0x018908 thru 0x01ae31 (0x252a bytes) 000036 1788870665.209242 MIRROR_RESYNC 2 mirrors 000037 1788870665.883643 mapwrite 0x01e31c thru 0x023913 (0x55f8 bytes) 000038 1788870666.006039 fallocate 0x011828 thru 0x014a0f (0x31e8 bytes) 000040 1788870666.021591 mapread 0x020b21 thru 0x0290a6 (0x8586 bytes) 000041 1788870666.035646 punch from 0x03620e to 0x036877 (0x66a bytes) 000042 1788870666.042645 mapread 0x01efb3 thru 0x022811 (0x385f bytes) 000043 1788870666.047144 fallocate 0x0296e0 thru 0x0330dc (0x99fd bytes) 000044 1788870666.057552 write 0x0392c6 thru 0x03ffff (0x6d3a bytes) 000045 1788870666.059000 write 0x016990 thru 0x025fcc (0xf63d bytes) 000047 1788870666.076477 MIRROR_RESYNC 2 mirrors 000048 1788870666.642266 MIRROR_RESYNC 2 mirrors 000049 1788870667.038276 trunc from 0x004697 to 0x00d4cf (0x8e39 bytes) 000050 1788870667.100262 punch from 0x003ef2 to 0x00d430 (0x953f bytes) 000051 1788870667.104684 trunc from 0x031fc7 to 0x03e9d5 (0xca0f bytes) 000053 1788870667.114352 punch from 0x000700 to 0x008855 (0x8156 bytes) 000054 1788870667.119165 mapread 0x02d5ba thru 0x02fe04 (0x284b bytes) 000055 1788870667.165108 read 0x014b55 thru 0x0229af (0xde5b bytes) 000056 1788870667.198420 fallocate 0x0215c3 thru 0x02d92f (0xc36d bytes) 000057 1788870667.212166 trunc from 0x005b3b to 0x0129b2 (0xce78 bytes) 000058 1788870667.243529 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17b:0x0]: Invalid argument (22) 000059 1788870667.457143 trunc from 0x040000 to 0x0305d8 (0xfa27 bytes) 000060 1788870667.532139 mapread 0x021d87 thru 0x0277e5 (0x5a5f bytes) 000061 1788870667.574512 MIRROR_EXTEND to 2 mirrors 000062 1788870668.414320 write 0x02592f thru 0x029e24 (0x44f6 bytes) 000063 1788870668.508406 punch from 0x022749 to 0x0305d8 (0xde90 bytes) 000064 1788870668.536365 fallocate 0x01c665 thru 0x02aa61 (0xe3fd bytes) 000065 1788870668.547405 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x305d9): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x305d9): Mirror 2: 0x639cb870 (stale) CRC-32 checksum value for chunk [0, 0x305d9): Mirror 1: 0x47a246fc Mirror 3: 0x47a246fc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x305d9: skipped 000066 1788870669.510281 mapread 0x028eda thru 0x0305d8 (0x76ff bytes) 000067 1788870669.578355 mapread 0x01c44b thru 0x01c849 (0x3ff bytes) 000068 1788870669.646300 mapwrite 0x013c55 thru 0x0219e4 (0xdd90 bytes) 000070 1788870669.787083 mapread 0x02a3c5 thru 0x0305d8 (0x6214 bytes) 000071 1788870669.798149 mapwrite 0x0257b0 thru 0x025f05 (0x756 bytes) 000072 1788870669.816416 trunc from 0x0305d9 to 0x00c907 (0x23cd1 bytes) 000074 1788870669.850591 trunc from 0x00c908 to 0x025daf (0x194a8 bytes) 000075 1788870669.874553 write 0x016988 thru 0x017196 (0x80f bytes) 000076 1788870669.890268 punch from 0x00f293 to 0x0112cc (0x203a bytes) 000077 1788870669.895779 mapread 0x0194d8 thru 0x025daf (0xc8d8 bytes) 000078 1788870669.914175 mapread 0x01e19a thru 0x0255d2 (0x7439 bytes) 000079 1788870669.915768 punch from 0x01ee6a to 0x0211b6 (0x234d bytes) 000080 1788870669.923133 trunc from 0x025a1b to 0x025daf (0x395 bytes) 000081 1788870669.929751 write 0x03c04e thru 0x03ffff (0x3fb2 bytes) 000082 1788870669.931136 write 0x005efe thru 0x00892c (0x2a2f bytes) 000084 1788870669.957685 fallocate 0x01a2d7 thru 0x01c85e (0x2588 bytes) 000085 1788870669.966384 trunc from 0x0120be to 0x0149be (0x2901 bytes) 000086 1788870669.974524 MIRROR_RESYNC 3 mirrors 000087 1788870670.896309 trunc from 0x040000 to 0x0077f6 (0x38809 bytes) 000088 1788870670.930292 read 0x000cbf thru 0x0077f6 (0x6b38 bytes) 000089 1788870670.939088 mapwrite 0x010e6b thru 0x0136b8 (0x284e bytes) 000090 1788870670.989347 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x136b9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x136b9): Mirror 2: 0xefb7f75e (stale) Mirror 3: 0xefb7f75e (stale) CRC-32 checksum value for chunk [0, 0x136b9): Mirror 1: 0x2ee188e Mirror 4: 0x2ee188e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x136b9: skipped 000091 1788870671.871581 MIRROR_RESYNC 4 mirrors 000092 1788870672.915787 mapread 0x001c9a thru 0x004af4 (0x2e5b bytes) 000093 1788870672.943437 MIRROR_EXTEND to 5 mirrors 000094 1788870673.852062 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17b:0x0]: Invalid argument (22) 000096 1788870674.345993 read 0x010aec thru 0x0136b8 (0x2bcd bytes) 000097 1788870674.354253 write 0x03d3d8 thru 0x03ffff (0x2c28 bytes) 000098 1788870674.428284 punch from 0x034d04 to 0x037bfd (0x2efa bytes) fallocating to largest ever: 0x3ad31 000099 1788870674.440598 fallocate 0x0308be thru 0x03ad30 (0xa473 bytes) 000100 1788870674.453738 mapwrite 0x008838 thru 0x017a82 (0xf24b bytes) 000103 1788870674.565920 read 0x0251aa thru 0x02bb4f (0x69a6 bytes) 000104 1788870674.581474 MIRROR_RESYNC 4 mirrors 000105 1788870675.701178 MIRROR_EXTEND to 5 mirrors 000106 1788870676.445369 mapwrite 0x01da08 thru 0x020fd3 (0x35cc bytes) 000107 1788870676.574658 trunc from 0x040000 to 0x02f169 (0x10e96 bytes) 000108 1788870676.610989 fallocate 0x011304 thru 0x014cca (0x39c7 bytes) 000109 1788870676.622228 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f16a): Mirror 2: 0xb1e286ba (stale) Mirror 5: 0xb1e286ba (stale) Mirror 6: 0xb1e286ba (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f16a): Mirror 2: 0xb1e286ba (stale) Mirror 5: 0xb1e286ba (stale) Mirror 6: 0xb1e286ba (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f16a: skipped 000110 1788870677.068157 MIRROR_RESYNC 4 mirrors 000111 1788870678.167923 trunc from 0x016a37 to 0x02235f (0xb929 bytes) 000112 1788870678.205614 fallocate 0x01f7cc thru 0x02cfff (0xd834 bytes) 000113 1788870678.214982 MIRROR_RESYNC 4 mirrors 000114 1788870679.169506 MIRROR_RESYNC 4 mirrors 000115 1788870679.630039 trunc from 0x019427 to 0x01ee0e (0x59e8 bytes) 000116 1788870679.683439 trunc from 0x0117b5 to 0x01cf69 (0xb7b5 bytes) 000117 1788870679.691865 mapwrite 0x02dd21 thru 0x031ba8 (0x3e88 bytes) 000118 1788870679.812397 trunc from 0x0001a5 to 0x0091fd (0x9059 bytes) 000119 1788870679.831548 read 0x006984 thru 0x0168ae (0xff2b bytes) 000121 1788870679.852781 write 0x006ecf thru 0x007129 (0x25b bytes) 000122 1788870679.854992 punch from 0x023d82 to 0x02720a (0x3489 bytes) 000123 1788870679.870157 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31ba9): Mirror 5: 0xff63efb (stale) Mirror 6: 0xff63efb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31ba9): Mirror 5: 0xff63efb (stale) Mirror 6: 0xff63efb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31ba9: skipped fallocating to largest ever: 0x3b185 000124 1788870680.233625 fallocate 0x039757 thru 0x03b184 (0x1a2e bytes) 000125 1788870680.241503 mapwrite 0x032d69 thru 0x03b8a9 (0x8b41 bytes) 000126 1788870680.304203 write 0x01bc22 thru 0x01c929 (0xd08 bytes) 000127 1788870680.318005 MIRROR_RESYNC 3 mirrors 000128 1788870681.395046 MIRROR_EXTEND to 4 mirrors 000129 1788870682.088359 punch from 0x022799 to 0x02ffc4 (0xd82c bytes) 000130 1788870682.128353 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3b8aa): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b8aa): Mirror 5: 0x552fd80b (stale) Mirror 6: 0x552fd80b (stale) Mirror 7: 0x552fd80b (stale) CRC-32 checksum value for chunk [0, 0x3b8aa): Mirror 1: 0xa759bd34 Mirror 8: 0xa759bd34 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b8aa: skipped 000131 1788870683.002501 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b8aa): Mirror 6: 0x552fd80b (stale) Mirror 7: 0x552fd80b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b8aa): Mirror 6: 0x552fd80b (stale) Mirror 7: 0x552fd80b (stale) CRC-32 checksum value for chunk [0, 0x3b8aa): Mirror 1: 0xa759bd34 Mirror 8: 0xa759bd34 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b8aa: skipped 000132 1788870683.659815 trunc from 0x03376c to 0x0350bd (0x1952 bytes) 000133 1788870683.720257 trunc from 0x0289db to 0x03038a (0x79b0 bytes) 000134 1788870683.726972 write 0x00cb5e thru 0x010665 (0x3b08 bytes) 000135 1788870683.739322 write 0x003589 thru 0x01196c (0xe3e4 bytes) 000137 1788870683.752337 read 0x023166 thru 0x023e92 (0xd2d bytes) 000138 1788870683.757273 punch from 0x038503 to 0x03b8a9 (0x33a7 bytes) 000140 1788870683.765349 trunc from 0x03b8aa to 0x00c75c (0x2f14d bytes) 000141 1788870683.790502 mapread 0x000117 thru 0x00c75c (0xc646 bytes) 000142 1788870683.808675 trunc from 0x003ddc to 0x007871 (0x3a96 bytes) 000143 1788870683.827420 write 0x031ee6 thru 0x038c46 (0x6d61 bytes) 000144 1788870683.835692 read 0x01905a thru 0x01b00d (0x1fb4 bytes) fallocating to largest ever: 0x3e647 000145 1788870683.845024 fallocate 0x034390 thru 0x03e646 (0xa2b7 bytes) 000146 1788870683.855188 trunc from 0x038c47 to 0x0231d9 (0x15a6d bytes) 000147 1788870683.889151 read 0x0125a8 thru 0x021393 (0xedec bytes) 000148 1788870683.890054 mapread 0x000883 thru 0x00a34d (0x9acb bytes) 000149 1788870683.895334 trunc from 0x0231da to 0x001821 (0x219b8 bytes) 000150 1788870683.914970 trunc from 0x001822 to 0x035154 (0x33933 bytes) 000151 1788870683.944708 MIRROR_RESYNC 4 mirrors 000152 1788870684.782081 mapwrite 0x01794f thru 0x0208f5 (0x8fa7 bytes) 000153 1788870684.932484 trunc from 0x035155 to 0x00de8f (0x272c5 bytes) 000154 1788870684.957312 read 0x008390 thru 0x00a9d0 (0x2641 bytes) 000155 1788870684.965174 trunc from 0x00de90 to 0x010c70 (0x2de1 bytes) 000156 1788870684.987562 trunc from 0x010c71 to 0x02cdda (0x1c16a bytes) 000158 1788870685.016360 mapread 0x01759f thru 0x026da7 (0xf809 bytes) 000159 1788870685.058908 write 0x016db2 thru 0x019002 (0x2251 bytes) 000160 1788870685.061325 read 0x00d1ed thru 0x014449 (0x725d bytes) 000161 1788870685.095254 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cddb): Mirror 6: 0x53281a4c (stale) Mirror 7: 0x53281a4c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cddb): Mirror 6: 0x53281a4c (stale) Mirror 7: 0x53281a4c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cddb: skipped fallocating to largest ever: 0x40000 000162 1788870685.424447 fallocate 0x0393e9 thru 0x03ffff (0x6c17 bytes) 000163 1788870685.430596 write 0x029872 thru 0x02d524 (0x3cb3 bytes) 000164 1788870685.441869 punch from 0x01dc99 to 0x029bc2 (0xbf2a bytes) 000165 1788870685.468206 mapread 0x013d02 thru 0x01ff25 (0xc224 bytes) 000166 1788870685.494428 trunc from 0x02d525 to 0x0047b9 (0x28d6b bytes) 000167 1788870685.515216 punch from 0x002141 to 0x0047b9 (0x2679 bytes) 000168 1788870685.522146 trunc from 0x004146 to 0x0047b9 (0x674 bytes) 000169 1788870685.528158 punch from 0x002599 to 0x0047b9 (0x2221 bytes) 000170 1788870685.538274 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x47ba): Mirror 7: 0xc49ac81b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x47ba): Mirror 7: 0xc49ac81b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x47ba: skipped 000171 1788870685.809768 MIRROR_RESYNC 2 mirrors 000172 1788870686.399694 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17b:0x0]: Invalid argument (22) 000175 1788870686.526758 read 0x000f1a thru 0x0047b9 (0x38a0 bytes) 000176 1788870686.547243 mapread 0x000801 thru 0x0047b9 (0x3fb9 bytes) 000177 1788870686.549239 fallocate 0x02a865 thru 0x0305b9 (0x5d55 bytes) 000178 1788870686.566665 mapwrite 0x01fe00 thru 0x027c4a (0x7e4b bytes) 000179 1788870686.624901 mapwrite 0x011937 thru 0x020730 (0xedfa bytes) 000180 1788870686.656902 trunc from 0x0305ba to 0x0370d2 (0x6b19 bytes) 000181 1788870686.675265 punch from 0x01b81b to 0x028aaf (0xd295 bytes) 000182 1788870686.680639 write 0x01a014 thru 0x01d238 (0x3225 bytes) 000183 1788870686.688939 punch from 0x02e9b0 to 0x0370a8 (0x86f9 bytes) 000184 1788870686.693658 write 0x01a1f4 thru 0x023ce5 (0x9af2 bytes) 000185 1788870686.700871 fallocate 0x01aaed thru 0x02243e (0x7952 bytes) 000186 1788870686.706159 mapread 0x029d94 thru 0x0370d2 (0xd33f bytes) 000187 1788870686.728488 read 0x009bb3 thru 0x00a0bf (0x50d bytes) 000189 1788870686.740217 trunc from 0x0044c9 to 0x005e04 (0x193c bytes) 000190 1788870686.747105 read 0x02e3ff thru 0x0370d2 (0x8cd4 bytes) 000192 1788870686.749006 mapread 0x023ac9 thru 0x033a83 (0xffbb bytes) 000194 1788870686.751427 mapread 0x022211 thru 0x02e4e4 (0xc2d4 bytes) 000195 1788870686.753265 trunc from 0x0370d3 to 0x03245c (0x4c76 bytes) 000196 1788870686.769986 read 0x00a7ad thru 0x013605 (0x8e59 bytes) 000197 1788870686.770702 punch from 0x0117f3 to 0x012032 (0x840 bytes) 000198 1788870686.781651 punch from 0x024cc7 to 0x02b206 (0x6540 bytes) 000199 1788870686.787584 punch from 0x02173d to 0x025913 (0x41d7 bytes) 000200 1788870686.806063 punch from 0x0174ab to 0x026214 (0xed6a bytes) 000201 1788870686.811776 trunc from 0x03245d to 0x026676 (0xbde6 bytes) 000202 1788870686.829727 trunc from 0x00a395 to 0x00e029 (0x3c95 bytes) 000203 1788870686.837649 mapwrite 0x017656 thru 0x0219e6 (0xa391 bytes) 000204 1788870686.876990 mapwrite 0x00a36e thru 0x019fdb (0xfc6e bytes) 000205 1788870686.904001 fallocate 0x026f26 thru 0x03077c (0x9857 bytes) 000206 1788870686.909576 write 0x030253 thru 0x03bc26 (0xb9d4 bytes) 000207 1788870686.927996 mapwrite 0x01915f thru 0x02372f (0xa5d1 bytes) 000208 1788870686.955049 trunc from 0x0078bd to 0x00985b (0x1f9f bytes) 000209 1788870686.966598 write 0x03340b thru 0x03ee21 (0xba17 bytes) 000210 1788870686.968269 mapwrite 0x00e188 thru 0x0165d4 (0x844d bytes) 000211 1788870686.995082 write 0x003315 thru 0x00c2bd (0x8fa9 bytes) 000212 1788870687.003733 MIRROR_EXTEND to 2 mirrors 000213 1788870687.735527 mapread 0x034f15 thru 0x03ee21 (0x9f0d bytes) 000214 1788870687.774491 mapwrite 0x02e169 thru 0x036517 (0x83af bytes) 000215 1788870687.907243 fallocate 0x010f5d thru 0x0144da (0x357e bytes) 000218 1788870687.916310 punch from 0x004697 to 0x00895e (0x42c8 bytes) 000219 1788870687.926331 trunc from 0x01dc4a to 0x026cac (0x9063 bytes) 000220 1788870687.934066 fallocate 0x01dae1 thru 0x02c7ed (0xed0d bytes) 000221 1788870687.948759 write 0x027d3b thru 0x030e11 (0x90d7 bytes) 000222 1788870687.955709 trunc from 0x02eb7a to 0x03180c (0x2c93 bytes) 000223 1788870687.976752 write 0x014978 thru 0x0163fd (0x1a86 bytes) 000224 1788870688.000398 trunc from 0x03ee22 to 0x007c69 (0x371b8 bytes) 000225 1788870688.035568 mapwrite 0x016467 thru 0x01e618 (0x81b2 bytes) 000226 1788870688.094088 mapread 0x01a900 thru 0x01e618 (0x3d19 bytes) 000227 1788870688.095358 trunc from 0x002893 to 0x002eb7 (0x625 bytes) 000228 1788870688.103884 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1e619): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e619): Mirror 2: 0x681401ad (stale) CRC-32 checksum value for chunk [0, 0x1e619): Mirror 1: 0xec5b7371 Mirror 3: 0xec5b7371 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e619: skipped 000229 1788870688.880680 trunc from 0x01e619 to 0x0314bd (0x12ea5 bytes) 000231 1788870688.951089 fallocate 0x02f413 thru 0x038c0c (0x97fa bytes) 000232 1788870688.957438 fallocate 0x02113b thru 0x02a1e2 (0x90a8 bytes) 000233 1788870688.962943 mapwrite 0x0173a8 thru 0x023320 (0xbf79 bytes) 000234 1788870689.018936 read 0x006dbb thru 0x00c58b (0x57d1 bytes) 000235 1788870689.031451 mapread 0x026e1e thru 0x03136d (0xa550 bytes) 000236 1788870689.040758 write 0x038f74 thru 0x03b6d0 (0x275d bytes) 000237 1788870689.048121 mapwrite 0x0369b9 thru 0x037940 (0xf88 bytes) 000238 1788870689.089017 read 0x0032dd thru 0x0076b9 (0x43dd bytes) 000239 1788870689.100318 trunc from 0x01e84d to 0x0272ed (0x8aa1 bytes) 000240 1788870689.108522 trunc from 0x03b6d1 to 0x019a7a (0x21c56 bytes) 000241 1788870689.125749 fallocate 0x02ed46 thru 0x031f2c (0x31e7 bytes) 000242 1788870689.131374 mapwrite 0x0140d0 thru 0x01bf98 (0x7ec9 bytes) 000243 1788870689.181931 punch from 0x01aa92 to 0x01bf98 (0x1507 bytes) 000244 1788870689.188402 trunc from 0x00ac75 to 0x01338f (0x871b bytes) 000245 1788870689.193342 trunc from 0x01bf99 to 0x00bec0 (0x100d8 bytes) 000246 1788870689.216554 mapwrite 0x026fc5 thru 0x02d162 (0x619e bytes) 000248 1788870689.335475 read 0x0082ef thru 0x0102d9 (0x7feb bytes) 000250 1788870689.358130 trunc from 0x022f58 to 0x0266f1 (0x379a bytes) 000251 1788870689.365373 mapread 0x00da85 thru 0x018545 (0xaac1 bytes) 000252 1788870689.367722 write 0x034e03 thru 0x03fa03 (0xac01 bytes) 000253 1788870689.370503 punch from 0x03b405 to 0x03c615 (0x1211 bytes) 000254 1788870689.383906 mapread 0x011c7e thru 0x01c6a4 (0xaa27 bytes) 000255 1788870689.386225 trunc from 0x032fbb to 0x034976 (0x19bc bytes) 000257 1788870689.396095 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3fa04): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fa04): Mirror 2: 0x72c708bf (stale) Mirror 3: 0x14d8ef4a (stale) CRC-32 checksum value for chunk [0, 0x3fa04): Mirror 1: 0x404d77ad Mirror 4: 0x404d77ad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fa04: skipped 000258 1788870690.375744 trunc from 0x03fa04 to 0x02cbb8 (0x12e4b bytes) 000259 1788870690.494120 fallocate 0x016300 thru 0x018377 (0x2078 bytes) 000260 1788870690.512018 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2cbb9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cbb9): Mirror 2: 0x9eb62e43 (stale) Mirror 3: 0xbd94ba10 (stale) Mirror 4: 0x3cd517d1 (stale) CRC-32 checksum value for chunk [0, 0x2cbb9): Mirror 1: 0x3cd517d1 Mirror 5: 0x3cd517d1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cbb9: skipped 000261 1788870691.396608 punch from 0x013c80 to 0x0147be (0xb3f bytes) 000262 1788870691.457545 fallocate 0x00e4b1 thru 0x01b524 (0xd074 bytes) 000264 1788870691.475012 punch from 0x01469a to 0x01744a (0x2db1 bytes) 000265 1788870691.482110 read 0x02b6e8 thru 0x02cbb8 (0x14d1 bytes) 000266 1788870691.494802 write 0x016e3d thru 0x01fd6d (0x8f31 bytes) 000267 1788870691.523803 trunc from 0x02cbb9 to 0x03633a (0x9782 bytes) 000269 1788870691.557305 fallocate 0x01fcb7 thru 0x029370 (0x96ba bytes) 000270 1788870691.562622 write 0x00469c thru 0x00b73c (0x70a1 bytes) 000271 1788870691.572927 mapwrite 0x03a9d3 thru 0x03ffff (0x562d bytes) 000272 1788870691.626137 read 0x004acc thru 0x009ccb (0x5200 bytes) 000275 1788870691.629202 punch from 0x031d44 to 0x038e89 (0x7146 bytes) 000276 1788870691.634596 mapread 0x0366b0 thru 0x03db1e (0x746f bytes) 000277 1788870691.656724 read 0x012cc0 thru 0x0143ac (0x16ed bytes) 000279 1788870691.658132 write 0x020fe9 thru 0x0246fa (0x3712 bytes) 000280 1788870691.660171 mapread 0x0332f1 thru 0x033357 (0x67 bytes) 000281 1788870691.661274 mapwrite 0x00ceb8 thru 0x01c64a (0xf793 bytes) 000282 1788870691.681592 trunc from 0x040000 to 0x025321 (0x1acde bytes) 000284 1788870691.697939 trunc from 0x025322 to 0x018336 (0xcfeb bytes) 000286 1788870691.719019 trunc from 0x018337 to 0x0359fd (0x1d6c7 bytes) 000287 1788870691.732691 punch from 0x00d623 to 0x0117af (0x418d bytes) 000288 1788870691.737712 write 0x034e00 thru 0x035336 (0x537 bytes) 000289 1788870691.747692 fallocate 0x035560 thru 0x03ffff (0xaaa0 bytes) 000290 1788870691.752295 punch from 0x023d38 to 0x02bbcf (0x7e98 bytes) 000291 1788870691.766773 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x37aa0fa6 (stale) Mirror 3: 0xaeebf75 (stale) Mirror 4: 0x5e9a1847 (stale) Mirror 5: 0x3839e8de (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xde1a743d Mirror 6: 0xde1a743d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000293 1788870692.841160 fallocate 0x0063e8 thru 0x015375 (0xef8e bytes) 000294 1788870692.882605 trunc from 0x01093d to 0x01ad4f (0xa413 bytes) 000295 1788870692.888587 trunc from 0x003bd4 to 0x00b19c (0x75c9 bytes) 000296 1788870692.895772 punch from 0x02773b to 0x02c3b7 (0x4c7d bytes) 000297 1788870692.912723 punch from 0x03d253 to 0x03ffff (0x2dad bytes) 000299 1788870692.933411 MIRROR_RESYNC 6 mirrors 000300 1788870694.671247 trunc from 0x02a5cc to 0x033a1f (0x9454 bytes) 000301 1788870694.726397 read 0x01d003 thru 0x0283e7 (0xb3e5 bytes) 000302 1788870694.744049 trunc from 0x03cce4 to 0x03ffff (0x331c bytes) 000303 1788870694.751091 mapwrite 0x021bff thru 0x02a5cf (0x89d1 bytes) 000304 1788870694.783260 fallocate 0x02104b thru 0x02e5ac (0xd562 bytes) 000306 1788870694.807577 read 0x03a38c thru 0x03ffff (0x5c74 bytes) 000307 1788870694.831502 fallocate 0x03ffe0 thru 0x03ffff (0x20 bytes) 000308 1788870694.839173 write 0x033d68 thru 0x03faa4 (0xbd3d bytes) 000309 1788870694.840742 trunc from 0x0003ba to 0x00cb35 (0xc77c bytes) 000310 1788870694.847970 fallocate 0x018580 thru 0x0229c9 (0xa44a bytes) 000311 1788870694.855424 mapread 0x03f6f5 thru 0x03ffff (0x90b bytes) 000312 1788870694.857155 mapread 0x03a3e8 thru 0x03a682 (0x29b bytes) 000313 1788870694.859138 trunc from 0x040000 to 0x01a08f (0x25f70 bytes) 000314 1788870694.892507 trunc from 0x0101bf to 0x018c13 (0x8a55 bytes) 000315 1788870694.907257 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a090): Mirror 3: 0x5eb53135 (stale) Mirror 4: 0x5eb53135 (stale) Mirror 5: 0x5eb53135 (stale) Mirror 6: 0x5eb53135 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a090): Mirror 3: 0x5eb53135 (stale) Mirror 4: 0x5eb53135 (stale) Mirror 5: 0x5eb53135 (stale) Mirror 6: 0x5eb53135 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a090: skipped 000316 1788870695.383915 fallocate 0x01af8c thru 0x02a401 (0xf476 bytes) 000318 1788870695.393973 trunc from 0x00402f to 0x012914 (0xe8e6 bytes) 000319 1788870695.404076 write 0x02b917 thru 0x02f70f (0x3df9 bytes) 000320 1788870695.413614 punch from 0x02a011 to 0x02f70f (0x56ff bytes) 000321 1788870695.450065 trunc from 0x02f710 to 0x039e15 (0xa706 bytes) 000322 1788870695.473879 mapread 0x030098 thru 0x034489 (0x43f2 bytes) 000323 1788870695.506030 write 0x034044 thru 0x039b80 (0x5b3d bytes) 000324 1788870695.507876 trunc from 0x027d97 to 0x033543 (0xb7ad bytes) 000325 1788870695.515480 mapread 0x01a581 thru 0x01c75e (0x21de bytes) 000326 1788870695.531981 punch from 0x03051a to 0x039647 (0x912e bytes) 000327 1788870695.548139 mapread 0x01c763 thru 0x01fc28 (0x34c6 bytes) 000328 1788870695.549448 punch from 0x0370f8 to 0x039e15 (0x2d1e bytes) 000329 1788870695.560055 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39e16): Mirror 3: 0x9d9c9cf8 (stale) Mirror 5: 0x9d9c9cf8 (stale) Mirror 6: 0x9d9c9cf8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39e16): Mirror 3: 0x9d9c9cf8 (stale) Mirror 5: 0x9d9c9cf8 (stale) Mirror 6: 0x9d9c9cf8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39e16: skipped 000331 1788870695.972542 punch from 0x0208ed to 0x02d564 (0xcc78 bytes) 000333 1788870695.986003 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x39e16): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39e16): Mirror 3: 0x9d9c9cf8 (stale) Mirror 5: 0x9d9c9cf8 (stale) Mirror 6: 0x9d9c9cf8 (stale) CRC-32 checksum value for chunk [0, 0x39e16): Mirror 1: 0x73a79e3 Mirror 7: 0x73a79e3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39e16: skipped 000334 1788870696.903134 mapread 0x01376c thru 0x01828b (0x4b20 bytes) 000335 1788870696.935411 mapread 0x02eb9d thru 0x03006d (0x14d1 bytes) 000336 1788870696.977055 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x39e16): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39e16): Mirror 3: 0x9d9c9cf8 (stale) Mirror 5: 0x9d9c9cf8 (stale) Mirror 6: 0x9d9c9cf8 (stale) CRC-32 checksum value for chunk [0, 0x39e16): Mirror 1: 0x73a79e3 Mirror 7: 0x73a79e3 Mirror 8: 0x73a79e3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39e16: skipped 000337 1788870697.848778 read 0x0114ae thru 0x01a3e4 (0x8f37 bytes) 000338 1788870697.857942 read 0x01991d thru 0x01d2a6 (0x398a bytes) 000339 1788870697.868198 mapread 0x008b60 thru 0x015dd2 (0xd273 bytes) 000340 1788870697.870263 mapread 0x026757 thru 0x0336f7 (0xcfa1 bytes) 000341 1788870697.871301 mapread 0x015656 thru 0x022f1f (0xd8ca bytes) 000343 1788870697.873449 punch from 0x02ecff to 0x037bbd (0x8ebf bytes) 000346 1788870697.937267 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x39e16): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39e16): Mirror 3: 0x9d9c9cf8 (stale) Mirror 5: 0x9d9c9cf8 (stale) Mirror 6: 0x9d9c9cf8 (stale) Mirror 7: 0x73a79e3 (stale) Mirror 8: 0x73a79e3 (stale) CRC-32 checksum value for chunk [0, 0x39e16): Mirror 1: 0x73a79e3 Mirror 9: 0x73a79e3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39e16: skipped 000347 1788870698.890736 trunc from 0x039e16 to 0x004ab1 (0x35364 bytes) 000348 1788870698.956195 trunc from 0x00481a to 0x004ab1 (0x298 bytes) 000349 1788870698.960986 trunc from 0x004ab2 to 0x03a8ff (0x35e4e bytes) 000351 1788870698.983994 mapwrite 0x016131 thru 0x0258c4 (0xf794 bytes) 000352 1788870699.042566 mapwrite 0x03726c thru 0x03e69e (0x7433 bytes) 000353 1788870699.084575 mapwrite 0x01e3bd thru 0x021296 (0x2eda bytes) 000354 1788870699.111724 mapread 0x013855 thru 0x0142b6 (0xa62 bytes) 000355 1788870699.113081 mapwrite 0x0250d6 thru 0x033d9f (0xecca bytes) 000356 1788870699.158814 mapread 0x033538 thru 0x03a395 (0x6e5e bytes) 000357 1788870699.162022 mapread 0x00a472 thru 0x00f456 (0x4fe5 bytes) 000358 1788870699.179137 mapwrite 0x020d25 thru 0x023cd1 (0x2fad bytes) 000361 1788870699.195891 read 0x0385ba thru 0x03a3dc (0x1e23 bytes) 000362 1788870699.196663 punch from 0x024e3f to 0x02dd29 (0x8eeb bytes) 000363 1788870699.203413 read 0x026e42 thru 0x035bdf (0xed9e bytes) 000364 1788870699.219215 mapread 0x00ce72 thru 0x011b5f (0x4cee bytes) 000365 1788870699.220405 write 0x022708 thru 0x02fbdb (0xd4d4 bytes) 000366 1788870699.224143 fallocate 0x026861 thru 0x03621f (0xf9bf bytes) 000367 1788870699.231965 punch from 0x023675 to 0x027129 (0x3ab5 bytes) 000368 1788870699.248756 punch from 0x035c8d to 0x03d305 (0x7679 bytes) 000370 1788870699.254798 fallocate 0x027205 thru 0x029d6f (0x2b6b bytes) 000372 1788870699.264420 MIRROR_RESYNC 7 mirrors 000373 1788870701.283113 trunc from 0x03e69f to 0x02014e (0x1e550 bytes) 000374 1788870701.367365 punch from 0x00d9bf to 0x013083 (0x56c5 bytes) 000375 1788870701.373770 fallocate 0x00b647 thru 0x015fb5 (0xa96f bytes) 000377 1788870701.381723 mapwrite 0x03d795 thru 0x03ffff (0x286b bytes) 000378 1788870701.437448 punch from 0x031bd2 to 0x03d62a (0xba59 bytes) 000379 1788870701.442754 read 0x033534 thru 0x03dcd9 (0xa7a6 bytes) 000380 1788870701.454298 fallocate 0x02a736 thru 0x02fc75 (0x5540 bytes) 000381 1788870701.460112 trunc from 0x001eb7 to 0x010654 (0xe79e bytes) 000382 1788870701.465910 mapread 0x031237 thru 0x03b6da (0xa4a4 bytes) 000383 1788870701.480292 punch from 0x028e63 to 0x033bac (0xad4a bytes) 000384 1788870701.486157 mapwrite 0x00cf07 thru 0x018bf7 (0xbcf1 bytes) 000385 1788870701.525792 trunc from 0x040000 to 0x01f340 (0x20cbf bytes) 000387 1788870701.545035 write 0x02ca31 thru 0x032a73 (0x6043 bytes) 000388 1788870701.550781 trunc from 0x032a74 to 0x015acf (0x1cfa4 bytes) 000389 1788870701.567651 trunc from 0x012dd4 to 0x01318c (0x3b9 bytes) 000390 1788870701.579030 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15ad0): Mirror 3: 0xd499a692 (stale) Mirror 5: 0xd499a692 (stale) Mirror 6: 0xd499a692 (stale) Mirror 8: 0xd499a692 (stale) Mirror 9: 0xd499a692 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15ad0): Mirror 3: 0xd499a692 (stale) Mirror 5: 0xd499a692 (stale) Mirror 6: 0xd499a692 (stale) Mirror 8: 0xd499a692 (stale) Mirror 9: 0xd499a692 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15ad0: skipped 000391 1788870702.029536 read 0x0069ed thru 0x00e0a8 (0x76bc bytes) 000392 1788870702.044343 trunc from 0x015ad0 to 0x03488a (0x1edbb bytes) 000393 1788870702.067962 trunc from 0x01f8c1 to 0x028bca (0x930a bytes) 000395 1788870702.074659 trunc from 0x03086b to 0x03488a (0x4020 bytes) 000396 1788870702.080752 write 0x0058ce thru 0x0124e9 (0xcc1c bytes) 000397 1788870702.093586 fallocate 0x001bb6 thru 0x002e66 (0x12b1 bytes) 000398 1788870702.103403 punch from 0x02b9bf to 0x03488a (0x8ecc bytes) 000399 1788870702.109404 mapwrite 0x0254de thru 0x026a1f (0x1542 bytes) 000401 1788870702.147600 trunc from 0x03488b to 0x013fef (0x2089b bytes) 000402 1788870702.175710 write 0x03381c thru 0x03ffe7 (0xc7cc bytes) 000403 1788870702.177735 mapread 0x0289d9 thru 0x02f710 (0x6d38 bytes) 000404 1788870702.185380 punch from 0x01b701 to 0x02020f (0x4b0f bytes) 000405 1788870702.189434 write 0x019b14 thru 0x01ae18 (0x1305 bytes) 000406 1788870702.198964 fallocate 0x020cdc thru 0x02bec5 (0xb1ea bytes) 000407 1788870702.203858 trunc from 0x03ffe8 to 0x01a144 (0x25ea3 bytes) 000408 1788870702.219800 trunc from 0x01a145 to 0x03cb60 (0x22a1c bytes) 000409 1788870702.238106 fallocate 0x00d133 thru 0x01c3ed (0xf2bb bytes) 000411 1788870702.243661 fallocate 0x03e0d4 thru 0x03ffff (0x1f2c bytes) 000412 1788870702.249137 mapread 0x00eb57 thru 0x018266 (0x9710 bytes) 000413 1788870702.266778 trunc from 0x03cb61 to 0x03fee1 (0x3381 bytes) 000414 1788870702.298519 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fee2): Mirror 3: 0x1196aa7e (stale) Mirror 5: 0x1196aa7e (stale) Mirror 6: 0x1196aa7e (stale) Mirror 9: 0x1196aa7e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fee2): Mirror 3: 0x1196aa7e (stale) Mirror 5: 0x1196aa7e (stale) Mirror 6: 0x1196aa7e (stale) Mirror 9: 0x1196aa7e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fee2: skipped 000415 1788870702.688036 trunc from 0x03fee2 to 0x01c356 (0x23b8b bytes) 000416 1788870702.714328 punch from 0x0119c2 to 0x01c356 (0xa995 bytes) 000417 1788870702.720318 punch from 0x01bab7 to 0x01c356 (0x8a0 bytes) 000418 1788870702.725890 read 0x00cbd5 thru 0x01764c (0xaa78 bytes) 000419 1788870702.735231 mapwrite 0x01e2b6 thru 0x01f3c8 (0x1113 bytes) 000420 1788870702.774502 fallocate 0x0069ba thru 0x014e7b (0xe4c2 bytes) 000421 1788870702.779881 write 0x000a1d thru 0x00820f (0x77f3 bytes) 000422 1788870702.790740 mapread 0x015f1c thru 0x016da8 (0xe8d bytes) 000424 1788870702.797452 fallocate 0x026594 thru 0x02b585 (0x4ff2 bytes) 000425 1788870702.806919 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b586): Mirror 3: 0x3210f0cf (stale) Mirror 5: 0x3210f0cf (stale) Mirror 9: 0x3210f0cf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b586): Mirror 3: 0x3210f0cf (stale) Mirror 5: 0x3210f0cf (stale) Mirror 9: 0x3210f0cf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b586: skipped 000426 1788870703.115033 write 0x0028c7 thru 0x00ff08 (0xd642 bytes) 000428 1788870703.130674 mapwrite 0x02a676 thru 0x02e380 (0x3d0b bytes) 000429 1788870703.185955 fallocate 0x02fdb1 thru 0x032d22 (0x2f72 bytes) 000430 1788870703.191407 punch from 0x01ca1b to 0x02c5e8 (0xfbce bytes) 000431 1788870703.197520 trunc from 0x032d23 to 0x011f32 (0x20df0 bytes) 000432 1788870703.217815 read 0x011508 thru 0x011f32 (0xa2b bytes) 000433 1788870703.223711 write 0x038188 thru 0x03ffff (0x7e78 bytes) 000434 1788870703.225484 trunc from 0x040000 to 0x01da2d (0x225d2 bytes) 000436 1788870703.243611 punch from 0x017066 to 0x01c1ca (0x5165 bytes) 000437 1788870703.249051 mapwrite 0x030659 thru 0x035473 (0x4e1b bytes) 000438 1788870703.290479 read 0x032440 thru 0x035473 (0x3034 bytes) 000439 1788870703.291618 mapread 0x024544 thru 0x02879e (0x425b bytes) 000440 1788870703.302313 mapread 0x02e6b4 thru 0x03294c (0x4299 bytes) 000443 1788870703.305061 mapwrite 0x013f0f thru 0x01d3a7 (0x9499 bytes) 000444 1788870703.321967 read 0x001a61 thru 0x003b11 (0x20b1 bytes) 000445 1788870703.322729 read 0x02986b thru 0x035473 (0xbc09 bytes) 000446 1788870703.324749 trunc from 0x035474 to 0x005778 (0x2fcfb bytes) 000448 1788870703.343665 mapwrite 0x00d0bd thru 0x01760b (0xa54f bytes) 000449 1788870703.400599 trunc from 0x0065fe to 0x00aca9 (0x46ac bytes) 000450 1788870703.408301 punch from 0x013da6 to 0x01760b (0x3866 bytes) 000451 1788870703.414471 read 0x0094d0 thru 0x00f4ed (0x601e bytes) 000452 1788870703.434818 trunc from 0x01760c to 0x00ca31 (0xabda bytes) 000453 1788870703.461867 read 0x008911 thru 0x00ca31 (0x4121 bytes) 000454 1788870703.463575 trunc from 0x0073d7 to 0x00ba47 (0x4671 bytes) 000455 1788870703.475347 mapwrite 0x013323 thru 0x01e925 (0xb603 bytes) 000456 1788870703.568650 punch from 0x00b611 to 0x01564a (0xa03a bytes) 000457 1788870703.574591 fallocate 0x0133e7 thru 0x018f78 (0x5b92 bytes) 000458 1788870703.579880 mapread 0x003526 thru 0x0092b0 (0x5d8b bytes) 000459 1788870703.581672 punch from 0x01ad7c to 0x01e7eb (0x3a70 bytes) 000460 1788870703.587690 read 0x00574e thru 0x00e40e (0x8cc1 bytes) 000461 1788870703.599706 mapread 0x0099d2 thru 0x015114 (0xb743 bytes) 000462 1788870703.601055 trunc from 0x01b428 to 0x01e925 (0x34fe bytes) 000464 1788870703.606666 mapread 0x00f8cb thru 0x0147a9 (0x4edf bytes) 000465 1788870703.607894 mapread 0x0149e3 thru 0x017e7e (0x349c bytes) 000466 1788870703.610620 trunc from 0x01e926 to 0x03825a (0x19935 bytes) 000468 1788870703.630992 read 0x00e059 thru 0x01443c (0x63e4 bytes) 000470 1788870703.633413 trunc from 0x03825b to 0x03942b (0x11d1 bytes) 000472 1788870703.654338 mapwrite 0x00130b thru 0x0057e3 (0x44d9 bytes) 000473 1788870703.683323 mapread 0x00accf thru 0x012da3 (0x80d5 bytes) 000474 1788870703.687405 fallocate 0x0195e5 thru 0x02925c (0xfc78 bytes) 000475 1788870703.717289 MIRROR_RESYNC 4 mirrors 000476 1788870705.006319 fallocate 0x013e3d thru 0x01ee46 (0xb00a bytes) 000477 1788870705.063026 mapread 0x00a182 thru 0x00bedf (0x1d5e bytes) 000478 1788870705.073169 trunc from 0x021d7a to 0x0304f7 (0xe77e bytes) 000479 1788870705.087262 punch from 0x00d6d7 to 0x01702d (0x9957 bytes) 000480 1788870705.092956 trunc from 0x03942c to 0x01cb73 (0x1c8b8 bytes) 000481 1788870705.123615 punch from 0x0030fe to 0x009285 (0x6188 bytes) 000482 1788870705.142476 mapwrite 0x027286 thru 0x033516 (0xc291 bytes) 000483 1788870705.268659 mapread 0x010c78 thru 0x015b59 (0x4ee2 bytes) 000484 1788870705.286090 mapwrite 0x037e1f thru 0x03a857 (0x2a39 bytes) 000485 1788870705.331540 punch from 0x034606 to 0x03a857 (0x6252 bytes) 000486 1788870705.338754 fallocate 0x021bc7 thru 0x02ac55 (0x908f bytes) 000487 1788870705.343410 read 0x016f13 thru 0x022de0 (0xbece bytes) 000488 1788870705.351917 mapwrite 0x017c4d thru 0x0277fd (0xfbb1 bytes) 000489 1788870705.378980 write 0x03e25a thru 0x03ffff (0x1da6 bytes) 000490 1788870705.380604 fallocate 0x001ef2 thru 0x011a74 (0xfb83 bytes) 000491 1788870705.385539 mapwrite 0x0069f0 thru 0x01052e (0x9b3f bytes) 000492 1788870705.410930 trunc from 0x0102c4 to 0x013ea7 (0x3be4 bytes) 000493 1788870705.416604 write 0x001ce9 thru 0x003699 (0x19b1 bytes) 000494 1788870705.417919 punch from 0x00519c to 0x00b26a (0x60cf bytes) 000496 1788870705.424087 punch from 0x00d110 to 0x01416a (0x705b bytes) 000498 1788870705.431028 trunc from 0x040000 to 0x02b8c3 (0x1473c bytes) 000499 1788870705.451813 mapwrite 0x0179af thru 0x022141 (0xa793 bytes) 000501 1788870705.475011 punch from 0x01e81f to 0x0205e8 (0x1dca bytes) 000502 1788870705.481465 read 0x001759 thru 0x00ee9b (0xd743 bytes) 000503 1788870705.502028 write 0x001481 thru 0x00f23e (0xddbe bytes) 000504 1788870705.503984 fallocate 0x02e659 thru 0x03a09c (0xba44 bytes) 000506 1788870705.512044 fallocate 0x019b2b thru 0x01b7c2 (0x1c98 bytes) 000507 1788870705.526805 mapread 0x0118dc thru 0x01890e (0x7033 bytes) 000508 1788870705.533364 MIRROR_RESYNC 4 mirrors 000509 1788870706.753012 punch from 0x01b97e to 0x028c12 (0xd295 bytes) 000510 1788870706.844022 write 0x00c12f thru 0x019cf1 (0xdbc3 bytes) 000511 1788870706.899580 MIRROR_RESYNC 4 mirrors 000512 1788870707.769718 trunc from 0x02b8c4 to 0x02f6c8 (0x3e05 bytes) 000513 1788870707.819021 mapwrite 0x03e151 thru 0x03ffff (0x1eaf bytes) 000514 1788870707.880224 write 0x034412 thru 0x03eb92 (0xa781 bytes) 000515 1788870707.898348 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x12ab9459 (stale) Mirror 5: 0x12ab9459 (stale) Mirror 9: 0x12ab9459 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x145151f4 Mirror 10: 0x145151f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000516 1788870709.072788 write 0x017e1e thru 0x01cc55 (0x4e38 bytes) 000517 1788870709.125815 trunc from 0x012eb8 to 0x0201cc (0xd315 bytes) 000518 1788870709.143801 MIRROR_RESYNC 5 mirrors 000519 1788870710.677107 read 0x001756 thru 0x0098e2 (0x818d bytes) 000520 1788870710.691307 write 0x03ae50 thru 0x03ffff (0x51b0 bytes) 000521 1788870710.734312 trunc from 0x040000 to 0x001401 (0x3ebfe bytes) 000522 1788870710.769319 trunc from 0x000a92 to 0x001401 (0x970 bytes) 000523 1788870710.787611 read 0x0013e2 thru 0x001401 (0x20 bytes) 000525 1788870710.796315 write 0x03b8c8 thru 0x03ffff (0x4738 bytes) 000526 1788870710.807303 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x459c8168 (stale) Mirror 5: 0x459c8168 (stale) Mirror 9: 0x459c8168 (stale) Mirror 10: 0x459c8168 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc4f0c04 Mirror 11: 0xc4f0c04 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000527 1788870711.892499 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x459c8168 (stale) Mirror 5: 0x459c8168 (stale) Mirror 9: 0x459c8168 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x459c8168 (stale) Mirror 5: 0x459c8168 (stale) Mirror 9: 0x459c8168 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc4f0c04 Mirror 11: 0xc4f0c04 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000528 1788870712.595782 trunc from 0x00d80a to 0x01d7d7 (0xffce bytes) 000529 1788870712.676148 mapread 0x00ad1a thru 0x00bd88 (0x106f bytes) 000530 1788870712.687131 read 0x016b5d thru 0x01abba (0x405e bytes) 000531 1788870712.709677 trunc from 0x03d32c to 0x03ffff (0x2cd4 bytes) 000532 1788870712.718339 trunc from 0x040000 to 0x00ccdc (0x33323 bytes) 000533 1788870712.749209 fallocate 0x0359da thru 0x03d8e8 (0x7f0f bytes) 000535 1788870712.756297 write 0x02b5f3 thru 0x036d29 (0xb737 bytes) 000536 1788870712.767309 trunc from 0x02a534 to 0x02bd22 (0x17ef bytes) 000537 1788870712.785485 fallocate 0x030439 thru 0x039be9 (0x97b1 bytes) 000538 1788870712.792258 mapwrite 0x03b8a7 thru 0x03fad2 (0x422c bytes) 000539 1788870712.865945 read 0x0177ef thru 0x019458 (0x1c6a bytes) 000540 1788870712.889378 trunc from 0x03fad3 to 0x03cbe8 (0x2eea bytes) 000541 1788870712.918321 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cbe9): Mirror 3: 0xc95b3dcd (stale) Mirror 9: 0xc95b3dcd (stale) Mirror 11: 0xfc33615 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cbe9): Mirror 3: 0xc95b3dcd (stale) Mirror 9: 0xc95b3dcd (stale) Mirror 11: 0xfc33615 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cbe9: skipped 000542 1788870713.371667 write 0x03f935 thru 0x03ffff (0x6cb bytes) 000543 1788870713.381917 mapread 0x029525 thru 0x02a812 (0x12ee bytes) 000544 1788870713.403066 fallocate 0x02f262 thru 0x030c9b (0x1a3a bytes) 000545 1788870713.412609 trunc from 0x03a702 to 0x03ae73 (0x772 bytes) 000546 1788870713.419784 mapwrite 0x00bb0d thru 0x00ffb2 (0x44a6 bytes) 000547 1788870713.456301 mapread 0x03a2f5 thru 0x03a531 (0x23d bytes) 000548 1788870713.470343 mapwrite 0x013cbf thru 0x01f4e0 (0xb822 bytes) 000550 1788870713.517536 punch from 0x0017b1 to 0x001982 (0x1d2 bytes) 000551 1788870713.523868 fallocate 0x0255dc thru 0x02fb99 (0xa5be bytes) 000552 1788870713.536046 punch from 0x036897 to 0x038557 (0x1cc1 bytes) 000553 1788870713.550011 trunc from 0x040000 to 0x023409 (0x1cbf6 bytes) 000554 1788870713.571561 trunc from 0x02340a to 0x018af6 (0xa913 bytes) 000555 1788870713.589797 mapread 0x017f23 thru 0x018af6 (0xbd4 bytes) 000556 1788870713.590891 punch from 0x0047ca to 0x012573 (0xddaa bytes) 000558 1788870713.598126 write 0x012df5 thru 0x022bbc (0xfdc8 bytes) 000560 1788870713.607056 mapread 0x012bc0 thru 0x01997c (0x6dbd bytes) 000561 1788870713.608444 mapread 0x014719 thru 0x014bb2 (0x49a bytes) 000562 1788870713.610082 mapwrite 0x019ee2 thru 0x019fbc (0xdb bytes) 000563 1788870713.625758 punch from 0x0070aa to 0x00850a (0x1461 bytes) 000564 1788870713.629959 trunc from 0x022bbd to 0x01ecf9 (0x3ec3 bytes) 000565 1788870713.643784 trunc from 0x01ecfa to 0x0063f8 (0x18901 bytes) 000566 1788870713.665025 mapwrite 0x026897 thru 0x02fe6e (0x95d8 bytes) 000567 1788870713.747010 trunc from 0x015277 to 0x019de8 (0x4b72 bytes) 000568 1788870713.758678 fallocate 0x0328b9 thru 0x03ffff (0xd747 bytes) 000569 1788870713.765644 punch from 0x027d34 to 0x036e68 (0xf135 bytes) 000570 1788870713.773303 read 0x036bf3 thru 0x03ffff (0x940d bytes) 000571 1788870713.793701 mapread 0x03ecab thru 0x03ffff (0x1355 bytes) 000572 1788870713.794951 mapwrite 0x019dad thru 0x027c8a (0xdede bytes) 000573 1788870713.852124 mapread 0x0357f4 thru 0x035ebf (0x6cc bytes) 000574 1788870713.873188 MIRROR_RESYNC 4 mirrors 000575 1788870715.044234 read 0x00727b thru 0x00eff7 (0x7d7d bytes) 000577 1788870715.053665 mapread 0x032b25 thru 0x03ffff (0xd4db bytes) 000579 1788870715.085543 write 0x032857 thru 0x036ec6 (0x4670 bytes) 000580 1788870715.152055 trunc from 0x040000 to 0x033189 (0xce76 bytes) 000581 1788870715.193331 punch from 0x02484e to 0x027a56 (0x3209 bytes) 000582 1788870715.204453 read 0x03022d thru 0x033189 (0x2f5d bytes) 000583 1788870715.229071 write 0x01557e thru 0x01b691 (0x6114 bytes) 000584 1788870715.234615 fallocate 0x02c959 thru 0x03a8fa (0xdfa2 bytes) 000585 1788870715.244599 read 0x016b05 thru 0x01b6b1 (0x4bad bytes) 000586 1788870715.245883 mapwrite 0x0341c6 thru 0x038cf0 (0x4b2b bytes) 000587 1788870715.318859 fallocate 0x01ea04 thru 0x0216be (0x2cbb bytes) 000588 1788870715.329696 mapwrite 0x035d62 thru 0x03ca70 (0x6d0f bytes) 000589 1788870715.399276 trunc from 0x03ca71 to 0x00ca09 (0x30067 bytes) 000590 1788870715.421483 read 0x005102 thru 0x00ca09 (0x7908 bytes) 000591 1788870715.427550 read 0x0049f2 thru 0x00ca09 (0x8018 bytes) 000592 1788870715.436138 trunc from 0x00ca0a to 0x00d08d (0x684 bytes) 000593 1788870715.458451 write 0x034e5c thru 0x0397e3 (0x4988 bytes) 000594 1788870715.469175 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x397e4): Mirror 9: 0x41bbd82c (stale) Mirror 11: 0x41bbd82c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x397e4): Mirror 9: 0x41bbd82c (stale) Mirror 11: 0x41bbd82c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x397e4: skipped 000595 1788870715.864847 trunc from 0x031a2c to 0x0397e3 (0x7db8 bytes) 000596 1788870715.872968 fallocate 0x0288d5 thru 0x028d4b (0x477 bytes) 000597 1788870715.879618 trunc from 0x0341a6 to 0x0397e3 (0x563e bytes) 000598 1788870715.890655 punch from 0x022ee2 to 0x02507b (0x219a bytes) 000599 1788870715.908192 punch from 0x016452 to 0x021ef9 (0xbaa8 bytes) 000600 1788870715.915685 punch from 0x032359 to 0x03864b (0x62f3 bytes) 000601 1788870715.923134 mapread 0x014d92 thru 0x016ca0 (0x1f0f bytes) 000602 1788870715.967415 MIRROR_RESYNC 3 mirrors 000603 1788870716.849197 fallocate 0x00831e thru 0x017744 (0xf427 bytes) 000604 1788870716.913669 read 0x0270bb thru 0x0321c3 (0xb109 bytes) 000605 1788870716.933723 fallocate 0x019e4f thru 0x01b4bc (0x166e bytes) 000606 1788870716.946376 write 0x02a365 thru 0x0302ef (0x5f8b bytes) 000607 1788870716.954251 write 0x03ded4 thru 0x03ffff (0x212c bytes) 000608 1788870716.956680 mapwrite 0x0274bf thru 0x02e3df (0x6f21 bytes) 000609 1788870717.003977 read 0x004cc2 thru 0x01279d (0xdadc bytes) 000610 1788870717.035978 trunc from 0x040000 to 0x02c632 (0x139cd bytes) 000611 1788870717.081445 punch from 0x006123 to 0x009fb3 (0x3e91 bytes) 000612 1788870717.101564 MIRROR_RESYNC 3 mirrors 000613 1788870717.829382 read 0x01bc08 thru 0x027d37 (0xc130 bytes) 000614 1788870717.845367 trunc from 0x00eda6 to 0x01b351 (0xc5ac bytes) 000615 1788870717.898257 mapwrite 0x01d167 thru 0x02275f (0x55f9 bytes) 000616 1788870717.946694 mapwrite 0x01fb03 thru 0x0288f9 (0x8df7 bytes) 000617 1788870718.022487 punch from 0x0166ae to 0x019e37 (0x378a bytes) 000618 1788870718.030663 mapwrite 0x00b40c thru 0x01064a (0x523f bytes) 000619 1788870718.112741 punch from 0x007f0c to 0x015c3c (0xdd31 bytes) 000620 1788870718.150403 mapread 0x013ed6 thru 0x01460a (0x735 bytes) 000621 1788870718.188872 trunc from 0x01000f to 0x01c1a3 (0xc195 bytes) 000622 1788870718.202571 trunc from 0x015e64 to 0x01ee66 (0x9003 bytes) 000623 1788870718.216765 mapread 0x027205 thru 0x027266 (0x62 bytes) 000624 1788870718.218736 write 0x01a744 thru 0x02813c (0xd9f9 bytes) 000625 1788870718.229181 fallocate 0x0087ab thru 0x008ee7 (0x73d bytes) 000626 1788870718.250976 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2c633): Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c633): Mirror 9: 0x48108b50 (stale) Mirror 11: 0x48108b50 (stale) CRC-32 checksum value for chunk [0, 0x2c633): Mirror 1: 0x18fb0ada Mirror 12: 0x18fb0ada lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c633: skipped 000628 1788870719.613568 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c633): Mirror 11: 0x48108b50 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c633): Mirror 11: 0x48108b50 (stale) CRC-32 checksum value for chunk [0, 0x2c633): Mirror 1: 0x18fb0ada Mirror 12: 0x18fb0ada lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c633: skipped 000629 1788870720.619070 trunc from 0x02a058 to 0x02c632 (0x25db bytes) 000630 1788870720.668222 trunc from 0x0177de to 0x01b9b1 (0x41d4 bytes) 000631 1788870720.677122 trunc from 0x02c633 to 0x025578 (0x70ba bytes) 000632 1788870720.744205 fallocate 0x031951 thru 0x03e4e5 (0xcb95 bytes) 000633 1788870720.750381 write 0x030a5a thru 0x033010 (0x25b7 bytes) 000634 1788870720.763542 trunc from 0x0204e3 to 0x022de1 (0x28ff bytes) 000635 1788870720.772544 write 0x03a7cd thru 0x03c4d5 (0x1d09 bytes) 000636 1788870720.784622 fallocate 0x0258dd thru 0x029518 (0x3c3c bytes) 000639 1788870720.792494 fallocate 0x031045 thru 0x03bada (0xaa96 bytes) 000641 1788870720.799301 mapread 0x02ab07 thru 0x02df16 (0x3410 bytes) 000642 1788870720.835686 MIRROR_RESYNC 3 mirrors 000643 1788870721.966071 mapread 0x028a61 thru 0x037278 (0xe818 bytes) 000644 1788870722.011961 fallocate 0x016aef thru 0x023d7a (0xd28c bytes) 000645 1788870722.041967 write 0x038b40 thru 0x03c23c (0x36fd bytes) 000647 1788870722.062690 mapread 0x01eab6 thru 0x027397 (0x88e2 bytes) 000648 1788870722.093046 punch from 0x017d9d to 0x01a14a (0x23ae bytes) 000649 1788870722.101165 trunc from 0x019f77 to 0x01f21c (0x52a6 bytes) 000650 1788870722.107121 mapwrite 0x011ec0 thru 0x0207e3 (0xe924 bytes) 000651 1788870722.206202 write 0x00f38a thru 0x0143e0 (0x5057 bytes) 000652 1788870722.208465 mapread 0x01cb84 thru 0x023bc7 (0x7044 bytes) 000653 1788870722.211259 write 0x035105 thru 0x039b15 (0x4a11 bytes) 000654 1788870722.220493 mapwrite 0x009b8b thru 0x00a072 (0x4e8 bytes) 000655 1788870722.281126 trunc from 0x03d3fc to 0x03e4e5 (0x10ea bytes) 000656 1788870722.289386 punch from 0x01560e to 0x016e3d (0x1830 bytes) 000658 1788870722.295612 fallocate 0x0314de thru 0x03ffff (0xeb22 bytes) 000659 1788870722.300912 read 0x03f827 thru 0x03ffff (0x7d9 bytes) 000660 1788870722.306620 read 0x014b2d thru 0x015899 (0xd6d bytes) 000661 1788870722.318652 read 0x02602b thru 0x02ae3d (0x4e13 bytes) 000662 1788870722.319942 read 0x007421 thru 0x00832e (0xf0e bytes) 000663 1788870722.320613 mapwrite 0x01029a thru 0x01a767 (0xa4ce bytes) 000665 1788870722.338742 write 0x00ccd5 thru 0x00d85a (0xb86 bytes) 000666 1788870722.344415 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xd9f49bc8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xd9f49bc8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000667 1788870722.648635 trunc from 0x040000 to 0x02afa1 (0x1505e bytes) 000668 1788870722.672067 write 0x004a2f thru 0x00f517 (0xaae9 bytes) 000669 1788870722.683516 trunc from 0x00ce50 to 0x01a044 (0xd1f5 bytes) 000670 1788870722.709951 punch from 0x00f3f1 to 0x01bbcd (0xc7dd bytes) 000671 1788870722.725557 mapwrite 0x022506 thru 0x02ca29 (0xa524 bytes) 000672 1788870722.871731 trunc from 0x02ca2a to 0x01b2e4 (0x11745 bytes) 000674 1788870722.896517 trunc from 0x00be4b to 0x00f36c (0x3522 bytes) 000675 1788870722.904150 punch from 0x017d4d to 0x01b2e4 (0x3598 bytes) 000676 1788870722.908963 fallocate 0x011bc9 thru 0x0156b0 (0x3ae8 bytes) 000677 1788870722.914235 trunc from 0x01b2e5 to 0x01c48e (0x11aa bytes) 000678 1788870722.936125 fallocate 0x02e150 thru 0x02e708 (0x5b9 bytes) 000679 1788870722.942557 fallocate 0x0275eb thru 0x036826 (0xf23c bytes) 000680 1788870722.948228 fallocate 0x0384f0 thru 0x03ffff (0x7b10 bytes) 000681 1788870722.954983 fallocate 0x011871 thru 0x01b8af (0xa03f bytes) 000682 1788870722.962657 punch from 0x00c5db to 0x00ef37 (0x295d bytes) 000683 1788870722.969569 read 0x0214c6 thru 0x02893c (0x7477 bytes) 000685 1788870722.985578 trunc from 0x01f5f7 to 0x0256d9 (0x60e3 bytes) 000686 1788870722.995426 read 0x02ceba thru 0x036826 (0x996d bytes) 000687 1788870723.016922 fallocate 0x038b42 thru 0x03ffff (0x74be bytes) 000688 1788870723.023547 punch from 0x016e6b to 0x020485 (0x961b bytes) 000689 1788870723.035285 MIRROR_RESYNC 2 mirrors 000690 1788870723.813012 write 0x036326 thru 0x03ffff (0x9cda bytes) 000691 1788870723.877662 fallocate 0x037459 thru 0x03ffff (0x8ba7 bytes) 000692 1788870723.912297 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17b:0x0]: Invalid argument (22) 000693 1788870724.217207 write 0x03c982 thru 0x03ffff (0x367e bytes) 000694 1788870724.229582 read 0x00e0ae thru 0x00f63a (0x158d bytes) 000695 1788870724.240536 mapwrite 0x02c954 thru 0x038c44 (0xc2f1 bytes) 000696 1788870724.315805 punch from 0x034831 to 0x038cd0 (0x44a0 bytes) 000697 1788870724.323518 mapread 0x032f67 thru 0x035f0d (0x2fa7 bytes) 000698 1788870724.334152 trunc from 0x040000 to 0x010430 (0x2fbcf bytes) 000699 1788870724.353980 fallocate 0x003412 thru 0x012a50 (0xf63f bytes) 000701 1788870724.362239 read 0x00c68d thru 0x01006f (0x39e3 bytes) 000702 1788870724.373306 punch from 0x00a3ae to 0x010430 (0x6083 bytes) 000703 1788870724.405180 MIRROR_EXTEND to 2 mirrors 000706 1788870724.929389 trunc from 0x010431 to 0x020807 (0x103d7 bytes) 000707 1788870725.033514 read 0x018691 thru 0x020807 (0x8177 bytes) 000708 1788870725.045450 mapwrite 0x01b872 thru 0x01e0fd (0x288c bytes) 000709 1788870725.077573 punch from 0x002ba9 to 0x00cb98 (0x9ff0 bytes) 000710 1788870725.084669 trunc from 0x016b60 to 0x01ea2b (0x7ecc bytes) 000711 1788870725.093975 mapwrite 0x01f984 thru 0x02d9f9 (0xe076 bytes) 000713 1788870725.182024 fallocate 0x039f03 thru 0x03ca34 (0x2b32 bytes) 000714 1788870725.188263 write 0x038628 thru 0x03ffff (0x79d8 bytes) 000715 1788870725.194205 trunc from 0x040000 to 0x0381a8 (0x7e57 bytes) 000716 1788870725.235371 trunc from 0x0381a9 to 0x01d5ac (0x1abfc bytes) 000717 1788870725.273404 fallocate 0x014fea thru 0x01ee63 (0x9e7a bytes) 000718 1788870725.290325 write 0x01828c thru 0x022aa1 (0xa816 bytes) 000719 1788870725.296348 mapwrite 0x0348a2 thru 0x03788d (0x2fec bytes) 000720 1788870725.402917 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17b:0x0]: Invalid argument (22) 000722 1788870725.658166 trunc from 0x001619 to 0x00f7d4 (0xe1bc bytes) 000723 1788870725.663283 trunc from 0x03788e to 0x02eb65 (0x8d28 bytes) 000725 1788870725.691147 mapwrite 0x02f7d2 thru 0x03e487 (0xecb6 bytes) 000729 1788870725.768299 mapread 0x03842d thru 0x03e487 (0x605b bytes) 000730 1788870725.772056 write 0x011ed0 thru 0x012617 (0x748 bytes) 000731 1788870725.784652 read 0x01a793 thru 0x01d112 (0x2980 bytes) 000733 1788870725.793322 fallocate 0x003f4b thru 0x008ef8 (0x4fae bytes) 000734 1788870725.799347 write 0x035ea4 thru 0x03ffff (0xa15c bytes) 000735 1788870725.801463 punch from 0x0217f5 to 0x0272e8 (0x5af4 bytes) 000736 1788870725.807439 trunc from 0x019874 to 0x01f872 (0x5fff bytes) 000738 1788870725.815046 read 0x02ae23 thru 0x0377e8 (0xc9c6 bytes) 000739 1788870725.816059 mapread 0x0014b9 thru 0x0026ea (0x1232 bytes) 000740 1788870725.829090 trunc from 0x040000 to 0x00e75c (0x318a3 bytes) 000741 1788870725.844501 read 0x00bd82 thru 0x00e75c (0x29db bytes) 000743 1788870725.848471 read 0x008450 thru 0x00e75c (0x630d bytes) 000745 1788870725.850275 write 0x0058ac thru 0x005deb (0x540 bytes) 000746 1788870725.852439 mapread 0x00d1df thru 0x00dbb0 (0x9d2 bytes) 000747 1788870725.854319 read 0x003ed6 thru 0x00e75c (0xa887 bytes) 000748 1788870725.855548 read 0x00808d thru 0x00e75c (0x66d0 bytes) 000749 1788870725.856662 fallocate 0x02e713 thru 0x03cbb7 (0xe4a5 bytes) 000750 1788870725.862326 fallocate 0x008cb1 thru 0x012ae0 (0x9e30 bytes) 000751 1788870725.870048 read 0x03c9bf thru 0x03cbb7 (0x1f9 bytes) 000752 1788870725.880787 mapwrite 0x02dd09 thru 0x03292a (0x4c22 bytes) 000753 1788870725.900294 mapwrite 0x0046f8 thru 0x00aceb (0x65f4 bytes) 000756 1788870725.924420 mapread 0x01fd49 thru 0x025998 (0x5c50 bytes) 000757 1788870725.926872 mapread 0x0206ea thru 0x02f5ff (0xef16 bytes) 000758 1788870725.928568 write 0x0240c4 thru 0x031f3f (0xde7c bytes) 000759 1788870725.931659 trunc from 0x00dd07 to 0x0118ce (0x3bc8 bytes) 000760 1788870725.937381 punch from 0x033776 to 0x03a8de (0x7169 bytes) 000761 1788870725.942668 mapwrite 0x01b490 thru 0x02061a (0x518b bytes) 000762 1788870725.962905 punch from 0x02eaaa to 0x032451 (0x39a8 bytes) 000763 1788870725.968935 mapread 0x03b5ac thru 0x03cbb7 (0x160c bytes) 000764 1788870725.970947 punch from 0x02e9f0 to 0x03bf16 (0xd527 bytes) 000765 1788870725.978150 trunc from 0x03cbb8 to 0x036c28 (0x5f8f bytes) 000766 1788870725.997398 trunc from 0x036c29 to 0x0394be (0x2896 bytes) 000767 1788870726.014067 trunc from 0x0394bf to 0x02ad6a (0xe754 bytes) 000769 1788870726.032207 fallocate 0x00342d thru 0x00a3a6 (0x6f7a bytes) 000771 1788870726.039949 trunc from 0x01587d to 0x0203ee (0xab72 bytes) 000772 1788870726.045687 fallocate 0x028e65 thru 0x037837 (0xe9d3 bytes) 000774 1788870726.055002 trunc from 0x037838 to 0x03e647 (0x6e10 bytes) 000776 1788870726.071952 write 0x01cce2 thru 0x0254a6 (0x87c5 bytes) 000777 1788870726.093212 punch from 0x02c90d to 0x032790 (0x5e84 bytes) 000778 1788870726.098641 read 0x01237b thru 0x016b53 (0x47d9 bytes) 000779 1788870726.119038 write 0x02d06e thru 0x02d56b (0x4fe bytes) 000781 1788870726.122851 trunc from 0x03487c to 0x03e647 (0x9dcc bytes) 000782 1788870726.131119 fallocate 0x01a7e3 thru 0x02225e (0x7a7c bytes) 000783 1788870726.137610 fallocate 0x03f704 thru 0x03ffff (0x8fc bytes) 000784 1788870726.144017 trunc from 0x03e648 to 0x038710 (0x5f37 bytes) 000786 1788870726.168235 punch from 0x037c22 to 0x038710 (0xaef bytes) 000787 1788870726.175316 trunc from 0x038711 to 0x019fbf (0x1e751 bytes) 000788 1788870726.199410 write 0x03caa9 thru 0x03ffff (0x3557 bytes) 000789 1788870726.203628 trunc from 0x040000 to 0x021a7b (0x1e584 bytes) 000790 1788870726.226304 trunc from 0x01f249 to 0x021a7b (0x2833 bytes) 000792 1788870726.232332 punch from 0x00623e to 0x0104be (0xa281 bytes) 000794 1788870726.240395 trunc from 0x012d53 to 0x020974 (0xdc22 bytes) 000795 1788870726.245798 punch from 0x0064ac to 0x015b11 (0xf666 bytes) 000796 1788870726.254517 punch from 0x0209ee to 0x021a7b (0x108e bytes) 000797 1788870726.261458 mapwrite 0x03c59b thru 0x03ffff (0x3a65 bytes) 000798 1788870726.306700 trunc from 0x040000 to 0x0302d4 (0xfd2b bytes) 000799 1788870726.325294 mapwrite 0x02251a thru 0x02ac28 (0x870f bytes) 000801 1788870726.399219 read 0x02654e thru 0x0302d4 (0x9d87 bytes) 000802 1788870726.421698 punch from 0x01a183 to 0x01ae61 (0xcdf bytes) 000803 1788870726.445065 MIRROR_EXTEND to 2 mirrors 000804 1788870727.102314 fallocate 0x01ee43 thru 0x01fd94 (0xf52 bytes) 000805 1788870727.137326 trunc from 0x00b26c to 0x0172a5 (0xc03a bytes) 000806 1788870727.143783 write 0x03dcb6 thru 0x03ffff (0x234a bytes) 000807 1788870727.149557 fallocate 0x03618d thru 0x03a942 (0x47b6 bytes) 000808 1788870727.156435 trunc from 0x040000 to 0x0276e4 (0x1891b bytes) 000809 1788870727.205181 trunc from 0x0276e5 to 0x00f28e (0x18456 bytes) 000810 1788870727.224567 mapread 0x0030b5 thru 0x008d27 (0x5c73 bytes) 000811 1788870727.248197 trunc from 0x000feb to 0x00f28e (0xe2a4 bytes) 000812 1788870727.255133 trunc from 0x00f28f to 0x00baf8 (0x3796 bytes) 000813 1788870727.278473 MIRROR_RESYNC 2 mirrors 000816 1788870727.844190 mapwrite 0x0232ef thru 0x031c23 (0xe935 bytes) 000817 1788870727.962926 trunc from 0x031c24 to 0x01bb66 (0x160bd bytes) 000818 1788870727.986970 mapread 0x016354 thru 0x01bb66 (0x5813 bytes) 000819 1788870728.001713 fallocate 0x0091c7 thru 0x014078 (0xaeb2 bytes) 000820 1788870728.010645 trunc from 0x01bb67 to 0x01b28e (0x8d8 bytes) 000821 1788870728.036530 trunc from 0x014f17 to 0x01b28e (0x6378 bytes) 000822 1788870728.043950 trunc from 0x01b28f to 0x0198ca (0x19c4 bytes) 000823 1788870728.065503 trunc from 0x0198cb to 0x011986 (0x7f44 bytes) 000825 1788870728.089785 punch from 0x00c150 to 0x00e899 (0x274a bytes) 000826 1788870728.096889 mapread 0x00aa2c thru 0x00f2b5 (0x488a bytes) 000827 1788870728.108376 punch from 0x00a25f to 0x00afc1 (0xd63 bytes) 000828 1788870728.113428 fallocate 0x01032f thru 0x0148c4 (0x4596 bytes) 000829 1788870728.119197 mapwrite 0x03264d thru 0x035a9a (0x344e bytes) 000830 1788870728.163895 mapwrite 0x02b80f thru 0x02ed59 (0x354b bytes) 000831 1788870728.178065 mapread 0x01aeb0 thru 0x01b283 (0x3d4 bytes) 000832 1788870728.191387 mapwrite 0x004248 thru 0x00af1d (0x6cd6 bytes) 000834 1788870728.219949 fallocate 0x034f81 thru 0x03ffff (0xb07f bytes) 000835 1788870728.224360 punch from 0x01d973 to 0x026f0b (0x9599 bytes) 000836 1788870728.229388 mapread 0x0245e6 thru 0x0325ec (0xe007 bytes) 000837 1788870728.239369 punch from 0x020b34 to 0x02489b (0x3d68 bytes) 000838 1788870728.244248 punch from 0x020068 to 0x0224f9 (0x2492 bytes) 000840 1788870728.254255 trunc from 0x011654 to 0x01d9b0 (0xc35d bytes) 000841 1788870728.258954 fallocate 0x02dee9 thru 0x02e840 (0x958 bytes) 000842 1788870728.272756 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x35a9b): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35a9b): Mirror 2: 0x2a226a4e (stale) CRC-32 checksum value for chunk [0, 0x35a9b): Mirror 1: 0xa5e8c85c Mirror 3: 0xa5e8c85c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35a9b: skipped 000843 1788870729.194031 fallocate 0x0245dd thru 0x026fc3 (0x29e7 bytes) 000844 1788870729.231549 trunc from 0x035a9b to 0x037e5a (0x23c0 bytes) 000845 1788870729.276223 fallocate 0x036713 thru 0x03ee0c (0x86fa bytes) 000846 1788870729.283177 fallocate 0x00d588 thru 0x0157e6 (0x825f bytes) 000847 1788870729.287803 trunc from 0x036034 to 0x03ee0c (0x8dd9 bytes) 000848 1788870729.296433 trunc from 0x01fc23 to 0x0288e7 (0x8cc5 bytes) 000849 1788870729.301590 write 0x01b09a thru 0x028092 (0xcff9 bytes) 000850 1788870729.313437 mapread 0x0041a1 thru 0x004f68 (0xdc8 bytes) 000851 1788870729.319803 mapwrite 0x03bfc7 thru 0x03ffff (0x4039 bytes) 000853 1788870729.366570 mapread 0x0320a1 thru 0x03dad7 (0xba37 bytes) 000854 1788870729.377107 trunc from 0x002f60 to 0x01060a (0xd6ab bytes) 000855 1788870729.382117 punch from 0x03e324 to 0x03ffff (0x1cdc bytes) 000856 1788870729.387107 punch from 0x0220d6 to 0x02cc3c (0xab67 bytes) 000857 1788870729.392155 mapread 0x007376 thru 0x00e8ec (0x7577 bytes) 000858 1788870729.402075 fallocate 0x011050 thru 0x01cbd9 (0xbb8a bytes) 000859 1788870729.406928 punch from 0x030f7c to 0x03cb49 (0xbbce bytes) 000860 1788870729.412186 mapread 0x036be9 thru 0x03ffff (0x9417 bytes) 000861 1788870729.431487 mapwrite 0x03adc8 thru 0x03ffff (0x5238 bytes) 000863 1788870729.450685 read 0x02f351 thru 0x03260e (0x32be bytes) 000864 1788870729.451376 trunc from 0x029306 to 0x038723 (0xf41e bytes) 000865 1788870729.456484 write 0x03f4c1 thru 0x03ffff (0xb3f bytes) 000866 1788870729.458391 trunc from 0x040000 to 0x03b31c (0x4ce3 bytes) 000867 1788870729.478572 mapwrite 0x013536 thru 0x019288 (0x5d53 bytes) 000868 1788870729.508097 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b31d): Mirror 3: 0x59892006 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b31d): Mirror 3: 0x59892006 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b31d: skipped 000869 1788870729.882633 mapwrite 0x01de54 thru 0x028896 (0xaa43 bytes) 000870 1788870729.970097 punch from 0x02bea3 to 0x037286 (0xb3e4 bytes) 000871 1788870729.986088 mapread 0x007cc8 thru 0x015fe7 (0xe320 bytes) 000872 1788870730.011932 mapread 0x00f1ba thru 0x01360d (0x4454 bytes) 000873 1788870730.013012 read 0x0395dd thru 0x03b31c (0x1d40 bytes) 000874 1788870730.020055 read 0x0191a4 thru 0x022156 (0x8fb3 bytes) 000875 1788870730.021440 fallocate 0x0398ff thru 0x03ffff (0x6701 bytes) 000876 1788870730.027159 punch from 0x0019fd to 0x0084a2 (0x6aa6 bytes) 000877 1788870730.033180 write 0x03bcc0 thru 0x03ffff (0x4340 bytes) 000878 1788870730.041195 write 0x01a129 thru 0x021be5 (0x7abd bytes) 000879 1788870730.042467 fallocate 0x02cc01 thru 0x03c7b6 (0xfbb6 bytes) 000880 1788870730.058147 mapwrite 0x0273b5 thru 0x03627c (0xeec8 bytes) 000881 1788870730.135463 trunc from 0x024283 to 0x032810 (0xe58e bytes) 000882 1788870730.146725 mapwrite 0x03bd43 thru 0x03ffff (0x42bd bytes) 000884 1788870730.208921 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xbcf0bd3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x82e82358 Mirror 4: 0x82e82358 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000885 1788870731.751582 punch from 0x00508c to 0x0141dd (0xf152 bytes) 000886 1788870731.840983 punch from 0x028a7c to 0x0368c4 (0xde49 bytes) 000887 1788870731.864399 fallocate 0x03b545 thru 0x03ffff (0x4abb bytes) 000888 1788870731.870221 fallocate 0x0332be thru 0x03ea4f (0xb792 bytes) 000889 1788870731.875406 trunc from 0x00f3b3 to 0x018f78 (0x9bc6 bytes) 000890 1788870731.881809 mapread 0x004028 thru 0x00bb0b (0x7ae4 bytes) 000891 1788870731.929860 fallocate 0x03a5e1 thru 0x03fdc2 (0x57e2 bytes) 000893 1788870731.942883 trunc from 0x02bcea to 0x030928 (0x4c3f bytes) 000894 1788870731.966248 mapwrite 0x002e7d thru 0x00368d (0x811 bytes) 000895 1788870731.984424 read 0x027a16 thru 0x02ae45 (0x3430 bytes) 000896 1788870731.998903 read 0x0090ed thru 0x0094a4 (0x3b8 bytes) 000897 1788870731.999644 trunc from 0x040000 to 0x0072a6 (0x38d59 bytes) 000898 1788870732.049283 mapread 0x00626f thru 0x0072a6 (0x1038 bytes) 000899 1788870732.050370 read 0x0020d1 thru 0x006e4e (0x4d7e bytes) 000900 1788870732.054809 mapread 0x0010fb thru 0x004866 (0x376c bytes) 000901 1788870732.056403 read 0x002a00 thru 0x0070e8 (0x46e9 bytes) 000902 1788870732.063080 write 0x00b62e thru 0x00cb39 (0x150c bytes) 000903 1788870732.064321 mapread 0x000e45 thru 0x00cb39 (0xbcf5 bytes) 000904 1788870732.079332 trunc from 0x00cb3a to 0x02d8fa (0x20dc1 bytes) 000905 1788870732.098121 trunc from 0x02b317 to 0x02d8fa (0x25e4 bytes) 000906 1788870732.105496 trunc from 0x02d8fb to 0x000bba (0x2cd40 bytes) 000907 1788870732.155696 fallocate 0x0133b5 thru 0x01b611 (0x825d bytes) 000908 1788870732.162038 mapwrite 0x03c1b9 thru 0x03ffff (0x3e47 bytes) 000910 1788870732.226428 fallocate 0x031e86 thru 0x03cbc4 (0xad3f bytes) 000911 1788870732.238635 trunc from 0x03cc1f to 0x03ffff (0x33e1 bytes) 000912 1788870732.253365 fallocate 0x01fa6b thru 0x024444 (0x49da bytes) 000913 1788870732.264264 punch from 0x02c701 to 0x03897b (0xc27b bytes) 000914 1788870732.275279 MIRROR_RESYNC 3 mirrors 000915 1788870733.236988 read 0x036937 thru 0x039d75 (0x343f bytes) 000916 1788870733.249439 trunc from 0x02ee6f to 0x03733d (0x84cf bytes) 000917 1788870733.300651 read 0x028a8d thru 0x0299bc (0xf30 bytes) 000918 1788870733.327810 trunc from 0x02432b to 0x02e350 (0xa026 bytes) 000919 1788870733.333798 mapread 0x02ac92 thru 0x0390a6 (0xe415 bytes) 000920 1788870733.379089 punch from 0x03ee55 to 0x03ffff (0x11ab bytes) 000921 1788870733.387755 write 0x0278ff thru 0x02a2ae (0x29b0 bytes) 000922 1788870733.395963 trunc from 0x040000 to 0x02e8b3 (0x1174c bytes) 000924 1788870733.451477 mapwrite 0x01ed16 thru 0x02b236 (0xc521 bytes) 000925 1788870733.509744 trunc from 0x02e8b4 to 0x02fb6e (0x12bb bytes) 000926 1788870733.530476 punch from 0x02f660 to 0x02fb6e (0x50f bytes) 000928 1788870733.536794 trunc from 0x02fb6f to 0x01d4f4 (0x1267a bytes) 000930 1788870733.563213 MIRROR_RESYNC 3 mirrors 000931 1788870734.190047 write 0x00e488 thru 0x01adbe (0xc937 bytes) 000932 1788870734.245138 trunc from 0x0122a6 to 0x017c81 (0x59dc bytes) 000933 1788870734.270010 read 0x00452a thru 0x00b253 (0x6d2a bytes) 000934 1788870734.280136 write 0x02b7b7 thru 0x033d50 (0x859a bytes) 000935 1788870734.282336 punch from 0x012d5e to 0x020a5a (0xdcfd bytes) 000936 1788870734.288144 trunc from 0x014396 to 0x01a59c (0x6207 bytes) 000937 1788870734.294812 mapwrite 0x00dde0 thru 0x01a63c (0xc85d bytes) 000938 1788870734.337900 mapread 0x013726 thru 0x01b5d4 (0x7eaf bytes) 000940 1788870734.340349 trunc from 0x028873 to 0x033d50 (0xb4de bytes) 000941 1788870734.345480 mapwrite 0x027760 thru 0x03702b (0xf8cc bytes) 000942 1788870734.425350 mapread 0x00d2bd thru 0x00f69b (0x23df bytes) 000943 1788870734.431496 MIRROR_RESYNC 3 mirrors 000945 1788870735.211298 trunc from 0x03702c to 0x0247c7 (0x12864 bytes) 000946 1788870735.350179 trunc from 0x0247c8 to 0x037ecc (0x13705 bytes) 000947 1788870735.399947 mapwrite 0x0042b2 thru 0x00d167 (0x8eb6 bytes) 000948 1788870735.480987 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37ecd): Mirror 3: 0xd912ca68 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37ecd): Mirror 3: 0xd912ca68 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37ecd: skipped 000949 1788870735.807486 trunc from 0x006d1a to 0x0078ad (0xb94 bytes) 000950 1788870735.813298 trunc from 0x00bdff to 0x017091 (0xb293 bytes) 000952 1788870735.819995 fallocate 0x009088 thru 0x00c491 (0x340a bytes) 000953 1788870735.827101 mapread 0x00743d thru 0x008160 (0xd24 bytes) 000954 1788870735.837099 punch from 0x00d41a to 0x013a5a (0x6641 bytes) 000955 1788870735.846028 MIRROR_RESYNC 2 mirrors 000957 1788870736.524309 trunc from 0x01f27e to 0x0278c4 (0x8647 bytes) 000958 1788870736.565794 fallocate 0x02bf71 thru 0x030fe5 (0x5075 bytes) 000959 1788870736.572646 mapread 0x009fcb thru 0x017cbc (0xdcf2 bytes) 000960 1788870736.606179 MIRROR_RESYNC 2 mirrors 000961 1788870737.515949 mapread 0x019ef0 thru 0x0275a9 (0xd6ba bytes) 000962 1788870737.560981 read 0x03378d thru 0x03536a (0x1bde bytes) 000963 1788870737.571387 write 0x037f07 thru 0x038fb2 (0x10ac bytes) 000964 1788870737.625597 fallocate 0x017130 thru 0x0246c0 (0xd591 bytes) 000965 1788870737.633204 punch from 0x030efb to 0x038fb2 (0x80b8 bytes) 000966 1788870737.655090 fallocate 0x037e23 thru 0x03ffff (0x81dd bytes) 000968 1788870737.662140 write 0x0284f8 thru 0x02948f (0xf98 bytes) 000969 1788870737.676614 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x729cf205 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1ac0566e Mirror 4: 0x1ac0566e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000971 1788870738.689552 MIRROR_RESYNC 3 mirrors 000972 1788870739.460977 trunc from 0x032ded to 0x03e957 (0xbb6b bytes) 000973 1788870739.497703 mapwrite 0x014ec7 thru 0x016cbf (0x1df9 bytes) 000975 1788870739.535283 mapwrite 0x03a912 thru 0x03c242 (0x1931 bytes) 000976 1788870739.560914 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1ac0566e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1ac0566e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000977 1788870739.866797 mapread 0x0074ce thru 0x014beb (0xd71e bytes) 000978 1788870739.898265 punch from 0x01a729 to 0x0271ab (0xca83 bytes) 000980 1788870739.904690 write 0x032d63 thru 0x03bff5 (0x9293 bytes) 000981 1788870739.923089 mapwrite 0x029ca2 thru 0x030f3b (0x729a bytes) 000982 1788870739.968700 read 0x03b834 thru 0x03dbe7 (0x23b4 bytes) 000983 1788870739.975217 mapread 0x03dc2f thru 0x03f715 (0x1ae7 bytes) 000985 1788870739.988058 mapwrite 0x004037 thru 0x009d80 (0x5d4a bytes) 000986 1788870740.014762 trunc from 0x024b3d to 0x025dc0 (0x1284 bytes) 000987 1788870740.022553 write 0x0105f2 thru 0x010ce2 (0x6f1 bytes) 000988 1788870740.024633 mapwrite 0x02eacf thru 0x033fbc (0x54ee bytes) 000989 1788870740.054655 write 0x00cce9 thru 0x01a364 (0xd67c bytes) 000990 1788870740.063161 fallocate 0x035e04 thru 0x03ffff (0xa1fc bytes) 000991 1788870740.071311 trunc from 0x00028a to 0x00da06 (0xd77d bytes) 000992 1788870740.091037 trunc from 0x000c94 to 0x010b4e (0xfebb bytes) 000993 1788870740.099157 mapread 0x012278 thru 0x01b00b (0x8d94 bytes) 000995 1788870740.115391 trunc from 0x000b71 to 0x003b16 (0x2fa6 bytes) 000996 1788870740.127020 mapread 0x031d14 thru 0x036462 (0x474f bytes) 000997 1788870740.130336 punch from 0x02adca to 0x02b524 (0x75b bytes) 000998 1788870740.140354 punch from 0x003de2 to 0x00ae8b (0x70aa bytes) 000999 1788870740.146598 read 0x02a393 thru 0x02f81f (0x548d bytes) 001000 1788870740.157617 trunc from 0x02de46 to 0x031f54 (0x410f bytes) All operations completed A-OK! PASS 70a (85s) == sanity-flr test 71: check mirror extend parent fid ==== 08:32:25 (1788870745) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x1b0:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (10s) == sanity-flr test 72: check append on FLR file ========== 08:32:35 (1788870755) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.316607 s, 6.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124714 s, 8.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x117:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 08:32:45 (1788870765) Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre2 Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 2153664 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2833760 .. done Extending file size to 2835168 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7522880 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5894112 .. Extending file size to 8039552 .. Extending file size to 9065856 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8737856 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9262816 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9136640 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61306 61307 61309 61311 61312 done 192.168.204.142@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg442-client.virtnet /mnt/lustre2 (opts:) 192.168.204.142@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg442-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 08:34:04 (1788870844) Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre2 Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5558496 .. Extending file size to 5449536 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5694400 .. Extending file size to 8191392 .. Extending file size to 8946272 .. Extending file size to 7984192 .. Extending file size to 9359008 .. Extending file size to 8505312 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8750560 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8938368 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8939104 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 62895 62897 62898 62899 62901 done 192.168.204.142@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg442-client.virtnet /mnt/lustre2 (opts:) 192.168.204.142@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg442-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 08:35:24 (1788870924) Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2272 1269032 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2176 1269128 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26072 3580948 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18064 3588956 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 44136 7169904 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2272 1269032 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2176 1269128 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 26072 3580948 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18064 3588956 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 44136 7169904 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0784732 s, 26.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.300011 s, 7.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x90e07176 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.142@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg442-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 08:35:45 (1788870945) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.125545 s, 16.7 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 08:35:52 (1788870952) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x63:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 08:35:58 (1788870958) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0930279 s, 22.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 08:36:05 (1788870965) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0868203 s, 24.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 08:36:12 (1788870972) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0928121 s, 22.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } PASS 204c (31s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 08:36:43 (1788871003) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg442-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498137 s, 21.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x219:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x219:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21a:0x0] } PASS 204d (51s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 08:37:34 (1788871054) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg442-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0873595 s, 24.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (51s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 08:38:25 (1788871105) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg442-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg442-server: Pool lustre.test_204f created Adding targets to pool oleg442-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg442-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0911077 s, 23.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg442-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg442-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg442-server: Pool lustre.test_204f destroyed PASS 204f (73s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 08:39:38 (1788871178) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 08:39:46 (1788871186) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 08:39:54 (1788871194) oleg442-server: Pool lustre.test_206 created Adding targets to pool oleg442-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg442-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg442-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg442-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg442-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 08:40:21 (1788871221) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718383 s, 14.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470265 s, 22.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xc15ea32f Mirror 3: 0xc15ea32f PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 08:40:32 (1788871232) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 08:40:36 (1788871236) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 08:40:39 (1788871239) Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0903267 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2490000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x11b0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x11b0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x11b0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xbb0000 Read this (4 bytes): BBBB 192.168.204.142@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg442-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 08:41:18 (1788871278) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.031257 s, 262 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 08:41:25 (1788871285) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673715 s, 15.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x84 lmm_fid: [0x200000405:0x84:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x84 lmm_fid: [0x200000405:0x84:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13e:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611268 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 08:41:27.000000000 -0400 Modify: 2026-09-08 08:41:27.000000000 -0400 Change: 2026-09-08 08:41:28.000000000 -0400 Birth: 2026-09-08 08:41:28.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 08:41:34 (1788871294) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269487 s, 38.9 MB/s oleg442-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg442-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg442-server: O/280000401/d31/13f: File not found by ext2_lookup PASS 210b (22s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 08:41:57 (1788871317) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.227514 s, 180 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0183261 s, 224 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 8 08:41 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x89:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 8 08:41 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 08:42:04 (1788871324) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 08:42:13 (1788871333) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1925 sec =========== 08:42:22 (1788871342) === sanity-flr: start cleanup 08:42:24 (1788871344) === === sanity-flr: finish cleanup 08:42:33 (1788871353) ===