-----============= acceptance-small: sanity-flr ============----- Thu Sep 17 18:16:22 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg457-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 18:16:44 (1789683404) === oleg457-client.virtnet: executing check_config_client /mnt/lustre oleg457-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg457-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff89380a4f6800.idle_timeout=debug osc.lustre-OST0001-osc-ffff89380a4f6800.idle_timeout=debug disable quota as required oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 18:17:24 (1789683444) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8226] == sanity-flr test 0a: lfs mirror create with -N option == 18:17:26 (1789683446) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 18:17:42 (1789683462) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 18:17:46 (1789683466) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 18:17:49 (1789683469) PASS 0d (21s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 18:18:11 (1789683491) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 18:18:14 (1789683494) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 18:18:18 (1789683498) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff89380a4f6800.stats=clear osc.lustre-OST0001-osc-ffff89380a4f6800.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 18:18:27 (1789683507) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 18:18:40 (1789683520) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 18:18:49 (1789683529) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 18:18:52 (1789683532) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 18:19:00 (1789683540) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 18:19:09 (1789683549) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 18:19:17 (1789683557) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 18:19:24 (1789683564) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 18:19:33 (1789683573) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 18:19:44 (1789683584) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 18:19:51 (1789683591) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.295668 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124031 s, 8.5 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 18:19:59 (1789683599) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.129739 s, 40.4 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 18:20:07 (1789683607) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 18:20:16 (1789683616) Stopping /mnt/lustre-ost1 (opts:) on oleg457-server oleg457-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89380a4f6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89380a4f6800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg457-server: oleg457-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg457-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg457-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg457-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg457-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg457-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg457-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg457-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg457-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg457-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg457-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg457-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg457-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg457-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg457-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg457-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg457-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg457-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg457-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg457-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg457-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg457-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg457-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg457-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg457-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg457-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg457-server oleg457-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff89380a4f6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89380a4f6800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg457-server: oleg457-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg457-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (133s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 18:22:30 (1789683750) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.398446 s, 15.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg457-server oleg457-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89380a4f6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89380a4f6800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg457-server: oleg457-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg457-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (43s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 18:23:13 (1789683793) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg457-server oleg457-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89380a4f6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89380a4f6800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg457-server: oleg457-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg457-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg457-server oleg457-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff89380a4f6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89380a4f6800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg457-server: oleg457-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg457-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (74s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 18:24:28 (1789683868) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.6063 s, 20.1 MB/s -rw-r--r-- 1 root root 52428800 Sep 17 18:24 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.935s user 0m0.001s sys 0m0.159s Stopping /mnt/lustre-ost1 (opts:) on oleg457-server oleg457-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89380a4f6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89380a4f6800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.268s user 0m0.001s sys 0m0.190s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg457-server: oleg457-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg457-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg457-server oleg457-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff89380a4f6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89380a4f6800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.129s user 0m0.000s sys 0m0.186s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg457-server: oleg457-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg457-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg457-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89380a4f6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89380a4f6800.ost_server_uuid in FULL state after 0 sec PASS 33b (92s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 18:25:59 (1789683959) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 18:26:03 (1789683963) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 18:26:06 (1789683966) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 18:26:10 (1789683970) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 18:26:16 (1789683976) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.473357 s, 26.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 18:26:27 (1789683987) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327635 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413589 s, 25.4 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 18:26:33 (1789683993) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 18:26:40 (1789684000) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00177637 s, 576 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000955347 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000724009 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00115673 s, 885 kB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 18:26:48 (1789684008) create test files with size 15753801 .. 5392c6de0ca86731efe094bb0c8c1f68 - 50e8bfa97230472936b21d0701848c34 - 9fc753b4a4ce0f097665d9ca0d543889 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg457-client.virtnet /mnt/lustre (opts:) Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre PASS 37 (19s) == sanity-flr test 38: resync ============================ 18:27:07 (1789684027) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 18:27:19 (1789684039) PASS 39 (5s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 18:27:24 (1789684044) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628994 s, 16.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x9a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x9b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xd8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234129 s, 44.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xd9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xda:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x9c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (7s) == sanity-flr test 41: lfs mirror resync check =========== 18:27:31 (1789684051) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0666442 s, 31.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.35044 s, 12.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.118685 s, 17.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (13s) == sanity-flr test 42: lfs mirror verify ================= 18:27:44 (1789684064) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 18:27:47 (1789684067) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 18:27:50 (1789684070) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191038 s, 54.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 18:27:58 (1789684078) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.358347 s, 8.8 MB/s PASS 44a (8s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 18:28:07 (1789684087) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa5:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xe4:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xe4:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 18:28:18 (1789684098) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.412015 s, 25.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg457-client.virtnet /mnt/lustre (opts:) Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 18:28:31 (1789684111) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff893804dbb000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.199879 s, 52.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg457-client.virtnet /mnt/lustre (opts:) Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff893805230000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 18:28:44 (1789684124) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (37s) == sanity-flr test 46: Verify setstripe --copy option ==== 18:29:21 (1789684161) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (37s) == sanity-flr test 47: Verify mirror obj alloc =========== 18:29:58 (1789684198) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 18:30:02 (1789684202) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.267196 s, 11.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.108325 s, 29.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xa29d822 (nosync,stale) Mirror 2: 0xa29d822 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 2cc226dc57c77851445c0f435e2332ba - mirror 1 checksum: f047ee3d7f3e2bfb3c17f16d2d12ddcb - mirror 2 checksum: 2cc226dc57c77851445c0f435e2332ba - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-17T18:30:04-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-17T18:30:04-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb1:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 2cc226dc57c77851445c0f435e2332ba - mirror 1 checksum: f047ee3d7f3e2bfb3c17f16d2d12ddcb - mirror 2 checksum: f047ee3d7f3e2bfb3c17f16d2d12ddcb - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-17T18:30:04-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-17T18:30:04-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb1:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 18:30:12 (1789684212) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513508 s, 20.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 312320.. 313343: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (9s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 18:30:21 (1789684221) Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 18:30:33 (1789684233) shut down facet: mds1 facet_host: oleg457-server facet_failover_host: oleg457-server Failover mds1 to oleg457-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 18:30:55 (1789684255) targets are mounted 18:30:55 (1789684255) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052949 s, 19.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb3:0x0] } PASS 50A (49s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 18:31:10 (1789684270) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0307784 s, 665 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00997313 s, 2.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 18:31:19 (1789684279) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0641117 s, 319 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0109327 s, 1.9 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 18:31:28 (1789684288) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00587093 s, 2.8 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 18:31:36 (1789684296) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0459737 s, 445 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 18:31:44 (1789684304) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0269516 s, 152 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0239644 s, 171 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 18:31:52 (1789684312) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) PASS 60b (18s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 18:32:10 (1789684330) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (26s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 18:32:36 (1789684356) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 18:32:58 (1789684378) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 18:33:21 (1789684401) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 18:33:28 (1789684408) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 57569 mirror split pid 57570 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57569 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57570 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 18:34:35 (1789684475) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 4477 skipping zero length punch hole 000003 1789684477.381367 write 0x01755a thru 0x022af0 (0xb597 bytes) zero_range to largest ever: 0x115c6 000004 1789684477.385234 trunc from 0x0077bc to 0x0115c5 (0x9e0a bytes) 000005 1789684477.390516 write 0x014e6d thru 0x01d9c0 (0x8b54 bytes) 000006 1789684477.408378 mapread 0x011252 thru 0x01b81c (0xa5cb bytes) 000007 1789684477.434762 mapread 0x0146f3 thru 0x01d798 (0x90a6 bytes) 000008 1789684477.438203 trunc from 0x0052af to 0x00e529 (0x927b bytes) 000009 1789684477.443240 punch from 0x018a62 to 0x022af0 (0xa08f bytes) 000010 1789684477.465841 punch from 0x0110ae to 0x017912 (0x6865 bytes) 000011 1789684477.470638 write 0x028cbf thru 0x034db2 (0xc0f4 bytes) 000013 1789684477.475504 write 0x01913f thru 0x01aaf8 (0x19ba bytes) fallocating to largest ever: 0x24b9b 000014 1789684477.483537 fallocate 0x0243b6 thru 0x024b9a (0x7e5 bytes) 000015 1789684477.490018 punch from 0x018072 to 0x01be2f (0x3dbe bytes) 000016 1789684477.515005 write 0x03c0b0 thru 0x03ffff (0x3f50 bytes) fallocating to largest ever: 0x3c3ca 000017 1789684477.517360 fallocate 0x0309b2 thru 0x03c3c9 (0xba18 bytes) 000018 1789684477.525135 trunc from 0x0308a0 to 0x03476f (0x3ed0 bytes) 000021 1789684477.563419 punch from 0x032ac3 to 0x0396d7 (0x6c15 bytes) 000022 1789684477.567903 fallocate 0x02c92a thru 0x0322d3 (0x59aa bytes) 000023 1789684477.576566 mapread 0x0312e0 thru 0x03a54d (0x926e bytes) 000024 1789684477.609551 read 0x000368 thru 0x006563 (0x61fc bytes) 000025 1789684477.630474 trunc from 0x000fbb to 0x00ad7c (0x9dc2 bytes) 000026 1789684477.641280 mapread 0x001fea thru 0x00222d (0x244 bytes) 000027 1789684477.676153 mapwrite 0x033d70 thru 0x03642c (0x26bd bytes) 000028 1789684477.696319 mapwrite 0x02ced4 thru 0x039e08 (0xcf35 bytes) 000030 1789684477.717135 read 0x00fff3 thru 0x01c0dc (0xc0ea bytes) 000032 1789684477.730687 trunc from 0x040000 to 0x004529 (0x3bad6 bytes) 000033 1789684477.753479 write 0x03bf7f thru 0x03ffff (0x4081 bytes) 000034 1789684477.756273 trunc from 0x031865 to 0x0357f5 (0x3f91 bytes) fallocating to largest ever: 0x40000 000036 1789684477.762464 fallocate 0x032f4c thru 0x03ffff (0xd0b4 bytes) 000037 1789684477.768319 trunc from 0x040000 to 0x02bd85 (0x1427a bytes) 000038 1789684477.782435 trunc from 0x02bd86 to 0x02bbce (0x1b7 bytes) 000039 1789684477.798935 mapwrite 0x03ecbd thru 0x03ffff (0x1343 bytes) 000041 1789684477.829644 read 0x03bf34 thru 0x03ffff (0x40cc bytes) 000043 1789684477.853464 mapread 0x031ad5 thru 0x03d356 (0xb882 bytes) 000044 1789684477.854465 punch from 0x01b16a to 0x024c6c (0x9b03 bytes) 000045 1789684477.860317 write 0x02819f thru 0x02f89e (0x7700 bytes) 000046 1789684477.866322 MIRROR_EXTEND to 2 mirrors 000047 1789684478.691855 fallocate 0x038291 thru 0x03ffff (0x7d6f bytes) 000048 1789684478.720909 punch from 0x012beb to 0x022aae (0xfec4 bytes) 000049 1789684478.727912 trunc from 0x040000 to 0x014dba (0x2b245 bytes) 000050 1789684478.752330 trunc from 0x005318 to 0x01140b (0xc0f4 bytes) 000051 1789684478.756818 trunc from 0x009950 to 0x00f47b (0x5b2c bytes) 000052 1789684478.761307 mapwrite 0x00453c thru 0x014481 (0xff46 bytes) 000053 1789684478.811146 MIRROR_RESYNC 2 mirrors 000054 1789684479.371968 punch from 0x00a084 to 0x00e296 (0x4213 bytes) 000055 1789684479.408982 mapread 0x004dbd thru 0x00dcaa (0x8eee bytes) 000056 1789684479.426273 write 0x02ad1d thru 0x03aae0 (0xfdc4 bytes) 000057 1789684479.428163 trunc from 0x0190f1 to 0x023b7b (0xaa8b bytes) 000058 1789684479.432879 punch from 0x025b79 to 0x02a375 (0x47fd bytes) 000059 1789684479.445666 trunc from 0x03aae1 to 0x01a74f (0x20391 bytes) 000060 1789684479.470130 write 0x02fb6f thru 0x03224e (0x26e0 bytes) 000061 1789684479.471356 punch from 0x022bec to 0x023558 (0x96d bytes) 000062 1789684479.477900 mapread 0x01c4c6 thru 0x01ff0e (0x3a49 bytes) 000063 1789684479.486671 mapread 0x027ac5 thru 0x031b75 (0xa0b1 bytes) 000064 1789684479.495066 fallocate 0x01a952 thru 0x021b09 (0x71b8 bytes) 000065 1789684479.500219 fallocate 0x039ed6 thru 0x03ffff (0x612a bytes) 000066 1789684479.504885 fallocate 0x02d3e9 thru 0x03541d (0x8035 bytes) 000067 1789684479.509957 mapread 0x034d41 thru 0x03541d (0x6dd bytes) 000068 1789684479.515571 read 0x0190bb thru 0x01c28a (0x31d0 bytes) 000069 1789684479.516377 trunc from 0x03541e to 0x03b86a (0x644d bytes) 000070 1789684479.530855 fallocate 0x0258bc thru 0x02dab0 (0x81f5 bytes) 000071 1789684479.539034 mapread 0x00f768 thru 0x0190c9 (0x9962 bytes) 000072 1789684479.545172 trunc from 0x03b86b to 0x009b0a (0x31d60 bytes) 000073 1789684479.562633 write 0x0151d4 thru 0x015c1b (0xa48 bytes) 000074 1789684479.569369 MIRROR_RESYNC 2 mirrors 000075 1789684480.105097 MIRROR_RESYNC 2 mirrors 000076 1789684480.463559 mapwrite 0x01806b thru 0x0261b1 (0xe147 bytes) 000077 1789684480.604575 mapwrite 0x024631 thru 0x02ff1e (0xb8ee bytes) 000078 1789684480.702513 fallocate 0x024f03 thru 0x02899b (0x3a99 bytes) 000079 1789684480.727251 MIRROR_RESYNC 2 mirrors 000080 1789684481.280045 fallocate 0x00629a thru 0x0106f7 (0xa45e bytes) 000082 1789684481.310357 write 0x012d68 thru 0x01ff15 (0xd1ae bytes) 000083 1789684481.324302 MIRROR_RESYNC 2 mirrors 000086 1789684481.952385 punch from 0x00815b to 0x0087c9 (0x66f bytes) 000087 1789684482.012749 write 0x002162 thru 0x00b7c0 (0x965f bytes) 000088 1789684482.043783 fallocate 0x000736 thru 0x00389d (0x3168 bytes) 000089 1789684482.061097 trunc from 0x02ff1f to 0x00a9f1 (0x2552d bytes) 000090 1789684482.115005 trunc from 0x00a9f2 to 0x00c028 (0x1637 bytes) 000091 1789684482.141856 punch from 0x0012c5 to 0x006f2c (0x5c68 bytes) 000092 1789684482.163063 write 0x00cdbc thru 0x014848 (0x7a8d bytes) 000094 1789684482.173767 read 0x002cd3 thru 0x0096bf (0x69ed bytes) 000095 1789684482.184138 mapwrite 0x028f5a thru 0x029bd7 (0xc7e bytes) 000096 1789684482.251855 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x207:0x0]: Invalid argument (22) 000097 1789684482.399454 write 0x00cbdc thru 0x01a32e (0xd753 bytes) 000098 1789684482.418544 write 0x0022f1 thru 0x00464e (0x235e bytes) 000099 1789684482.427455 trunc from 0x01e18a to 0x021690 (0x3507 bytes) 000100 1789684482.432260 mapread 0x00c93e thru 0x011558 (0x4c1b bytes) 000101 1789684482.433244 fallocate 0x01df44 thru 0x021706 (0x37c3 bytes) 000102 1789684482.437268 mapread 0x017f47 thru 0x021e63 (0x9f1d bytes) 000103 1789684482.452889 mapwrite 0x01803c thru 0x022648 (0xa60d bytes) 000104 1789684482.479692 punch from 0x01ea89 to 0x026311 (0x7889 bytes) 000105 1789684482.484467 trunc from 0x02331e to 0x027fe6 (0x4cc9 bytes) 000106 1789684482.495086 MIRROR_EXTEND to 2 mirrors 000107 1789684483.258831 MIRROR_RESYNC 2 mirrors 000108 1789684483.651360 trunc from 0x029bd8 to 0x00e27f (0x1b958 bytes) 000109 1789684483.789998 trunc from 0x004452 to 0x00981f (0x53ce bytes) 000110 1789684483.796102 mapread 0x002502 thru 0x00e27f (0xbd7e bytes) 000111 1789684483.833360 mapread 0x00d9c9 thru 0x00db7c (0x1b4 bytes) 000112 1789684483.834208 punch from 0x0072f5 to 0x00e27f (0x6f8b bytes) 000113 1789684483.841056 write 0x02a2cf thru 0x035625 (0xb357 bytes) 000114 1789684483.844310 mapread 0x004ce4 thru 0x00e500 (0x981d bytes) 000116 1789684483.858356 write 0x03ec05 thru 0x03ffff (0x13fb bytes) 000117 1789684483.863214 mapread 0x0300c5 thru 0x038843 (0x877f bytes) 000118 1789684483.874636 read 0x022e05 thru 0x027547 (0x4743 bytes) 000119 1789684483.883388 mapread 0x02454b thru 0x0335a6 (0xf05c bytes) 000121 1789684483.891317 mapread 0x03fa56 thru 0x03ffff (0x5aa bytes) 000124 1789684483.894267 write 0x02bfa8 thru 0x033f80 (0x7fd9 bytes) 000125 1789684483.902544 mapread 0x035e93 thru 0x036b67 (0xcd5 bytes) 000127 1789684483.912452 write 0x036bce thru 0x03ffff (0x9432 bytes) 000128 1789684483.913828 write 0x02ff00 thru 0x034f89 (0x508a bytes) 000129 1789684483.916046 mapread 0x0143f4 thru 0x01aa9f (0x66ac bytes) 000130 1789684483.929193 read 0x00243e thru 0x01109a (0xec5d bytes) 000131 1789684483.929997 mapread 0x035888 thru 0x03c88f (0x7008 bytes) 000132 1789684483.941819 mapwrite 0x019517 thru 0x022894 (0x937e bytes) 000133 1789684483.979097 write 0x0207b0 thru 0x02c6d4 (0xbf25 bytes) 000134 1789684483.981408 punch from 0x017f34 to 0x01dae4 (0x5bb1 bytes) 000135 1789684483.989449 read 0x037236 thru 0x038088 (0xe53 bytes) 000136 1789684483.991047 fallocate 0x031714 thru 0x03c8e1 (0xb1ce bytes) 000137 1789684483.996388 mapread 0x00a4b9 thru 0x00f639 (0x5181 bytes) 000138 1789684483.997226 mapwrite 0x0209a3 thru 0x0295fc (0x8c5a bytes) 000139 1789684484.021389 read 0x039f7e thru 0x03d836 (0x38b9 bytes) 000140 1789684484.026829 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x207:0x0]: Invalid argument (22) 000141 1789684484.209441 fallocate 0x014719 thru 0x022ffd (0xe8e5 bytes) 000143 1789684484.217963 fallocate 0x01cb36 thru 0x0240e0 (0x75ab bytes) 000144 1789684484.228009 fallocate 0x001bfd thru 0x004c82 (0x3086 bytes) 000145 1789684484.234197 trunc from 0x040000 to 0x01c3e8 (0x23c17 bytes) 000146 1789684484.274485 mapread 0x00a3b9 thru 0x01a190 (0xfdd8 bytes) 000147 1789684484.391560 mapwrite 0x017e45 thru 0x022133 (0xa2ef bytes) 000148 1789684484.476920 fallocate 0x011447 thru 0x01425f (0x2e19 bytes) 000149 1789684484.483829 fallocate 0x02d4d4 thru 0x02f554 (0x2081 bytes) 000150 1789684484.505090 MIRROR_EXTEND to 2 mirrors 000151 1789684485.317042 write 0x02eff7 thru 0x034cc1 (0x5ccb bytes) 000152 1789684485.391960 MIRROR_RESYNC 2 mirrors 000153 1789684486.146163 write 0x013555 thru 0x022cff (0xf7ab bytes) 000154 1789684486.222571 trunc from 0x034cc2 to 0x0370b1 (0x23f0 bytes) 000155 1789684486.258339 trunc from 0x035d40 to 0x0370b1 (0x1372 bytes) 000156 1789684486.277477 trunc from 0x008d4f to 0x00dbdb (0x4e8d bytes) 000157 1789684486.293923 trunc from 0x023aa9 to 0x02bbf2 (0x814a bytes) 000158 1789684486.311198 trunc from 0x01777c to 0x023b33 (0xc3b8 bytes) 000159 1789684486.368073 punch from 0x0208a8 to 0x02ea8b (0xe1e4 bytes) 000160 1789684486.378899 mapread 0x01354d thru 0x014092 (0xb46 bytes) 000161 1789684486.379769 fallocate 0x013fa3 thru 0x022642 (0xe6a0 bytes) 000162 1789684486.402420 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x370b2): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x370b2): Mirror 2: 0x6e367b29 (stale) CRC-32 checksum value for chunk [0, 0x370b2): Mirror 1: 0x9148d821 Mirror 3: 0x9148d821 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x370b2: skipped 000163 1789684487.745948 read 0x011c3d thru 0x018464 (0x6828 bytes) 000164 1789684487.764787 fallocate 0x00481a thru 0x009397 (0x4b7e bytes) 000165 1789684487.812428 trunc from 0x0370b2 to 0x010b02 (0x265af bytes) 000166 1789684487.841448 trunc from 0x010b03 to 0x005dcd (0xad35 bytes) 000167 1789684487.868280 MIRROR_RESYNC 3 mirrors 000168 1789684488.494049 trunc from 0x005dce to 0x01beb3 (0x160e6 bytes) 000169 1789684488.530844 write 0x007640 thru 0x00f92d (0x82ee bytes) 000170 1789684488.546189 trunc from 0x010a25 to 0x01325f (0x283b bytes) 000171 1789684488.554425 trunc from 0x01beb4 to 0x015c9c (0x6217 bytes) 000172 1789684488.576298 mapwrite 0x0323b4 thru 0x03ffff (0xdc4c bytes) 000173 1789684488.660940 punch from 0x013b22 to 0x01928f (0x576e bytes) 000174 1789684488.669063 trunc from 0x040000 to 0x01dc70 (0x2238f bytes) 000175 1789684488.701145 read 0x018db1 thru 0x01dc70 (0x4ec0 bytes) 000176 1789684488.708422 write 0x0206c6 thru 0x02c790 (0xc0cb bytes) 000177 1789684488.719901 mapwrite 0x032e52 thru 0x03fb58 (0xcd07 bytes) 000178 1789684488.804376 trunc from 0x03fb59 to 0x00cd65 (0x32df3 bytes) 000179 1789684488.821886 read 0x0073f0 thru 0x00cd65 (0x5976 bytes) 000180 1789684488.826957 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xcd66): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xcd66): Mirror 2: 0xe5bbc14f (stale) Mirror 3: 0xe5bbc14f (stale) CRC-32 checksum value for chunk [0, 0xcd66): Mirror 1: 0x46590c78 Mirror 4: 0x46590c78 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xcd66: skipped 000181 1789684489.707658 MIRROR_RESYNC 4 mirrors 000182 1789684490.348147 trunc from 0x00cd66 to 0x009003 (0x3d62 bytes) 000183 1789684490.403073 trunc from 0x009004 to 0x01fcb5 (0x16cb2 bytes) 000184 1789684490.432886 MIRROR_RESYNC 4 mirrors 000185 1789684491.292566 trunc from 0x01fcb6 to 0x00080e (0x1f4a7 bytes) 000187 1789684491.338427 write 0x00f5b4 thru 0x018b9f (0x95ec bytes) 000188 1789684491.342981 mapread 0x00825e thru 0x013c92 (0xba35 bytes) 000189 1789684491.375061 fallocate 0x03dbdf thru 0x03ffff (0x2421 bytes) 000190 1789684491.383748 write 0x011f0f thru 0x012631 (0x723 bytes) 000191 1789684491.390863 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x207:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18ba0): Mirror 3: 0xb87c0ed6 (stale) Mirror 4: 0xb87c0ed6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18ba0): Mirror 3: 0xb87c0ed6 (stale) Mirror 4: 0xb87c0ed6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18ba0: skipped 000193 1789684491.785182 fallocate 0x02b89c thru 0x031d73 (0x64d8 bytes) 000194 1789684491.790802 mapread 0x001289 thru 0x00f31a (0xe092 bytes) 000195 1789684491.814183 read 0x013142 thru 0x018b9f (0x5a5e bytes) 000196 1789684491.824193 trunc from 0x018ba0 to 0x02691c (0xdd7d bytes) 000199 1789684491.858497 fallocate 0x026f6a thru 0x028816 (0x18ad bytes) 000200 1789684491.864734 write 0x022cbe thru 0x02819a (0x54dd bytes) 000201 1789684491.882368 trunc from 0x01d7e3 to 0x01fb93 (0x23b1 bytes) 000202 1789684491.888454 fallocate 0x027870 thru 0x029be7 (0x2378 bytes) 000203 1789684491.894515 mapwrite 0x003e66 thru 0x00d3bc (0x9557 bytes) 000204 1789684491.961381 trunc from 0x00839f to 0x0086dc (0x33e bytes) 000205 1789684491.988974 MIRROR_RESYNC 3 mirrors 000206 1789684493.149725 punch from 0x006c8e to 0x00b358 (0x46cb bytes) 000207 1789684493.185547 fallocate 0x01bd4f thru 0x02a457 (0xe709 bytes) 000208 1789684493.192849 trunc from 0x01b286 to 0x01c3f4 (0x116f bytes) 000209 1789684493.198550 write 0x0381f6 thru 0x03a881 (0x268c bytes) 000210 1789684493.200828 mapwrite 0x003d84 thru 0x00f37d (0xb5fa bytes) 000211 1789684493.265456 mapwrite 0x03a09d thru 0x03f3ca (0x532e bytes) 000213 1789684493.331395 mapwrite 0x01e467 thru 0x027685 (0x921f bytes) 000214 1789684493.380906 trunc from 0x03f3cb to 0x033279 (0xc151 bytes) 000215 1789684493.397570 fallocate 0x03ce21 thru 0x03d0f6 (0x2d6 bytes) 000216 1789684493.403173 mapread 0x0061d8 thru 0x0105b5 (0xa3de bytes) 000217 1789684493.404293 mapwrite 0x035271 thru 0x0385fb (0x338b bytes) 000219 1789684493.433366 fallocate 0x01ec4e thru 0x028b3e (0x9ef1 bytes) 000220 1789684493.438221 mapwrite 0x012524 thru 0x019458 (0x6f35 bytes) 000221 1789684493.469377 punch from 0x0018b2 to 0x00c708 (0xae57 bytes) 000222 1789684493.481214 trunc from 0x03d0f7 to 0x021b33 (0x1b5c3 bytes) 000223 1789684493.499150 fallocate 0x00600f thru 0x00bc01 (0x5bf3 bytes) 000224 1789684493.504044 mapwrite 0x03d938 thru 0x03ffff (0x26c8 bytes) 000225 1789684493.542588 mapwrite 0x000df6 thru 0x00562a (0x4835 bytes) 000227 1789684493.590052 write 0x001381 thru 0x001b61 (0x7e1 bytes) 000228 1789684493.593539 punch from 0x012cd7 to 0x01ce87 (0xa1b1 bytes) 000229 1789684493.601340 write 0x013a20 thru 0x02397c (0xff5d bytes) 000230 1789684493.616679 fallocate 0x018f26 thru 0x01adfb (0x1ed6 bytes) 000231 1789684493.626484 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa3d704b4 (stale) Mirror 4: 0xa3d704b4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x78d2c8d4 Mirror 5: 0x78d2c8d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000232 1789684494.642540 fallocate 0x021501 thru 0x025a81 (0x4581 bytes) 000233 1789684494.705089 mapwrite 0x03f275 thru 0x03ffff (0xd8b bytes) 000235 1789684494.736669 trunc from 0x040000 to 0x00e50f (0x31af0 bytes) 000236 1789684494.786815 mapread 0x0052d3 thru 0x005ecc (0xbfa bytes) 000237 1789684494.794255 fallocate 0x03b4e7 thru 0x03ffff (0x4b19 bytes) 000238 1789684494.802094 punch from 0x03e88b to 0x03ffff (0x1775 bytes) 000240 1789684494.812917 trunc from 0x027ac9 to 0x02fe90 (0x83c8 bytes) 000241 1789684494.817173 trunc from 0x040000 to 0x021cbb (0x1e344 bytes) 000244 1789684494.854872 punch from 0x01f47b to 0x021cbb (0x2841 bytes) 000245 1789684494.859996 mapwrite 0x0264e9 thru 0x02c993 (0x64ab bytes) 000246 1789684494.901639 punch from 0x02bfc7 to 0x02c993 (0x9cd bytes) 000247 1789684494.910140 trunc from 0x0235d8 to 0x023b85 (0x5ae bytes) 000248 1789684494.918243 mapwrite 0x02878c thru 0x033aa2 (0xb317 bytes) 000249 1789684494.984925 fallocate 0x01f57a thru 0x01fe9e (0x925 bytes) 000250 1789684494.993634 trunc from 0x018f67 to 0x028317 (0xf3b1 bytes) 000251 1789684495.022461 read 0x01da02 thru 0x01e909 (0xf08 bytes) 000252 1789684495.031688 punch from 0x00f4c7 to 0x0115a3 (0x20dd bytes) 000253 1789684495.039411 punch from 0x032b58 to 0x033aa2 (0xf4b bytes) 000254 1789684495.047268 fallocate 0x02c834 thru 0x036179 (0x9946 bytes) 000257 1789684495.058218 mapread 0x002145 thru 0x004a7b (0x2937 bytes) 000258 1789684495.069497 punch from 0x02b1d6 to 0x033aa2 (0x88cd bytes) 000259 1789684495.076979 trunc from 0x033aa3 to 0x03eb86 (0xb0e4 bytes) 000260 1789684495.094484 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x207:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3eb87): Mirror 3: 0x1e19790c (stale) Mirror 5: 0x38c633a9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eb87): Mirror 3: 0x1e19790c (stale) Mirror 5: 0x38c633a9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eb87: skipped 000261 1789684495.414033 mapwrite 0x000bec thru 0x005e78 (0x528d bytes) 000262 1789684495.456172 trunc from 0x030d04 to 0x0313b9 (0x6b6 bytes) 000263 1789684495.461398 trunc from 0x03eb87 to 0x002a06 (0x3c180 bytes) 000266 1789684495.487996 read 0x0029cd thru 0x002a06 (0x3a bytes) 000267 1789684495.489819 fallocate 0x01dc8e thru 0x029712 (0xba85 bytes) 000268 1789684495.493979 mapread 0x0020c0 thru 0x002a06 (0x947 bytes) 000270 1789684495.496040 mapread 0x00298c thru 0x002a06 (0x7b bytes) 000271 1789684495.500562 MIRROR_RESYNC 3 mirrors 000272 1789684496.099042 punch from 0x0001e0 to 0x002a06 (0x2827 bytes) 000273 1789684496.185353 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x207:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a07): Mirror 3: 0x6edf94e2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a07): Mirror 3: 0x6edf94e2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a07: skipped 000274 1789684496.539722 fallocate 0x000591 thru 0x003868 (0x32d8 bytes) 000275 1789684496.548049 trunc from 0x001293 to 0x002a06 (0x1774 bytes) 000276 1789684496.559117 MIRROR_RESYNC 2 mirrors 000277 1789684497.030755 write 0x01a7ae thru 0x029a8f (0xf2e2 bytes) 000278 1789684497.059878 trunc from 0x013eed to 0x023a2a (0xfb3e bytes) 000279 1789684497.073113 mapwrite 0x03717e thru 0x03ffff (0x8e82 bytes) 000280 1789684497.125904 write 0x002cc2 thru 0x00bf15 (0x9254 bytes) 000281 1789684497.135441 punch from 0x0146e5 to 0x0158e9 (0x1205 bytes) 000282 1789684497.140832 trunc from 0x01d87d to 0x02bd73 (0xe4f7 bytes) 000283 1789684497.145860 fallocate 0x017e2c thru 0x018acc (0xca1 bytes) 000284 1789684497.150867 write 0x03bb94 thru 0x03ef28 (0x3395 bytes) 000285 1789684497.152805 trunc from 0x020044 to 0x02a89b (0xa858 bytes) 000286 1789684497.160281 fallocate 0x007860 thru 0x015b67 (0xe308 bytes) 000287 1789684497.166062 trunc from 0x00c199 to 0x00eaf1 (0x2959 bytes) 000288 1789684497.171099 mapwrite 0x03e677 thru 0x03ffff (0x1989 bytes) 000289 1789684497.188427 mapwrite 0x028470 thru 0x02a22c (0x1dbd bytes) 000290 1789684497.223304 fallocate 0x0130de thru 0x013844 (0x767 bytes) 000291 1789684497.230545 fallocate 0x0362a3 thru 0x03dc3f (0x799d bytes) 000292 1789684497.237522 fallocate 0x018bb1 thru 0x0223d6 (0x9826 bytes) 000293 1789684497.254895 mapread 0x031763 thru 0x036276 (0x4b14 bytes) 000294 1789684497.264576 trunc from 0x040000 to 0x0076fa (0x38905 bytes) 000295 1789684497.289314 fallocate 0x0268cb thru 0x030388 (0x9abe bytes) 000296 1789684497.293917 read 0x012561 thru 0x01ef04 (0xc9a4 bytes) 000297 1789684497.313303 read 0x013203 thru 0x01ecb8 (0xbab6 bytes) 000298 1789684497.317690 trunc from 0x02fcd1 to 0x030388 (0x6b8 bytes) 000299 1789684497.325698 trunc from 0x02e3bd to 0x030388 (0x1fcc bytes) 000300 1789684497.331442 trunc from 0x01662c to 0x01ddb7 (0x778c bytes) 000301 1789684497.339823 write 0x0196a4 thru 0x027573 (0xded0 bytes) 000302 1789684497.352827 mapwrite 0x03d220 thru 0x03ffff (0x2de0 bytes) 000303 1789684497.421873 mapread 0x02bc7f thru 0x037813 (0xbb95 bytes) 000305 1789684497.423505 punch from 0x016028 to 0x01a5ef (0x45c8 bytes) 000306 1789684497.432051 punch from 0x026bac to 0x0365ed (0xfa42 bytes) 000307 1789684497.440376 fallocate 0x03f84d thru 0x03ffff (0x7b3 bytes) 000308 1789684497.444798 punch from 0x00ed70 to 0x01e3cf (0xf660 bytes) 000309 1789684497.450158 read 0x0333d3 thru 0x03a7aa (0x73d8 bytes) 000310 1789684497.483081 MIRROR_RESYNC 2 mirrors 000311 1789684498.267098 trunc from 0x033736 to 0x03c412 (0x8cdd bytes) 000312 1789684498.325290 write 0x00b806 thru 0x00e5cc (0x2dc7 bytes) 000314 1789684498.341905 trunc from 0x040000 to 0x0339ef (0xc610 bytes) 000315 1789684498.423586 MIRROR_RESYNC 2 mirrors 000316 1789684499.495903 fallocate 0x022258 thru 0x022b7b (0x924 bytes) 000317 1789684499.538571 punch from 0x013aa1 to 0x01d2f0 (0x9850 bytes) 000318 1789684499.546463 fallocate 0x02e53b thru 0x03e116 (0xfbdc bytes) 000319 1789684499.554011 trunc from 0x02eeaa to 0x03d785 (0xe8dc bytes) 000320 1789684499.560743 trunc from 0x02fb77 to 0x0361f2 (0x667c bytes) 000321 1789684499.567215 trunc from 0x01634b to 0x020815 (0xa4cb bytes) 000322 1789684499.574399 trunc from 0x03e117 to 0x0238e4 (0x1a832 bytes) 000323 1789684499.599881 write 0x03e768 thru 0x03ffff (0x1898 bytes) 000324 1789684499.607199 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x207:0x0]: Invalid argument (22) 000326 1789684499.785013 trunc from 0x040000 to 0x02303d (0x1cfc2 bytes) 000327 1789684499.815100 fallocate 0x02900c thru 0x02cac7 (0x3abc bytes) 000328 1789684499.822536 write 0x009ad8 thru 0x00afaa (0x14d3 bytes) 000329 1789684499.835986 mapwrite 0x03528e thru 0x03983e (0x45b1 bytes) 000330 1789684499.966469 trunc from 0x03983f to 0x03cf3b (0x36fd bytes) 000331 1789684499.981722 mapread 0x03a56b thru 0x03cf3b (0x29d1 bytes) 000333 1789684499.989784 punch from 0x00d874 to 0x010eb3 (0x3640 bytes) 000334 1789684499.993537 write 0x01de3f thru 0x025592 (0x7754 bytes) 000336 1789684500.008532 punch from 0x014c8e to 0x021a14 (0xcd87 bytes) 000337 1789684500.026581 trunc from 0x02023a to 0x02a7aa (0xa571 bytes) 000338 1789684500.034534 mapread 0x03c415 thru 0x03cf3b (0xb27 bytes) 000339 1789684500.035792 fallocate 0x03d4e5 thru 0x03ffff (0x2b1b bytes) 000340 1789684500.042550 trunc from 0x040000 to 0x033265 (0xcd9a bytes) 000341 1789684500.061789 write 0x03ff59 thru 0x03ffff (0xa7 bytes) 000343 1789684500.063328 trunc from 0x0334c2 to 0x034bcf (0x170e bytes) 000344 1789684500.083050 trunc from 0x01958b to 0x0224a1 (0x8f17 bytes) 000345 1789684500.088980 mapwrite 0x005e46 thru 0x010f5d (0xb118 bytes) 000347 1789684500.157020 trunc from 0x03b88d to 0x03ffff (0x4773 bytes) 000348 1789684500.165331 read 0x010fb2 thru 0x013a3a (0x2a89 bytes) 000350 1789684500.180253 fallocate 0x011b8e thru 0x01786f (0x5ce2 bytes) 000351 1789684500.186875 punch from 0x01fe88 to 0x022e85 (0x2ffe bytes) 000353 1789684500.201944 mapread 0x02e874 thru 0x034866 (0x5ff3 bytes) 000355 1789684500.218753 mapwrite 0x0263a8 thru 0x032342 (0xbf9b bytes) 000357 1789684500.274106 fallocate 0x0293e4 thru 0x02cff3 (0x3c10 bytes) 000359 1789684500.282092 trunc from 0x040000 to 0x02b5e8 (0x14a17 bytes) 000360 1789684500.317380 read 0x02702c thru 0x02b5e8 (0x45bd bytes) 000361 1789684500.318092 read 0x02a080 thru 0x02b5e8 (0x1569 bytes) 000363 1789684500.320088 fallocate 0x030a47 thru 0x031686 (0xc40 bytes) 000364 1789684500.325544 mapread 0x0107dc thru 0x012b6a (0x238f bytes) 000365 1789684500.326496 punch from 0x0034f9 to 0x00edd3 (0xb8db bytes) 000366 1789684500.334402 trunc from 0x00b30d to 0x016f58 (0xbc4c bytes) 000367 1789684500.342181 trunc from 0x031687 to 0x00dfbb (0x236cb bytes) 000368 1789684500.390987 write 0x015179 thru 0x01cc98 (0x7b20 bytes) 000369 1789684500.392332 trunc from 0x01cc99 to 0x038d3a (0x1c0a2 bytes) 000371 1789684500.431447 trunc from 0x01b33e to 0x025a69 (0xa72c bytes) 000372 1789684500.458197 punch from 0x0270d5 to 0x030fcb (0x9ef7 bytes) 000373 1789684500.465986 trunc from 0x038d3b to 0x02cb44 (0xc1f6 bytes) 000374 1789684500.496521 mapwrite 0x032408 thru 0x0384db (0x60d4 bytes) 000375 1789684500.573051 write 0x036fe7 thru 0x03ffff (0x9019 bytes) 000376 1789684500.575385 read 0x012692 thru 0x017425 (0x4d94 bytes) 000378 1789684500.576751 trunc from 0x040000 to 0x026add (0x19522 bytes) 000379 1789684500.589552 punch from 0x018f73 to 0x021e0d (0x8e9b bytes) 000380 1789684500.595144 punch from 0x01a120 to 0x023102 (0x8fe3 bytes) 000381 1789684500.602473 fallocate 0x004d73 thru 0x006439 (0x16c7 bytes) 000382 1789684500.612473 trunc from 0x0256ac to 0x026add (0x1432 bytes) 000383 1789684500.617906 trunc from 0x00f839 to 0x012557 (0x2d1f bytes) 000384 1789684500.624422 write 0x00ef58 thru 0x01968f (0xa738 bytes) 000385 1789684500.637825 trunc from 0x026ade to 0x01c1ea (0xa8f3 bytes) 000386 1789684500.672467 trunc from 0x003208 to 0x007570 (0x4369 bytes) 000387 1789684500.682836 trunc from 0x00f39e to 0x0132ef (0x3f52 bytes) 000388 1789684500.707548 write 0x011229 thru 0x0185ba (0x7392 bytes) 000389 1789684500.723550 trunc from 0x01c1eb to 0x036a36 (0x1a84c bytes) 000390 1789684500.741768 mapwrite 0x026eb5 thru 0x03340b (0xc557 bytes) 000391 1789684500.825378 trunc from 0x036a37 to 0x03b3d3 (0x499d bytes) 000393 1789684500.870856 mapwrite 0x0027bd thru 0x00f23d (0xca81 bytes) 000395 1789684501.058757 punch from 0x0168c2 to 0x0214be (0xabfd bytes) 000396 1789684501.072528 read 0x003639 thru 0x003d3f (0x707 bytes) 000399 1789684501.078339 trunc from 0x008c1a to 0x0162d4 (0xd6bb bytes) 000401 1789684501.105845 punch from 0x021bbd to 0x02450d (0x2951 bytes) 000402 1789684501.119431 write 0x0373cf thru 0x039aa3 (0x26d5 bytes) 000403 1789684501.122209 mapwrite 0x01ee90 thru 0x023df0 (0x4f61 bytes) 000404 1789684501.210929 mapwrite 0x01597f thru 0x01a7e1 (0x4e63 bytes) 000405 1789684501.243686 trunc from 0x033302 to 0x03b3d3 (0x80d2 bytes) 000406 1789684501.253493 fallocate 0x00b4f8 thru 0x010c03 (0x570c bytes) 000408 1789684501.259552 fallocate 0x00ae11 thru 0x0169a8 (0xbb98 bytes) 000409 1789684501.265195 fallocate 0x029277 thru 0x02b96d (0x26f7 bytes) 000410 1789684501.270408 mapwrite 0x013cbd thru 0x02196c (0xdcb0 bytes) 000411 1789684501.293948 trunc from 0x03b3d4 to 0x0378b8 (0x3b1b bytes) 000412 1789684501.309175 fallocate 0x030443 thru 0x035c2c (0x57ea bytes) 000413 1789684501.316861 trunc from 0x0378b9 to 0x035778 (0x2140 bytes) 000414 1789684501.330958 read 0x025baf thru 0x03330e (0xd760 bytes) 000415 1789684501.339240 trunc from 0x020af8 to 0x02d58f (0xca98 bytes) 000416 1789684501.345071 fallocate 0x017041 thru 0x025a68 (0xea28 bytes) 000417 1789684501.349968 write 0x013909 thru 0x018100 (0x47f8 bytes) 000418 1789684501.351867 read 0x031b3c thru 0x0347a5 (0x2c6a bytes) 000419 1789684501.352924 punch from 0x002cd1 to 0x0058b0 (0x2be0 bytes) 000421 1789684501.359879 write 0x016d29 thru 0x024469 (0xd741 bytes) 000424 1789684501.377944 punch from 0x022d4e to 0x0243d6 (0x1689 bytes) 000427 1789684501.400052 punch from 0x0331ca to 0x034853 (0x168a bytes) 000428 1789684501.415098 trunc from 0x035779 to 0x01bb25 (0x19c53 bytes) 000429 1789684501.454661 mapwrite 0x019baa thru 0x02394d (0x9da4 bytes) 000432 1789684501.536835 write 0x01a56d thru 0x01cc1c (0x26b0 bytes) 000434 1789684501.538892 mapread 0x017e78 thru 0x01c047 (0x41d0 bytes) 000435 1789684501.540059 trunc from 0x02394e to 0x026902 (0x2fb5 bytes) 000436 1789684501.560637 read 0x00c492 thru 0x0167a3 (0xa312 bytes) 000437 1789684501.561278 trunc from 0x026903 to 0x0329fd (0xc0fb bytes) 000438 1789684501.572002 mapread 0x0294e2 thru 0x0329fd (0x951c bytes) 000439 1789684501.579166 fallocate 0x001cf6 thru 0x009089 (0x7394 bytes) 000440 1789684501.584939 MIRROR_EXTEND to 2 mirrors 000441 1789684502.552491 trunc from 0x0215bd to 0x029c90 (0x86d4 bytes) 000442 1789684502.583827 trunc from 0x0329fe to 0x023429 (0xf5d4 bytes) 000443 1789684502.609537 punch from 0x00f5a6 to 0x017faa (0x8a05 bytes) 000444 1789684502.614180 read 0x01d4b8 thru 0x023429 (0x5f72 bytes) 000445 1789684502.623084 mapwrite 0x00b014 thru 0x00f5c8 (0x45b5 bytes) 000446 1789684502.671070 read 0x002afd thru 0x0064cc (0x39d0 bytes) 000447 1789684502.681389 fallocate 0x03bad9 thru 0x03ffff (0x4527 bytes) 000448 1789684502.689936 mapwrite 0x018b84 thru 0x02170f (0x8b8c bytes) 000449 1789684502.731039 trunc from 0x02342a to 0x01608b (0xd39e bytes) 000450 1789684502.783119 fallocate 0x01751e thru 0x025eb2 (0xe995 bytes) 000451 1789684502.788465 punch from 0x01104c to 0x01dcde (0xcc93 bytes) 000452 1789684502.813062 trunc from 0x016668 to 0x01d3a4 (0x6d3d bytes) 000453 1789684502.822971 write 0x014376 thru 0x01dc63 (0x98ee bytes) 000454 1789684502.856250 write 0x00618e thru 0x006e5c (0xccf bytes) 000455 1789684502.859717 trunc from 0x025eb3 to 0x02a56b (0x46b9 bytes) 000456 1789684502.896302 read 0x02388a thru 0x02885d (0x4fd4 bytes) 000457 1789684502.908392 fallocate 0x03bf57 thru 0x03ffff (0x40a9 bytes) 000458 1789684502.913824 mapread 0x01b584 thru 0x029ee3 (0xe960 bytes) 000459 1789684502.916075 trunc from 0x025f4a to 0x029f6e (0x4025 bytes) 000460 1789684502.919426 mapread 0x007446 thru 0x012f6d (0xbb28 bytes) 000461 1789684502.920501 trunc from 0x02a56c to 0x029e66 (0x705 bytes) 000462 1789684502.937539 read 0x010f80 thru 0x0130ef (0x2170 bytes) 000463 1789684502.938741 mapread 0x012774 thru 0x016050 (0x38dd bytes) 000464 1789684502.940052 read 0x01c3e5 thru 0x022055 (0x5c71 bytes) 000465 1789684502.951287 MIRROR_RESYNC 2 mirrors 000466 1789684503.954473 read 0x013a8e thru 0x01aee7 (0x745a bytes) 000467 1789684503.966852 read 0x00e85a thru 0x019af2 (0xb299 bytes) 000468 1789684503.977424 punch from 0x0167bd to 0x01c07e (0x58c2 bytes) 000469 1789684504.034771 write 0x0200a4 thru 0x02a07e (0x9fdb bytes) 000470 1789684504.045424 mapread 0x01eb22 thru 0x01f90f (0xdee bytes) 000471 1789684504.065991 mapwrite 0x038903 thru 0x03ffff (0x76fd bytes) 000472 1789684504.131213 mapwrite 0x03304e thru 0x0388b6 (0x5869 bytes) 000473 1789684504.157860 mapread 0x03a5fc thru 0x03ffff (0x5a04 bytes) 000474 1789684504.158956 mapwrite 0x002625 thru 0x00372f (0x110b bytes) 000475 1789684504.195240 trunc from 0x040000 to 0x023173 (0x1ce8c bytes) 000476 1789684504.211753 read 0x00bd4f thru 0x01a5f6 (0xe8a8 bytes) 000477 1789684504.226486 mapwrite 0x0263cf thru 0x033348 (0xcf7a bytes) 000479 1789684504.333047 write 0x00ec5b thru 0x0102ec (0x1692 bytes) 000480 1789684504.337419 mapwrite 0x02c968 thru 0x0374c8 (0xab61 bytes) 000482 1789684504.444116 trunc from 0x00b853 to 0x00da14 (0x21c2 bytes) 000483 1789684504.453968 fallocate 0x03cae3 thru 0x03ffff (0x351d bytes) 000484 1789684504.464900 read 0x0212e4 thru 0x02167d (0x39a bytes) 000485 1789684504.465276 fallocate 0x00d92a thru 0x0118df (0x3fb6 bytes) 000487 1789684504.472750 mapread 0x00a5a3 thru 0x00b091 (0xaef bytes) 000488 1789684504.498418 mapread 0x035dbf thru 0x0374c8 (0x170a bytes) 000489 1789684504.501247 trunc from 0x0374c9 to 0x00c768 (0x2ad60 bytes) 000490 1789684504.524439 read 0x0071b7 thru 0x00c768 (0x55b2 bytes) 000491 1789684504.525241 write 0x026427 thru 0x02c13e (0x5d18 bytes) 000492 1789684504.526659 trunc from 0x02c13f to 0x01d5fe (0xeb40 bytes) 000493 1789684504.558154 trunc from 0x01d5ff to 0x00b8c6 (0x11d38 bytes) 000494 1789684504.583895 mapread 0x00270e thru 0x00b8c6 (0x91b9 bytes) 000495 1789684504.596726 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xb8c7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb8c7): Mirror 2: 0x22826e5a (stale) CRC-32 checksum value for chunk [0, 0xb8c7): Mirror 1: 0xe664d20a Mirror 3: 0xe664d20a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb8c7: skipped 000496 1789684505.856632 punch from 0x00378d to 0x004019 (0x88d bytes) 000497 1789684505.976023 punch from 0x006165 to 0x00810d (0x1fa9 bytes) 000498 1789684505.995258 mapread 0x0013a2 thru 0x00b8c6 (0xa525 bytes) 000499 1789684506.046312 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xb8c7): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb8c7): Mirror 2: 0x22826e5a (stale) Mirror 3: 0xe664d20a (stale) CRC-32 checksum value for chunk [0, 0xb8c7): Mirror 1: 0xbdb451fd Mirror 4: 0xbdb451fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb8c7: skipped 000500 1789684507.185357 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x207:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb8c7): Mirror 3: 0xe664d20a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb8c7): Mirror 3: 0xe664d20a (stale) CRC-32 checksum value for chunk [0, 0xb8c7): Mirror 1: 0xbdb451fd Mirror 4: 0xbdb451fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb8c7: skipped 000501 1789684508.038160 trunc from 0x0050c1 to 0x00b8c6 (0x6806 bytes) 000502 1789684508.116767 MIRROR_RESYNC 3 mirrors 000503 1789684509.370170 fallocate 0x00c98f thru 0x015b3f (0x91b1 bytes) 000504 1789684509.428142 write 0x0154fb thru 0x0236e9 (0xe1ef bytes) 000505 1789684509.436788 trunc from 0x0048ef to 0x013b14 (0xf226 bytes) 000506 1789684509.442353 mapwrite 0x02dcad thru 0x038cac (0xb000 bytes) 000507 1789684509.522195 read 0x0222fc thru 0x031be3 (0xf8e8 bytes) 000508 1789684509.528996 mapwrite 0x005484 thru 0x009883 (0x4400 bytes) 000509 1789684509.553460 mapread 0x00b74f thru 0x00e9ef (0x32a1 bytes) 000510 1789684509.555050 write 0x019044 thru 0x0199ef (0x9ac bytes) 000511 1789684509.556109 trunc from 0x038cad to 0x03a055 (0x13a9 bytes) 000513 1789684509.591354 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x207:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a056): Mirror 3: 0x2671b583 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a056): Mirror 3: 0x2671b583 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a056: skipped 000514 1789684509.972788 trunc from 0x03a056 to 0x001932 (0x38723 bytes) 000515 1789684510.015031 read 0x001591 thru 0x001932 (0x3a2 bytes) 000516 1789684510.022059 punch from 0x0011f7 to 0x001932 (0x73c bytes) 000517 1789684510.028359 trunc from 0x00086c to 0x001932 (0x10c7 bytes) 000518 1789684510.033040 fallocate 0x029891 thru 0x03827d (0xe9ed bytes) 000520 1789684510.039540 write 0x0338a5 thru 0x03932a (0x5a86 bytes) 000521 1789684510.052855 MIRROR_RESYNC 2 mirrors 000522 1789684510.970886 fallocate 0x009b74 thru 0x0137bd (0x9c4a bytes) 000523 1789684511.051307 mapread 0x0026c3 thru 0x0041b4 (0x1af2 bytes) 000524 1789684511.081568 punch from 0x009bf5 to 0x018668 (0xea74 bytes) 000525 1789684511.087492 mapwrite 0x03aa86 thru 0x03ffff (0x557a bytes) 000526 1789684511.210009 write 0x003181 thru 0x005ac9 (0x2949 bytes) 000527 1789684511.211644 punch from 0x021b2b to 0x02a59c (0x8a72 bytes) 000528 1789684511.215014 mapwrite 0x03076d thru 0x039bdd (0x9471 bytes) 000529 1789684511.246976 mapwrite 0x03a7fb thru 0x03ffff (0x5805 bytes) 000530 1789684511.305313 punch from 0x0243ce to 0x026591 (0x21c4 bytes) 000531 1789684511.314719 read 0x0050c6 thru 0x009d01 (0x4c3c bytes) 000532 1789684511.330722 punch from 0x03942a to 0x03b8a9 (0x2480 bytes) 000533 1789684511.347743 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf98a23e9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb17f9a00 Mirror 4: 0xb17f9a00 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000535 1789684512.445734 write 0x034996 thru 0x03ee00 (0xa46b bytes) 000536 1789684512.516702 mapread 0x00d3bf thru 0x01c447 (0xf089 bytes) 000537 1789684512.555407 read 0x0099c1 thru 0x00da24 (0x4064 bytes) 000539 1789684512.556452 trunc from 0x022caf to 0x02bd22 (0x9074 bytes) 000540 1789684512.570929 punch from 0x029a35 to 0x035e69 (0xc435 bytes) 000541 1789684512.604281 trunc from 0x02ac40 to 0x02d3dd (0x279e bytes) 000542 1789684512.612559 write 0x0266b2 thru 0x02f085 (0x89d4 bytes) 000543 1789684512.625544 fallocate 0x012940 thru 0x0184dd (0x5b9e bytes) 000545 1789684512.630736 trunc from 0x036909 to 0x03f053 (0x874b bytes) 000546 1789684512.636222 read 0x006541 thru 0x007692 (0x1152 bytes) 000547 1789684512.666851 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf98a23e9 (stale) Mirror 4: 0xb17f9a00 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa508d9bb Mirror 5: 0xa508d9bb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000549 1789684513.818074 trunc from 0x040000 to 0x00480c (0x3b7f3 bytes) 000550 1789684513.911853 write 0x005be7 thru 0x01056d (0xa987 bytes) 000551 1789684513.917559 mapwrite 0x00e332 thru 0x0177db (0x94aa bytes) 000552 1789684514.139352 trunc from 0x0177dc to 0x01985e (0x2083 bytes) 000553 1789684514.177482 write 0x027c77 thru 0x02ddad (0x6137 bytes) 000554 1789684514.180453 trunc from 0x02ddae to 0x02fca6 (0x1ef9 bytes) 000555 1789684514.227256 trunc from 0x02de6f to 0x02fb35 (0x1cc7 bytes) 000556 1789684514.252803 trunc from 0x022615 to 0x02fca6 (0xd692 bytes) 000557 1789684514.273719 trunc from 0x0226c1 to 0x0275bd (0x4efd bytes) 000558 1789684514.288026 fallocate 0x01afa6 thru 0x023f74 (0x8fcf bytes) 000559 1789684514.294667 punch from 0x016e0e to 0x024481 (0xd674 bytes) 000560 1789684514.300953 read 0x00c5d6 thru 0x014997 (0x83c2 bytes) 000561 1789684514.307102 mapread 0x01ee2c thru 0x022711 (0x38e6 bytes) 000563 1789684514.319977 write 0x019a6a thru 0x02571d (0xbcb4 bytes) 000565 1789684514.328716 trunc from 0x019e2e to 0x01e8f3 (0x4ac6 bytes) 000566 1789684514.352956 fallocate 0x005055 thru 0x00f9d6 (0xa982 bytes) 000567 1789684514.373709 punch from 0x0152a4 to 0x017f2b (0x2c88 bytes) 000568 1789684514.393916 read 0x0264fd thru 0x02f347 (0x8e4b bytes) 000569 1789684514.465361 mapwrite 0x036c31 thru 0x03a823 (0x3bf3 bytes) 000571 1789684514.575995 punch from 0x01af19 to 0x025679 (0xa761 bytes) 000572 1789684514.586530 mapwrite 0x002b6b thru 0x010b5c (0xdff2 bytes) 000573 1789684514.628428 read 0x022852 thru 0x026ec4 (0x4673 bytes) 000574 1789684514.642367 fallocate 0x027368 thru 0x030395 (0x902e bytes) 000575 1789684514.654105 mapwrite 0x02f7a5 thru 0x02fc5c (0x4b8 bytes) 000576 1789684514.693476 punch from 0x0382a7 to 0x03a823 (0x257d bytes) 000577 1789684514.715274 mapread 0x013612 thru 0x01d643 (0xa032 bytes) 000578 1789684514.722594 punch from 0x0126a9 to 0x019628 (0x6f80 bytes) 000579 1789684514.740080 write 0x01dd8c thru 0x023815 (0x5a8a bytes) 000580 1789684514.741851 punch from 0x017fa0 to 0x022bdd (0xac3e bytes) 000582 1789684514.768468 trunc from 0x00b81b to 0x019f4d (0xe733 bytes) 000583 1789684514.778435 mapwrite 0x011906 thru 0x01b78a (0x9e85 bytes) 000584 1789684514.891093 read 0x02bf60 thru 0x0339c8 (0x7a69 bytes) 000585 1789684514.892063 punch from 0x0123e7 to 0x021529 (0xf143 bytes) 000586 1789684514.901934 mapwrite 0x009b8c thru 0x00d425 (0x389a bytes) 000587 1789684514.945975 fallocate 0x03affa thru 0x03ffff (0x5006 bytes) 000588 1789684514.964827 trunc from 0x0300f9 to 0x03a823 (0xa72b bytes) 000590 1789684514.973795 mapwrite 0x02a686 thru 0x02f49f (0x4e1a bytes) 000591 1789684515.025004 write 0x004675 thru 0x00801a (0x39a6 bytes) 000592 1789684515.028909 mapread 0x019402 thru 0x0291e5 (0xfde4 bytes) 000593 1789684515.030750 punch from 0x006383 to 0x012546 (0xc1c4 bytes) 000594 1789684515.057566 fallocate 0x03498e thru 0x03fc3d (0xb2b0 bytes) 000595 1789684515.066682 read 0x02e854 thru 0x03a823 (0xbfd0 bytes) 000596 1789684515.107496 MIRROR_RESYNC 4 mirrors 000597 1789684518.602005 trunc from 0x02e56f to 0x02fa7e (0x1510 bytes) 000598 1789684518.635901 write 0x027c11 thru 0x027c34 (0x24 bytes) 000599 1789684518.647911 read 0x0284de thru 0x029652 (0x1175 bytes) 000601 1789684518.661499 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3a824): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a824): Mirror 3: 0x62c54b24 (stale) Mirror 4: 0x62c54b24 (stale) Mirror 5: 0x62c54b24 (stale) CRC-32 checksum value for chunk [0, 0x3a824): Mirror 1: 0x52b2dbaf Mirror 6: 0x52b2dbaf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a824: skipped 000602 1789684520.199578 mapread 0x00d430 thru 0x011dbe (0x498f bytes) 000603 1789684520.261149 punch from 0x016524 to 0x022cb6 (0xc793 bytes) 000604 1789684520.345717 trunc from 0x03a824 to 0x019182 (0x216a1 bytes) 000605 1789684520.396485 mapread 0x016982 thru 0x019182 (0x2801 bytes) 000606 1789684520.418141 mapread 0x01374e thru 0x0168d6 (0x3189 bytes) 000608 1789684520.425685 trunc from 0x019183 to 0x01e753 (0x55d1 bytes) 000609 1789684520.446558 write 0x037f6e thru 0x03ffff (0x8092 bytes) 000610 1789684520.447958 trunc from 0x040000 to 0x005a21 (0x3a5de bytes) 000611 1789684520.476589 mapread 0x0043a8 thru 0x005a21 (0x167a bytes) 000612 1789684520.494127 mapwrite 0x0109c4 thru 0x012d28 (0x2365 bytes) 000613 1789684520.554824 mapwrite 0x02fd61 thru 0x036dfb (0x709b bytes) 000615 1789684520.626109 write 0x03f6c7 thru 0x03ffff (0x939 bytes) 000616 1789684520.629176 write 0x00ab70 thru 0x013e93 (0x9324 bytes) 000617 1789684520.635765 mapwrite 0x0194eb thru 0x026332 (0xce48 bytes) 000618 1789684520.689115 trunc from 0x040000 to 0x03fc5d (0x3a2 bytes) 000619 1789684520.718555 fallocate 0x003bf3 thru 0x00f857 (0xbc65 bytes) 000620 1789684520.728224 trunc from 0x03fbdf to 0x03fc5d (0x7f bytes) 000621 1789684520.743345 write 0x03d655 thru 0x03ffff (0x29ab bytes) 000622 1789684520.753302 write 0x011c88 thru 0x01529c (0x3615 bytes) 000623 1789684520.754434 mapread 0x00f619 thru 0x01276b (0x3153 bytes) 000624 1789684520.770706 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x55657d39 (stale) Mirror 4: 0x55657d39 (stale) Mirror 5: 0x55657d39 (stale) Mirror 6: 0xb20b8335 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4e1aa1a5 Mirror 7: 0x4e1aa1a5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000625 1789684522.575242 write 0x01932e thru 0x0239a7 (0xa67a bytes) 000626 1789684522.678361 trunc from 0x040000 to 0x0016b5 (0x3e94a bytes) 000627 1789684522.716134 punch from 0x001333 to 0x0016b5 (0x383 bytes) 000628 1789684522.730108 trunc from 0x000283 to 0x0016b5 (0x1433 bytes) 000629 1789684522.739028 mapwrite 0x023797 thru 0x026a75 (0x32df bytes) 000630 1789684522.890495 write 0x00ce81 thru 0x01159c (0x471c bytes) 000631 1789684522.916746 read 0x017fda thru 0x01850b (0x532 bytes) 000633 1789684522.946024 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x207:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26a76): Mirror 4: 0x16a9d87d (stale) Mirror 5: 0x16a9d87d (stale) Mirror 6: 0x16a9d87d (stale) Mirror 7: 0xf8b1ea7e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26a76): Mirror 4: 0x16a9d87d (stale) Mirror 5: 0x16a9d87d (stale) Mirror 6: 0x16a9d87d (stale) Mirror 7: 0xf8b1ea7e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26a76: skipped 000634 1789684523.456698 read 0x0201c3 thru 0x026a75 (0x68b3 bytes) 000635 1789684523.468115 read 0x01fc17 thru 0x0201ea (0x5d4 bytes) 000637 1789684523.485695 read 0x001c23 thru 0x00c652 (0xaa30 bytes) 000638 1789684523.489813 fallocate 0x026e88 thru 0x02977c (0x28f5 bytes) 000639 1789684523.502466 punch from 0x00b921 to 0x010ad1 (0x51b1 bytes) 000640 1789684523.509101 mapwrite 0x03ca28 thru 0x03ffff (0x35d8 bytes) 000641 1789684523.558034 punch from 0x01da52 to 0x029d0f (0xc2be bytes) 000644 1789684523.566347 mapwrite 0x02c616 thru 0x031d72 (0x575d bytes) 000645 1789684523.592594 punch from 0x0110e6 to 0x01acd1 (0x9bec bytes) 000646 1789684523.603844 punch from 0x03e50d to 0x03ffff (0x1af3 bytes) 000647 1789684523.616411 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x207:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x55657d39 (stale) Mirror 5: 0x55657d39 (stale) Mirror 6: 0xb20b8335 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x55657d39 (stale) Mirror 5: 0x55657d39 (stale) Mirror 6: 0xb20b8335 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000648 1789684523.913131 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x55657d39 (stale) Mirror 5: 0x55657d39 (stale) Mirror 6: 0xb20b8335 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbda633d5 Mirror 7: 0xbda633d5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000649 1789684525.366982 trunc from 0x0365f0 to 0x03ffff (0x9a10 bytes) 000650 1789684525.437438 trunc from 0x040000 to 0x00afc1 (0x3503e bytes) 000651 1789684525.545068 MIRROR_RESYNC 5 mirrors 000652 1789684526.985651 MIRROR_RESYNC 5 mirrors 000653 1789684527.538105 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x207:0x0]: Invalid argument (22) 000654 1789684528.045163 read 0x0081af thru 0x00afc1 (0x2e13 bytes) 000655 1789684528.050480 mapread 0x0042d6 thru 0x00afc1 (0x6cec bytes) 000656 1789684528.107078 trunc from 0x00afc2 to 0x033d8f (0x28dce bytes) 000657 1789684528.175978 mapwrite 0x03e9c1 thru 0x03ffff (0x163f bytes) 000661 1789684528.253183 read 0x00e931 thru 0x012aed (0x41bd bytes) 000662 1789684528.266999 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2667acd2 Mirror 8: 0x2667acd2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000663 1789684529.706750 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2667acd2 Mirror 8: 0x2667acd2 Mirror 9: 0x2667acd2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000664 1789684531.083839 write 0x013452 thru 0x015c53 (0x2802 bytes) 000665 1789684531.187136 punch from 0x02b68c to 0x034220 (0x8b95 bytes) 000666 1789684531.197312 punch from 0x00adcd to 0x018e51 (0xe085 bytes) 000667 1789684531.206535 write 0x024474 thru 0x02f53e (0xb0cb bytes) 000668 1789684531.227459 trunc from 0x007c9c to 0x007d4b (0xb0 bytes) 000669 1789684531.239084 mapread 0x00096e thru 0x00fb33 (0xf1c6 bytes) 000670 1789684531.260224 mapwrite 0x0105f9 thru 0x0113db (0xde3 bytes) 000671 1789684531.284712 mapread 0x03def0 thru 0x03ffff (0x2110 bytes) 000673 1789684531.297929 trunc from 0x007077 to 0x00869e (0x1628 bytes) 000674 1789684531.303334 read 0x028d66 thru 0x02cfa8 (0x4243 bytes) 000675 1789684531.303951 trunc from 0x02bdc9 to 0x02fa2f (0x3c67 bytes) 000676 1789684531.314975 write 0x030998 thru 0x03a7a6 (0x9e0f bytes) 000677 1789684531.333970 MIRROR_RESYNC 6 mirrors 000678 1789684533.598186 punch from 0x0228ba to 0x023f22 (0x1669 bytes) 000679 1789684533.631212 punch from 0x002383 to 0x0118d9 (0xf557 bytes) 000680 1789684533.640740 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xd8df0730 (stale) Mirror 6: 0xd8df0730 (stale) Mirror 7: 0xd8df0730 (stale) Mirror 8: 0xd8df0730 (stale) Mirror 9: 0xd8df0730 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x99347e72 Mirror 10: 0x99347e72 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000681 1789684534.947812 read 0x012113 thru 0x01ea71 (0xc95f bytes) 000682 1789684534.963850 mapread 0x0327b4 thru 0x037d3c (0x5589 bytes) 000683 1789684535.013023 mapwrite 0x03dbd5 thru 0x03ffff (0x242b bytes) 000685 1789684535.157092 read 0x03f85a thru 0x03ffff (0x7a6 bytes) 000686 1789684535.160538 trunc from 0x03cbbf to 0x03ffff (0x3441 bytes) 000687 1789684535.169261 mapread 0x0140ea thru 0x017d6a (0x3c81 bytes) 000688 1789684535.179424 read 0x011161 thru 0x01f1ab (0xe04b bytes) 000689 1789684535.180091 trunc from 0x040000 to 0x033b62 (0xc49d bytes) 000690 1789684535.198370 fallocate 0x01310f thru 0x01d01c (0x9f0e bytes) 000691 1789684535.206253 MIRROR_RESYNC 7 mirrors 000692 1789684537.259508 trunc from 0x033b63 to 0x010d74 (0x22dee bytes) 000694 1789684537.344294 read 0x0095b3 thru 0x010d74 (0x77c2 bytes) 000695 1789684537.354838 read 0x00ff34 thru 0x010d74 (0xe41 bytes) 000696 1789684537.356264 punch from 0x0050fd to 0x00c838 (0x773c bytes) 000697 1789684537.363234 fallocate 0x017ba5 thru 0x025f56 (0xe3b2 bytes) 000698 1789684537.371740 read 0x00215d thru 0x010d74 (0xec18 bytes) 000699 1789684537.381179 mapread 0x000aa4 thru 0x0071e2 (0x673f bytes) 000700 1789684537.382545 trunc from 0x00d4d8 to 0x010d74 (0x389d bytes) 000701 1789684537.389196 fallocate 0x0045be thru 0x00bcab (0x76ee bytes) 000702 1789684537.403297 mapwrite 0x00d19a thru 0x0152b2 (0x8119 bytes) 000703 1789684537.521074 MIRROR_RESYNC 7 mirrors 000704 1789684538.600104 read 0x011929 thru 0x0152b2 (0x398a bytes) 000705 1789684538.612733 trunc from 0x0152b3 to 0x03c52f (0x2727d bytes) 000706 1789684538.654467 mapwrite 0x01e534 thru 0x02c9e8 (0xe4b5 bytes) 000707 1789684538.700834 mapwrite 0x0174ac thru 0x019f6b (0x2ac0 bytes) 000708 1789684538.733214 fallocate 0x0201dc thru 0x02d85d (0xd682 bytes) 000709 1789684538.744532 punch from 0x0182ac to 0x0205ba (0x830f bytes) 000710 1789684538.749909 trunc from 0x03c530 to 0x02f202 (0xd32d bytes) 000711 1789684538.773479 write 0x027f86 thru 0x02e3f8 (0x6473 bytes) 000712 1789684538.777268 write 0x01f0f4 thru 0x0267a1 (0x76ae bytes) 000713 1789684538.789010 trunc from 0x02f203 to 0x000e38 (0x2e3ca bytes) 000714 1789684538.811247 read 0x000831 thru 0x000e38 (0x608 bytes) 000715 1789684538.828755 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xe39): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe39): Mirror 5: 0x8c190deb (stale) Mirror 6: 0x8c190deb (stale) Mirror 7: 0x8c190deb (stale) Mirror 8: 0x8c190deb (stale) Mirror 9: 0x8c190deb (stale) Mirror 10: 0x8c190deb (stale) CRC-32 checksum value for chunk [0, 0xe39): Mirror 1: 0x8c190deb Mirror 11: 0x8c190deb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe39: skipped 000716 1789684539.677464 write 0x02c41a thru 0x037e49 (0xba30 bytes) 000717 1789684539.722887 mapread 0x017276 thru 0x022177 (0xaf02 bytes) 000718 1789684539.772806 mapread 0x02175b thru 0x025f94 (0x483a bytes) 000719 1789684539.775353 write 0x01fb2a thru 0x02537b (0x5852 bytes) 000720 1789684539.782556 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x207:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37e4a): Mirror 5: 0x49e1406 (stale) Mirror 7: 0x49e1406 (stale) Mirror 8: 0x49e1406 (stale) Mirror 9: 0x49e1406 (stale) Mirror 10: 0x49e1406 (stale) Mirror 11: 0x68e52fd1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37e4a): Mirror 5: 0x49e1406 (stale) Mirror 7: 0x49e1406 (stale) Mirror 8: 0x49e1406 (stale) Mirror 9: 0x49e1406 (stale) Mirror 10: 0x49e1406 (stale) Mirror 11: 0x68e52fd1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37e4a: skipped 000721 1789684540.217951 trunc from 0x00169c to 0x010be9 (0xf54e bytes) 000723 1789684540.225142 write 0x0207c4 thru 0x02a061 (0x989e bytes) 000724 1789684540.261537 fallocate 0x009f1f thru 0x00a2e7 (0x3c9 bytes) 000725 1789684540.271146 trunc from 0x0311c6 to 0x033aca (0x2905 bytes) 000726 1789684540.284410 mapwrite 0x026dc2 thru 0x03075e (0x999d bytes) 000727 1789684540.334707 trunc from 0x029c75 to 0x037e49 (0xe1d5 bytes) 000728 1789684540.340122 read 0x030fd0 thru 0x037e49 (0x6e7a bytes) 000729 1789684540.350856 punch from 0x008ccd to 0x009dc5 (0x10f9 bytes) 000731 1789684540.356904 read 0x000f6b thru 0x007763 (0x67f9 bytes) 000732 1789684540.370209 trunc from 0x005494 to 0x00bedf (0x6a4c bytes) 000733 1789684540.376392 mapread 0x02aa5f thru 0x0358a8 (0xae4a bytes) 000734 1789684540.377872 read 0x001e7f thru 0x00c866 (0xa9e8 bytes) 000735 1789684540.389436 punch from 0x035b74 to 0x037e49 (0x22d6 bytes) 000736 1789684540.401314 MIRROR_RESYNC 7 mirrors 000737 1789684542.019533 mapwrite 0x01bc54 thru 0x0209f2 (0x4d9f bytes) 000738 1789684542.167584 fallocate 0x03499a thru 0x03fb9d (0xb204 bytes) 000739 1789684542.175874 read 0x035b04 thru 0x037e49 (0x2346 bytes) 000740 1789684542.182220 mapwrite 0x039998 thru 0x03a18e (0x7f7 bytes) 000741 1789684542.237015 trunc from 0x032d76 to 0x037c56 (0x4ee1 bytes) 000743 1789684542.243376 trunc from 0x02ba90 to 0x02cf9c (0x150d bytes) 000744 1789684542.247874 read 0x01f4ad thru 0x029714 (0xa268 bytes) 000745 1789684542.257711 trunc from 0x02aedb to 0x035fc7 (0xb0ed bytes) 000746 1789684542.262094 trunc from 0x03a18f to 0x03ed1a (0x4b8c bytes) 000747 1789684542.274436 read 0x014e73 thru 0x01be98 (0x7026 bytes) 000748 1789684542.275162 mapread 0x023e77 thru 0x02d0ad (0x9237 bytes) 000749 1789684542.289391 trunc from 0x02c542 to 0x03ac03 (0xe6c2 bytes) 000750 1789684542.294980 fallocate 0x001efc thru 0x00230a (0x40f bytes) 000751 1789684542.300194 trunc from 0x03ed1b to 0x03709d (0x7c7d bytes) 000752 1789684542.311299 punch from 0x015c19 to 0x017aef (0x1ed7 bytes) 000754 1789684542.315818 mapread 0x0290e3 thru 0x033378 (0xa296 bytes) 000755 1789684542.364537 mapread 0x029eef thru 0x0312ce (0x73e0 bytes) 000756 1789684542.365369 trunc from 0x0288a6 to 0x036096 (0xd7f1 bytes) 000757 1789684542.372300 mapread 0x01db1c thru 0x02030f (0x27f4 bytes) 000759 1789684542.374915 trunc from 0x03709e to 0x015b0b (0x21592 bytes) 000760 1789684542.397091 read 0x009d29 thru 0x015b0b (0xbde3 bytes) 000761 1789684542.403343 punch from 0x00a9a7 to 0x00e2eb (0x3945 bytes) 000762 1789684542.413350 read 0x0072ba thru 0x015b0b (0xe852 bytes) 000763 1789684542.428160 trunc from 0x015b0c to 0x026fd0 (0x114c5 bytes) 000764 1789684542.453223 punch from 0x0202ea to 0x0269c1 (0x66d8 bytes) 000765 1789684542.462634 trunc from 0x026fd1 to 0x00087c (0x26754 bytes) 000766 1789684542.482215 mapread 0x0005c5 thru 0x00087c (0x2b8 bytes) 000768 1789684542.501480 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x207:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x87d): Mirror 5: 0xa7727a95 (stale) Mirror 8: 0xa7727a95 (stale) Mirror 9: 0xa7727a95 (stale) Mirror 10: 0xa7727a95 (stale) Mirror 11: 0xa7727a95 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x87d): Mirror 5: 0xa7727a95 (stale) Mirror 8: 0xa7727a95 (stale) Mirror 9: 0xa7727a95 (stale) Mirror 10: 0xa7727a95 (stale) Mirror 11: 0xa7727a95 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x87d: skipped 000769 1789684542.868402 trunc from 0x00087d to 0x0120a1 (0x11825 bytes) 000770 1789684542.894848 trunc from 0x00cf01 to 0x0120a1 (0x51a1 bytes) 000771 1789684542.901115 mapwrite 0x015c4e thru 0x0240ca (0xe47d bytes) 000772 1789684542.987253 fallocate 0x031edf thru 0x03bcf9 (0x9e1b bytes) 000773 1789684543.010505 trunc from 0x03bcfa to 0x00724a (0x34aaf bytes) 000774 1789684543.040943 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x207:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x724b): Mirror 5: 0xfb794947 (stale) Mirror 8: 0xfb794947 (stale) Mirror 10: 0xfb794947 (stale) Mirror 11: 0xfb794947 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x724b): Mirror 5: 0xfb794947 (stale) Mirror 8: 0xfb794947 (stale) Mirror 10: 0xfb794947 (stale) Mirror 11: 0xfb794947 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x724b: skipped 000775 1789684543.363417 mapread 0x000e7b thru 0x002017 (0x119d bytes) 000776 1789684543.380570 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x207:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x724b): Mirror 8: 0xfb794947 (stale) Mirror 10: 0xfb794947 (stale) Mirror 11: 0xfb794947 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x724b): Mirror 8: 0xfb794947 (stale) Mirror 10: 0xfb794947 (stale) Mirror 11: 0xfb794947 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x724b: skipped 000777 1789684543.823308 trunc from 0x00039e to 0x00724a (0x6ead bytes) 000778 1789684543.834660 fallocate 0x00abd3 thru 0x00d13b (0x2569 bytes) 000779 1789684543.848114 MIRROR_RESYNC 4 mirrors 000780 1789684544.704802 punch from 0x004561 to 0x00724a (0x2cea bytes) 000781 1789684544.740117 mapread 0x00244d thru 0x00724a (0x4dfe bytes) 000782 1789684544.758407 mapread 0x004541 thru 0x00709b (0x2b5b bytes) 000783 1789684544.760045 trunc from 0x00724b to 0x007b5f (0x915 bytes) 000785 1789684544.785233 punch from 0x003aa1 to 0x003cba (0x21a bytes) 000786 1789684544.792008 mapwrite 0x005b68 thru 0x01339c (0xd835 bytes) 000787 1789684544.864034 write 0x000204 thru 0x010072 (0xfe6f bytes) 000788 1789684544.873985 punch from 0x006e58 to 0x01339c (0xc545 bytes) 000789 1789684544.889496 trunc from 0x000a9a to 0x002523 (0x1a8a bytes) 000790 1789684544.895043 trunc from 0x00db54 to 0x01339c (0x5849 bytes) 000791 1789684544.900400 punch from 0x00df28 to 0x012b50 (0x4c29 bytes) 000792 1789684544.905222 write 0x017c2d thru 0x019dd2 (0x21a6 bytes) 000793 1789684544.906571 punch from 0x003e58 to 0x012cd7 (0xee80 bytes) 000794 1789684544.911927 mapwrite 0x03ef0d thru 0x03ffff (0x10f3 bytes) 000795 1789684544.952983 punch from 0x0303ff to 0x03409d (0x3c9f bytes) 000796 1789684544.959063 mapread 0x0198e0 thru 0x01f3ad (0x5ace bytes) 000797 1789684544.973168 punch from 0x01321a to 0x01f335 (0xc11c bytes) 000798 1789684544.978970 punch from 0x0241b5 to 0x02897a (0x47c6 bytes) 000799 1789684544.983664 trunc from 0x040000 to 0x01a479 (0x25b86 bytes) 000800 1789684545.003110 trunc from 0x003742 to 0x0122ab (0xeb6a bytes) 000801 1789684545.011799 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x207:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a47a): Mirror 10: 0xf01e646e (stale) Mirror 11: 0xf01e646e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a47a): Mirror 10: 0xf01e646e (stale) Mirror 11: 0xf01e646e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a47a: skipped 000802 1789684545.414192 read 0x010bc8 thru 0x012233 (0x166c bytes) 000803 1789684545.424282 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x207:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a47a): Mirror 10: 0xf01e646e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a47a): Mirror 10: 0xf01e646e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a47a: skipped 000804 1789684545.797906 trunc from 0x012cf0 to 0x0133cf (0x6e0 bytes) 000805 1789684545.807747 read 0x01298a thru 0x01a479 (0x7af0 bytes) 000806 1789684545.830437 read 0x00417a thru 0x012239 (0xe0c0 bytes) 000807 1789684545.841230 mapread 0x010d37 thru 0x0133d2 (0x269c bytes) 000808 1789684545.847855 MIRROR_RESYNC 2 mirrors 000809 1789684546.547630 trunc from 0x01a47a to 0x00bf57 (0xe522 bytes) 000810 1789684546.576983 read 0x006cdb thru 0x00bf57 (0x527d bytes) 000811 1789684546.593140 write 0x024849 thru 0x028a4a (0x4202 bytes) 000812 1789684546.595308 fallocate 0x034bfe thru 0x03c532 (0x7935 bytes) 000813 1789684546.602339 trunc from 0x028a4b to 0x00c386 (0x1c6c4 bytes) 000814 1789684546.636999 mapwrite 0x00050c thru 0x00c0dd (0xbbd2 bytes) 000815 1789684546.682757 mapwrite 0x015c1d thru 0x019bbf (0x3fa3 bytes) 000817 1789684546.739313 read 0x00b8bb thru 0x00db5d (0x22a3 bytes) 000819 1789684546.746238 trunc from 0x019bc0 to 0x0264a8 (0xc8e9 bytes) 000820 1789684546.763071 write 0x00a57f thru 0x013ba2 (0x9624 bytes) 000822 1789684546.765394 punch from 0x015e17 to 0x025046 (0xf230 bytes) 000823 1789684546.770244 trunc from 0x0264a9 to 0x02fbf3 (0x974b bytes) 000825 1789684546.783294 trunc from 0x01be61 to 0x024121 (0x82c1 bytes) 000826 1789684546.788040 read 0x0144d6 thru 0x022b15 (0xe640 bytes) 000827 1789684546.797064 mapread 0x01fbea thru 0x0221c7 (0x25de bytes) 000828 1789684546.799306 fallocate 0x0234ec thru 0x02b368 (0x7e7d bytes) 000829 1789684546.805133 mapread 0x0028fd thru 0x00d3c3 (0xaac7 bytes) 000831 1789684546.809656 punch from 0x02c292 to 0x02fbf3 (0x3962 bytes) 000832 1789684546.816107 fallocate 0x00bc67 thru 0x01756c (0xb906 bytes) 000833 1789684546.823846 mapwrite 0x0300c6 thru 0x03da01 (0xd93c bytes) 000835 1789684546.876827 mapread 0x00731f thru 0x00aed9 (0x3bbb bytes) 000836 1789684546.884859 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x207:0x0]: Invalid argument (22) 000838 1789684547.031185 trunc from 0x03da02 to 0x03c6f2 (0x130f bytes) 000839 1789684547.050654 fallocate 0x018303 thru 0x01b1b5 (0x2eb3 bytes) 000840 1789684547.058964 read 0x005b20 thru 0x015912 (0xfdf3 bytes) 000841 1789684547.073637 mapread 0x00d473 thru 0x0144d1 (0x705f bytes) 000843 1789684547.082274 mapread 0x00c0dc thru 0x014937 (0x885c bytes) 000844 1789684547.084353 mapwrite 0x00d6d5 thru 0x01b2b2 (0xdbde bytes) 000846 1789684547.163173 mapread 0x0151f7 thru 0x01d8fb (0x8705 bytes) 000847 1789684547.165825 punch from 0x019fb2 to 0x01ce32 (0x2e81 bytes) 000848 1789684547.173352 write 0x03c134 thru 0x03ffff (0x3ecc bytes) 000849 1789684547.183411 fallocate 0x021800 thru 0x02237b (0xb7c bytes) 000850 1789684547.188507 trunc from 0x0048fc to 0x011d96 (0xd49b bytes) 000851 1789684547.194058 write 0x03e1a1 thru 0x03ffff (0x1e5f bytes) 000852 1789684547.195029 read 0x027678 thru 0x03210d (0xaa96 bytes) 000854 1789684547.213801 MIRROR_EXTEND to 2 mirrors 000855 1789684547.854152 trunc from 0x009102 to 0x009548 (0x447 bytes) 000856 1789684547.905969 read 0x0334be thru 0x03ffff (0xcb42 bytes) 000857 1789684547.922391 write 0x028628 thru 0x03294f (0xa328 bytes) 000858 1789684547.944501 read 0x03b472 thru 0x03b863 (0x3f2 bytes) 000859 1789684547.945181 mapwrite 0x00a057 thru 0x016da3 (0xcd4d bytes) 000860 1789684548.002255 write 0x0044f1 thru 0x01426d (0xfd7d bytes) 000861 1789684548.007641 trunc from 0x040000 to 0x015c94 (0x2a36b bytes) 000862 1789684548.055829 fallocate 0x00c0a4 thru 0x017a6a (0xb9c7 bytes) 000863 1789684548.066656 trunc from 0x01684c to 0x017a6a (0x121f bytes) 000864 1789684548.074709 mapwrite 0x029493 thru 0x02f0b7 (0x5c25 bytes) 000865 1789684548.205507 write 0x00798d thru 0x00ec26 (0x729a bytes) 000866 1789684548.209239 fallocate 0x03ea02 thru 0x03ffff (0x15fe bytes) 000867 1789684548.216818 mapread 0x0289b6 thru 0x030101 (0x774c bytes) 000868 1789684548.230487 mapread 0x01a249 thru 0x0266dd (0xc495 bytes) 000869 1789684548.310399 read 0x0326f0 thru 0x03ffff (0xd910 bytes) 000870 1789684548.329460 read 0x02d0e0 thru 0x03a97f (0xd8a0 bytes) 000871 1789684548.330356 fallocate 0x039065 thru 0x03ffff (0x6f9b bytes) 000873 1789684548.337331 read 0x030fbf thru 0x03e193 (0xd1d5 bytes) 000874 1789684548.337951 trunc from 0x000593 to 0x009eb9 (0x9927 bytes) 000875 1789684548.356951 mapread 0x025835 thru 0x028ef3 (0x36bf bytes) 000876 1789684548.358078 mapread 0x0151b4 thru 0x017ea5 (0x2cf2 bytes) 000877 1789684548.360288 punch from 0x0187e0 to 0x022790 (0x9fb1 bytes) 000878 1789684548.366549 write 0x032af6 thru 0x03354c (0xa57 bytes) 000879 1789684548.367774 read 0x03b44d thru 0x03ffff (0x4bb3 bytes) 000880 1789684548.369484 trunc from 0x040000 to 0x027bf1 (0x1840e bytes) 000881 1789684548.391743 mapread 0x007bc4 thru 0x00a528 (0x2965 bytes) 000882 1789684548.411988 fallocate 0x01d6ec thru 0x028e47 (0xb75c bytes) 000883 1789684548.420397 write 0x0220e6 thru 0x025c9a (0x3bb5 bytes) 000884 1789684548.428128 punch from 0x007089 to 0x00854e (0x14c6 bytes) 000885 1789684548.433904 trunc from 0x01b5da to 0x01d8b7 (0x22de bytes) 000886 1789684548.439586 trunc from 0x0042cf to 0x007777 (0x34a9 bytes) 000887 1789684548.445109 trunc from 0x028e48 to 0x03007b (0x7234 bytes) 000889 1789684548.458764 trunc from 0x03007c to 0x02dafa (0x2581 bytes) 000890 1789684548.473009 read 0x025f3b thru 0x02dafa (0x7bc0 bytes) 000891 1789684548.485299 trunc from 0x00eb93 to 0x01be78 (0xd2e6 bytes) 000892 1789684548.491174 punch from 0x014b56 to 0x01dc4c (0x90f7 bytes) 000893 1789684548.495713 read 0x022724 thru 0x026988 (0x4265 bytes) 000894 1789684548.499918 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x207:0x0]: Invalid argument (22) 000897 1789684548.659583 punch from 0x01b8e5 to 0x023ab4 (0x81d0 bytes) 000898 1789684548.666121 trunc from 0x02dafb to 0x000fc6 (0x2cb34 bytes) 000899 1789684548.686552 write 0x01aeb4 thru 0x020b09 (0x5c56 bytes) 000901 1789684548.690983 punch from 0x0186d0 to 0x020b09 (0x843a bytes) 000902 1789684548.739790 MIRROR_EXTEND to 2 mirrors 000904 1789684549.321221 mapwrite 0x018a12 thru 0x0259d0 (0xcfbf bytes) 000906 1789684549.447813 read 0x0120d9 thru 0x01e6ab (0xc5d3 bytes) 000907 1789684549.454547 write 0x003401 thru 0x012501 (0xf101 bytes) 000909 1789684549.462176 write 0x02aa9c thru 0x038c6c (0xe1d1 bytes) 000910 1789684549.465151 punch from 0x015832 to 0x020701 (0xaed0 bytes) 000911 1789684549.470218 fallocate 0x012ad3 thru 0x01a438 (0x7966 bytes) 000912 1789684549.474845 read 0x011490 thru 0x01c473 (0xafe4 bytes) 000914 1789684549.488796 read 0x0348a9 thru 0x038c6c (0x43c4 bytes) 000915 1789684549.489358 read 0x01e731 thru 0x0223fb (0x3ccb bytes) 000916 1789684549.490898 trunc from 0x038c6d to 0x039af5 (0xe89 bytes) 000917 1789684549.508049 mapread 0x024f7e thru 0x029539 (0x45bc bytes) 000918 1789684549.509012 fallocate 0x02e343 thru 0x03b56e (0xd22c bytes) 000919 1789684549.512447 mapread 0x026b54 thru 0x0337eb (0xcc98 bytes) 000920 1789684549.516992 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b56f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b56f): Mirror 2: 0xe4a85a5b (stale) CRC-32 checksum value for chunk [0, 0x3b56f): Mirror 1: 0x93d8afe2 Mirror 3: 0x93d8afe2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b56f: skipped 000921 1789684550.523463 trunc from 0x00b430 to 0x0146a3 (0x9274 bytes) 000922 1789684550.573823 read 0x03b317 thru 0x03b56e (0x258 bytes) 000923 1789684550.580012 punch from 0x00ce35 to 0x013d4d (0x6f19 bytes) 000924 1789684550.584943 mapwrite 0x0229b7 thru 0x02d9ca (0xb014 bytes) 000925 1789684550.629195 read 0x01aab1 thru 0x027355 (0xc8a5 bytes) 000926 1789684550.645710 punch from 0x034726 to 0x03afc4 (0x689f bytes) 000927 1789684550.650001 write 0x009631 thru 0x0126ba (0x908a bytes) 000928 1789684550.656373 fallocate 0x01fce8 thru 0x02477f (0x4a98 bytes) 000929 1789684550.670563 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b56f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b56f): Mirror 2: 0xe4a85a5b (stale) Mirror 3: 0x93d8afe2 (stale) CRC-32 checksum value for chunk [0, 0x3b56f): Mirror 1: 0x7ad2756b Mirror 4: 0x7ad2756b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b56f: skipped 000930 1789684551.469465 punch from 0x031dba to 0x035da8 (0x3fef bytes) 000931 1789684551.517654 trunc from 0x03b56f to 0x01b223 (0x2034b bytes) 000932 1789684551.535370 trunc from 0x00f746 to 0x013009 (0x38c4 bytes) 000933 1789684551.543147 punch from 0x018220 to 0x01b223 (0x3004 bytes) 000934 1789684551.551150 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x207:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b224): Mirror 3: 0xd0e9cfff (stale) Mirror 4: 0x72e0d965 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b224): Mirror 3: 0xd0e9cfff (stale) Mirror 4: 0x72e0d965 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b224: skipped 000935 1789684551.859284 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x207:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b224): Mirror 4: 0x72e0d965 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b224): Mirror 4: 0x72e0d965 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b224: skipped 000936 1789684552.172034 fallocate 0x0163b1 thru 0x018338 (0x1f88 bytes) 000937 1789684552.179740 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x207:0x0]: Invalid argument (22) 000938 1789684552.334062 mapread 0x00a06f thru 0x011f15 (0x7ea7 bytes) 000939 1789684552.361416 trunc from 0x01b224 to 0x00d8eb (0xd938 bytes) 000940 1789684552.381140 trunc from 0x00d8ec to 0x036901 (0x29016 bytes) 000942 1789684552.399121 mapwrite 0x03f24b thru 0x03ffff (0xdb5 bytes) 000943 1789684552.453522 trunc from 0x040000 to 0x0113fb (0x2ec04 bytes) 000944 1789684552.490865 fallocate 0x02f01e thru 0x0366c8 (0x76ab bytes) 000945 1789684552.500159 punch from 0x0044e7 to 0x0113fb (0xcf15 bytes) 000946 1789684552.507744 fallocate 0x02e9b8 thru 0x038e17 (0xa460 bytes) 000947 1789684552.516710 trunc from 0x028a5f to 0x02fa8c (0x702e bytes) 000948 1789684552.525475 trunc from 0x0027b0 to 0x0036a4 (0xef5 bytes) 000949 1789684552.531698 mapread 0x02374c thru 0x02ff1b (0xc7d0 bytes) 000952 1789684552.568945 MIRROR_EXTEND to 2 mirrors 000953 1789684553.140724 mapread 0x010545 thru 0x01ec59 (0xe715 bytes) 000954 1789684553.189772 read 0x001de9 thru 0x00331f (0x1537 bytes) 000955 1789684553.209334 MIRROR_RESYNC 2 mirrors 000956 1789684553.501885 trunc from 0x038e18 to 0x017219 (0x21bfe bytes) 000957 1789684553.579646 mapread 0x006374 thru 0x0131c3 (0xce50 bytes) 000958 1789684553.612674 read 0x002f5d thru 0x010867 (0xd90b bytes) 000960 1789684553.619350 mapread 0x010e15 thru 0x017219 (0x6405 bytes) 000961 1789684553.621541 mapread 0x00a0c3 thru 0x011071 (0x6faf bytes) 000962 1789684553.623869 mapread 0x00c0cb thru 0x00eec4 (0x2dfa bytes) 000963 1789684553.629255 MIRROR_RESYNC 2 mirrors 000964 1789684554.256534 trunc from 0x01721a to 0x02fc84 (0x18a6b bytes) 000965 1789684554.300994 trunc from 0x004df2 to 0x00e5c4 (0x97d3 bytes) 000966 1789684554.308087 mapread 0x003503 thru 0x010f3e (0xda3c bytes) 000967 1789684554.340787 MIRROR_RESYNC 2 mirrors 000968 1789684554.880659 read 0x0188f6 thru 0x022077 (0x9782 bytes) 000969 1789684554.895097 MIRROR_RESYNC 2 mirrors 000970 1789684555.225453 fallocate 0x01764e thru 0x01ee88 (0x783b bytes) 000971 1789684555.263585 trunc from 0x02fc85 to 0x0253c7 (0xa8bd bytes) 000972 1789684555.302021 read 0x014f97 thru 0x020fe1 (0xc04b bytes) 000973 1789684555.311219 trunc from 0x0253c8 to 0x013b10 (0x118b7 bytes) 000974 1789684555.328293 trunc from 0x00dbad to 0x011acd (0x3f21 bytes) 000975 1789684555.333906 mapwrite 0x021761 thru 0x02f7f3 (0xe093 bytes) 000976 1789684555.402705 mapwrite 0x0089b8 thru 0x00bcbd (0x3306 bytes) 000977 1789684555.428474 punch from 0x01ed3e to 0x02d24a (0xe50d bytes) 000978 1789684555.435009 fallocate 0x009888 thru 0x00b31c (0x1a95 bytes) 000979 1789684555.439343 mapread 0x01761a thru 0x01a4f3 (0x2eda bytes) 000980 1789684555.451332 write 0x02e287 thru 0x030f42 (0x2cbc bytes) 000981 1789684555.459089 MIRROR_RESYNC 2 mirrors 000982 1789684556.169952 trunc from 0x010ef6 to 0x016d69 (0x5e74 bytes) 000983 1789684556.214941 write 0x032c18 thru 0x03a390 (0x7779 bytes) 000985 1789684556.218890 trunc from 0x038af0 to 0x03a390 (0x18a1 bytes) 000986 1789684556.238878 write 0x039d20 thru 0x03b512 (0x17f3 bytes) 000987 1789684556.245588 punch from 0x01c754 to 0x02a300 (0xdbad bytes) 000988 1789684556.251331 fallocate 0x03f904 thru 0x03ffff (0x6fc bytes) 000989 1789684556.257979 fallocate 0x00db1f thru 0x015763 (0x7c45 bytes) 000990 1789684556.264932 trunc from 0x01ad69 to 0x02722c (0xc4c4 bytes) 000991 1789684556.280029 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b513): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b513): Mirror 2: 0x76c3681e (stale) CRC-32 checksum value for chunk [0, 0x3b513): Mirror 1: 0x6681044d Mirror 3: 0x6681044d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b513: skipped 000992 1789684557.191931 punch from 0x03054c to 0x0369a5 (0x645a bytes) 000993 1789684557.251079 mapwrite 0x00f26c thru 0x018dbb (0x9b50 bytes) 000994 1789684557.316517 trunc from 0x03b513 to 0x004093 (0x3747f bytes) 000995 1789684557.365107 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x207:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x4094): Mirror 3: 0x30ec18b9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4094): Mirror 3: 0x30ec18b9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4094: skipped 000996 1789684557.757939 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x4094): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4094): Mirror 3: 0x30ec18b9 (stale) CRC-32 checksum value for chunk [0, 0x4094): Mirror 1: 0x30ec18b9 Mirror 4: 0x30ec18b9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4094: skipped 000997 1789684558.678177 mapread 0x001b17 thru 0x004093 (0x257d bytes) 000998 1789684558.733000 write 0x001d1a thru 0x005514 (0x37fb bytes) 000999 1789684558.788846 write 0x02c472 thru 0x03907f (0xcc0e bytes) 001000 1789684558.792134 punch from 0x031757 to 0x0358b0 (0x415a bytes) All operations completed A-OK! PASS 70a (88s) == sanity-flr test 71: check mirror extend parent fid ==== 18:36:03 (1789684563) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x23c:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (9s) == sanity-flr test 72: check append on FLR file ========== 18:36:12 (1789684572) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.184055 s, 11.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112927 s, 9.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (7s) == sanity-flr test 200a: stress test ===================== 18:36:19 (1789684579) Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre2 Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 6702528 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5948896 .. Extending file size to 8201312 .. Extending file size to 8839360 .. Extending file size to 7382944 .. Extending file size to 8431840 .. Extending file size to 9177920 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8748416 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8822368 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 61662 61663 61665 61668 61669 192.168.204.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre2 (opts:) 192.168.204.157@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 18:37:38 (1789684658) Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre2 Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3438880 .. Extending file size to 3053024 .. done Extending file size to 4372928 .. Extending file size to 8292672 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5721184 .. Extending file size to 6740576 .. Extending file size to 9026784 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8556064 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9109504 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 63241 63242 63243 63246 63247 done 192.168.204.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre2 (opts:) 192.168.204.157@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (74s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 18:38:52 (1789684732) Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2292 1269012 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2244 1269060 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18756 3588264 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24620 3582400 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43376 7170664 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2292 1269012 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2244 1269060 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18756 3588264 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24620 3582400 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43376 7170664 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0923678 s, 22.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.190712 s, 11.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x44537f0c (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre2 (opts:) PASS 200c (12s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 18:39:05 (1789684745) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.102067 s, 20.5 MB/s PASS 202 (5s) == sanity-flr test 203: mirror file preserve mirror ID === 18:39:11 (1789684751) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5c:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14a:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14a:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14b:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } PASS 203 (5s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 18:39:15 (1789684755) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0545777 s, 38.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2d8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2d9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (5s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 18:39:20 (1789684760) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0697068 s, 30.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2da:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2da:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2db:0x0] } PASS 204b (5s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 18:39:25 (1789684765) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0601812 s, 34.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } PASS 204c (28s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 18:39:53 (1789684793) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg457-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662392 s, 15.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } PASS 204d (43s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 18:40:36 (1789684836) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg457-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0470273 s, 44.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (44s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 18:41:20 (1789684880) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg457-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg457-server: Pool lustre.test_204f created Adding targets to pool oleg457-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg457-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0538359 s, 39.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg457-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg457-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg457-server: Pool lustre.test_204f destroyed PASS 204f (57s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 18:42:18 (1789684938) PASS 205a (4s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 18:42:22 (1789684942) PASS 205b (5s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 18:42:27 (1789684947) oleg457-server: Pool lustre.test_206 created Adding targets to pool oleg457-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg457-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg457-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg457-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg457-server: Pool lustre.test_206 destroyed PASS 206 (20s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 18:42:47 (1789684967) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532931 s, 19.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310629 s, 33.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x5c22aaaf Mirror 3: 0x5c22aaaf PASS 207 (6s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 18:42:53 (1789684973) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 18:42:55 (1789684975) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 18:42:57 (1789684977) Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0303835 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x13d0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1740000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1740000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1740000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2110000 Read this (4 bytes): BBBB 192.168.204.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre2 (opts:) PASS 209a (22s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 18:43:19 (1789684999) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00829102 s, 988 kB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 18:43:22 (1789685002) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272246 s, 38.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7d lmm_fid: [0x200000405:0x7d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7d lmm_fid: [0x200000405:0x7d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11c:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611261 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-17 18:43:23.000000000 -0400 Modify: 2026-09-17 18:43:23.000000000 -0400 Change: 2026-09-17 18:43:24.000000000 -0400 Birth: 2026-09-17 18:43:23.000000000 -0400 PASS 210a (5s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 18:43:27 (1789685007) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264754 s, 39.6 MB/s oleg457-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg457-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg457-server: O/280000401/d28/15c: File not found by ext2_lookup PASS 210b (16s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 18:43:43 (1789685023) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.166009 s, 247 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0149271 s, 274 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 17 18:43 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x82:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 17 18:43 /mnt/lustre/f211.sanity-flr PASS 211 (5s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 18:43:48 (1789685028) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (4s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 18:43:52 (1789685032) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (6s) == sanity-flr test complete, duration 1654 sec =========== 18:43:58 (1789685038) === sanity-flr: start cleanup 18:43:59 (1789685039) === === sanity-flr: finish cleanup 18:44:04 (1789685044) ===