-----============= acceptance-small: sanity-flr ============----- Fri Jun 26 09:33:06 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg437-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 09:33:27 (1782480807) === oleg437-client.virtnet: executing check_config_client /mnt/lustre oleg437-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg437-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff919ba06ef800.idle_timeout=debug osc.lustre-OST0001-osc-ffff919ba06ef800.idle_timeout=debug disable quota as required oleg437-server: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 09:34:09 (1782480849) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8266] == sanity-flr test 0a: lfs mirror create with -N option == 09:34:11 (1782480851) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:34:28 (1782480868) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:34:32 (1782480872) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:34:35 (1782480875) PASS 0d (22s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:34:57 (1782480897) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:35:00 (1782480900) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 09:35:04 (1782480904) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff919ba06ef800.stats=clear osc.lustre-OST0001-osc-ffff919ba06ef800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:35:12 (1782480912) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:35:22 (1782480922) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:35:31 (1782480931) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 09:35:34 (1782480934) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:35:42 (1782480942) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 09:35:50 (1782480950) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:35:58 (1782480958) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:36:05 (1782480965) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:36:12 (1782480972) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:36:23 (1782480983) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:36:30 (1782480990) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 1.19286 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125638 s, 8.3 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:36:38 (1782480998) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 0.616162 s, 30.6 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 09:36:46 (1782481006) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:36:56 (1782481016) Stopping /mnt/lustre-ost1 (opts:) on oleg437-server oleg437-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff919ba06ef800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff919ba06ef800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg437-server: oleg437-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg437-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg437-server oleg437-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff919ba06ef800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff919ba06ef800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg437-server: oleg437-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg437-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (73s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:38:09 (1782481089) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.365163 s, 23.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg437-server oleg437-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff919ba06ef800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff919ba06ef800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg437-server: oleg437-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg437-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (45s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:38:54 (1782481134) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg437-server oleg437-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff919ba06ef800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff919ba06ef800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg437-server: oleg437-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg437-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg437-server oleg437-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff919ba06ef800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff919ba06ef800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg437-server: oleg437-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg437-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (83s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:40:17 (1782481217) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.86311 s, 18.3 MB/s -rw-r--r-- 1 root root 52428800 Jun 26 09:40 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.021s user 0m0.001s sys 0m0.179s Stopping /mnt/lustre-ost1 (opts:) on oleg437-server oleg437-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff919ba06ef800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff919ba06ef800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.059s user 0m0.001s sys 0m0.181s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg437-server: oleg437-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg437-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg437-server oleg437-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff919ba06ef800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff919ba06ef800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.009s user 0m0.000s sys 0m0.174s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg437-server: oleg437-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg437-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg437-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff919ba06ef800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff919ba06ef800.ost_server_uuid in FULL state after 0 sec PASS 33b (100s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:41:57 (1782481317) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:42:01 (1782481321) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:42:06 (1782481326) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 09:42:09 (1782481329) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 09:42:18 (1782481338) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.635941 s, 19.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:42:31 (1782481351) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231637 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226107 s, 46.4 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:42:38 (1782481358) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:42:47 (1782481367) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000854905 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00120249 s, 852 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00104145 s, 983 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00174463 s, 587 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 09:42:57 (1782481377) create test files with size 15759045 .. 7805d0ced08b068a17ccf69cb86be2f4 - b4160c94e573f6b7171da9f17c154135 - e4c5e59cac6764d76d8f836d62d2795f - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg437-client.virtnet /mnt/lustre (opts:) Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 09:43:19 (1782481399) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:43:33 (1782481413) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:43:40 (1782481420) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674597 s, 15.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314053 s, 33.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 09:43:50 (1782481430) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0560676 s, 37.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.311626 s, 13.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.206668 s, 10.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 09:44:04 (1782481444) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 09:44:08 (1782481448) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:44:11 (1782481451) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460681 s, 22.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 09:44:18 (1782481458) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.263647 s, 11.9 MB/s PASS 44a (9s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:44:28 (1782481468) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:44:36 (1782481476) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.323124 s, 32.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg437-client.virtnet /mnt/lustre (opts:) Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:44:47 (1782481487) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff919b866a7800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.311958 s, 33.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg437-client.virtnet /mnt/lustre (opts:) Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff919b91161000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:45:02 (1782481502) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (34s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:45:36 (1782481536) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (31s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:46:07 (1782481567) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 09:46:10 (1782481570) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.264225 s, 11.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.190568 s, 16.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x5b940785 (nosync,stale) Mirror 2: 0x5b940785 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 2cb7f965ebcb3420af9150735a1e8891 - mirror 1 checksum: 2c4618af27d88f709fc15fceb763586c - mirror 2 checksum: 2cb7f965ebcb3420af9150735a1e8891 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-26T09:46:13-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-26T09:46:13-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 2cb7f965ebcb3420af9150735a1e8891 - mirror 1 checksum: 2c4618af27d88f709fc15fceb763586c - mirror 2 checksum: 2c4618af27d88f709fc15fceb763586c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-26T09:46:13-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-26T09:46:13-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 09:46:21 (1782481581) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307151 s, 34.1 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 397312.. 398335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:46:28 (1782481588) Failing mds1 on oleg437-server Stopping /mnt/lustre-mds1 (opts:) on oleg437-server 09:46:40 (1782481600) shut down facet: mds1 facet_host: oleg437-server facet_failover_host: oleg437-server Failover mds1 to oleg437-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg437-server: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-MDT0000 09:47:09 (1782481629) targets are mounted 09:47:09 (1782481629) facet_failover done oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299591 s, 35.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } PASS 50A (55s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:47:23 (1782481643) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.049777 s, 411 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0201519 s, 1.0 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (8s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:47:31 (1782481651) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0385201 s, 532 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0127893 s, 1.6 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (7s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:47:38 (1782481658) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0118766 s, 1.4 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:47:45 (1782481665) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0375356 s, 546 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (7s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:47:52 (1782481672) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0108265 s, 378 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0174582 s, 235 kB/s PASS 60a (6s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:47:58 (1782481678) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) PASS 60b (16s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:48:14 (1782481694) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (25s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:48:39 (1782481719) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x19:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (21s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 09:49:00 (1782481740) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (21s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 09:49:21 (1782481761) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 09:49:27 (1782481767) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57386 mirror split pid 57388 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 57386 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 57388 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (65s) == sanity-flr test 70a: flr mode fsx test ================ 09:50:32 (1782481832) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 1834 000001 1782481834.382532 mapwrite 0x037f34 thru 0x0397dd (0x18aa bytes) fallocating to largest ever: 0x40000 000002 1782481834.467684 fallocate 0x03fe64 thru 0x03ffff (0x19c bytes) 000003 1782481834.471913 punch from 0x021645 to 0x02ef10 (0xd8cc bytes) 000005 1782481834.483691 punch from 0x009a58 to 0x0136a4 (0x9c4d bytes) 000006 1782481834.491104 trunc from 0x0397de to 0x024281 (0x1555c bytes) 000007 1782481834.513895 mapwrite 0x03a1c6 thru 0x03c44d (0x2288 bytes) 000008 1782481834.571710 mapread 0x0390a4 thru 0x03c44d (0x33aa bytes) 000009 1782481834.573002 write 0x03be4d thru 0x03c5e4 (0x798 bytes) 000010 1782481834.574464 trunc from 0x036301 to 0x03c5e4 (0x62e4 bytes) 000011 1782481834.588768 write 0x016215 thru 0x01cd60 (0x6b4c bytes) 000012 1782481834.605345 mapwrite 0x037324 thru 0x03ffff (0x8cdc bytes) 000013 1782481834.684417 write 0x0072a5 thru 0x00f990 (0x86ec bytes) 000015 1782481834.704146 read 0x0184c9 thru 0x028473 (0xffab bytes) 000018 1782481834.724187 read 0x004dc8 thru 0x006735 (0x196e bytes) 000019 1782481834.733899 trunc from 0x000d4e to 0x00c9ad (0xbc60 bytes) 000021 1782481834.750245 fallocate 0x02c81d thru 0x02fed5 (0x36b9 bytes) 000022 1782481834.755370 write 0x023a5e thru 0x02b533 (0x7ad6 bytes) 000023 1782481834.757515 fallocate 0x034d3f thru 0x03a62c (0x58ee bytes) 000024 1782481834.762612 write 0x01f95e thru 0x022416 (0x2ab9 bytes) 000025 1782481834.764298 read 0x03febc thru 0x03ffff (0x144 bytes) 000026 1782481834.765054 trunc from 0x03b708 to 0x03b9e7 (0x2e0 bytes) 000027 1782481834.769577 trunc from 0x0325fa to 0x034116 (0x1b1d bytes) 000028 1782481834.774020 mapwrite 0x034f3d thru 0x035cf9 (0xdbd bytes) 000029 1782481834.805613 mapread 0x0034b6 thru 0x008af9 (0x5644 bytes) 000030 1782481834.807434 write 0x028a36 thru 0x02fc1d (0x71e8 bytes) 000031 1782481834.809057 punch from 0x0104d9 to 0x010732 (0x25a bytes) 000032 1782481834.814004 trunc from 0x040000 to 0x01bfbc (0x24043 bytes) 000033 1782481834.853097 MIRROR_EXTEND to 2 mirrors 000034 1782481835.508445 write 0x0332df thru 0x033e59 (0xb7b bytes) 000035 1782481835.530372 punch from 0x03260a to 0x033e59 (0x1850 bytes) 000036 1782481835.542374 write 0x00c192 thru 0x01b2e1 (0xf150 bytes) 000038 1782481835.555499 read 0x012f66 thru 0x022e51 (0xfeec bytes) 000039 1782481835.572953 punch from 0x007549 to 0x00e812 (0x72ca bytes) 000040 1782481835.591859 trunc from 0x01296c to 0x01b736 (0x8dcb bytes) 000041 1782481835.597528 trunc from 0x033e5a to 0x023c3b (0x1021e bytes) 000042 1782481835.618563 trunc from 0x0235c4 to 0x023c3b (0x678 bytes) 000043 1782481835.623514 punch from 0x007bf5 to 0x008a7c (0xe88 bytes) 000044 1782481835.634647 trunc from 0x01326d to 0x015a85 (0x2819 bytes) 000045 1782481835.644275 fallocate 0x008e44 thru 0x00c61f (0x37dc bytes) 000046 1782481835.652082 mapwrite 0x00c364 thru 0x019e7f (0xdb1c bytes) 000047 1782481835.734177 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4e:0x0]: Invalid argument (22) 000049 1782481835.883059 fallocate 0x01f775 thru 0x023a3d (0x42c9 bytes) 000050 1782481835.890041 mapread 0x00486c thru 0x00d46d (0x8c02 bytes) 000051 1782481835.909298 write 0x0142e4 thru 0x0149d3 (0x6f0 bytes) 000052 1782481835.919023 mapread 0x01e15b thru 0x01fb4b (0x19f1 bytes) 000053 1782481835.928734 trunc from 0x00bc54 to 0x00c1d2 (0x57f bytes) 000054 1782481835.934650 mapread 0x01c596 thru 0x023c3b (0x76a6 bytes) 000055 1782481835.937289 trunc from 0x023c3c to 0x00f124 (0x14b17 bytes) 000056 1782481835.958270 punch from 0x0022ad to 0x009cde (0x7a32 bytes) 000057 1782481835.974323 MIRROR_EXTEND to 2 mirrors 000058 1782481836.523897 mapread 0x001f28 thru 0x00b10f (0x91e8 bytes) 000059 1782481836.565549 trunc from 0x009a27 to 0x00f124 (0x56fe bytes) 000060 1782481836.603390 fallocate 0x022f9c thru 0x024aa6 (0x1b0b bytes) 000061 1782481836.607563 fallocate 0x03d4c6 thru 0x03ffff (0x2b3a bytes) 000062 1782481836.613904 trunc from 0x03a7b7 to 0x03b923 (0x116d bytes) 000063 1782481836.618450 trunc from 0x02b6a3 to 0x032347 (0x6ca5 bytes) 000064 1782481836.622956 write 0x019976 thru 0x0280c8 (0xe753 bytes) 000065 1782481836.647686 trunc from 0x00a2c5 to 0x01465b (0xa397 bytes) 000067 1782481836.655927 mapwrite 0x03bc37 thru 0x03ffff (0x43c9 bytes) 000068 1782481836.691313 trunc from 0x040000 to 0x02dfca (0x12035 bytes) 000069 1782481836.706936 mapwrite 0x003274 thru 0x006385 (0x3112 bytes) 000070 1782481836.731377 MIRROR_RESYNC 2 mirrors 000071 1782481837.298756 mapwrite 0x01cc12 thru 0x0243cd (0x77bc bytes) 000072 1782481837.384999 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2dfcb): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dfcb): Mirror 2: 0x6dbd0b43 (stale) CRC-32 checksum value for chunk [0, 0x2dfcb): Mirror 1: 0x7bf77cb9 Mirror 3: 0x7bf77cb9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dfcb: skipped 000073 1782481838.206984 MIRROR_RESYNC 3 mirrors 000074 1782481838.959288 read 0x011a6d thru 0x0145d7 (0x2b6b bytes) 000075 1782481838.970264 mapwrite 0x03cf03 thru 0x03ffff (0x30fd bytes) 000076 1782481839.045017 fallocate 0x01928f thru 0x01f074 (0x5de6 bytes) 000077 1782481839.050946 punch from 0x01c43b to 0x01e0d4 (0x1c9a bytes) 000079 1782481839.056013 punch from 0x022d07 to 0x023a91 (0xd8b bytes) 000080 1782481839.061015 write 0x0048d8 thru 0x012fbb (0xe6e4 bytes) 000081 1782481839.071453 mapread 0x0117d6 thru 0x0215bc (0xfde7 bytes) 000084 1782481839.097975 trunc from 0x01208e to 0x01d7c9 (0xb73c bytes) 000085 1782481839.110088 trunc from 0x02e714 to 0x03c716 (0xe003 bytes) 000086 1782481839.119778 trunc from 0x040000 to 0x02ff56 (0x100a9 bytes) 000087 1782481839.151320 mapwrite 0x00c2ee thru 0x0194da (0xd1ed bytes) 000088 1782481839.197031 mapwrite 0x01abf3 thru 0x027d8b (0xd199 bytes) 000089 1782481839.257733 punch from 0x02124b to 0x02588d (0x4643 bytes) 000090 1782481839.273468 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2ff57): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ff57): Mirror 2: 0x341d91d (stale) Mirror 3: 0x341d91d (stale) CRC-32 checksum value for chunk [0, 0x2ff57): Mirror 1: 0x9452e486 Mirror 4: 0x9452e486 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ff57: skipped 000091 1782481839.952687 mapread 0x00675c thru 0x016097 (0xf93c bytes) 000093 1782481839.997694 punch from 0x011e98 to 0x018930 (0x6a99 bytes) 000094 1782481840.032870 mapread 0x014f51 thru 0x016a49 (0x1af9 bytes) 000095 1782481840.045330 fallocate 0x037a0f thru 0x03ffff (0x85f1 bytes) 000096 1782481840.055634 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2ff57): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ff57): Mirror 2: 0x341d91d (stale) Mirror 3: 0x341d91d (stale) Mirror 4: 0x9452e486 (stale) CRC-32 checksum value for chunk [0, 0x2ff57): Mirror 1: 0x5c9473ba Mirror 5: 0x5c9473ba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ff57: skipped 000097 1782481840.887263 read 0x01efd1 thru 0x0267e7 (0x7817 bytes) 000098 1782481840.910659 mapread 0x01342a thru 0x0170b6 (0x3c8d bytes) 000099 1782481840.961061 write 0x01be63 thru 0x0225b7 (0x6755 bytes) 000100 1782481841.019772 MIRROR_RESYNC 5 mirrors 000101 1782481842.446851 read 0x01c689 thru 0x022e5c (0x67d4 bytes) 000102 1782481842.455875 trunc from 0x00e2c5 to 0x00f677 (0x13b3 bytes) 000103 1782481842.489598 punch from 0x017e1e to 0x02482d (0xca10 bytes) 000104 1782481842.494814 mapread 0x015d18 thru 0x02079b (0xaa84 bytes) 000105 1782481842.518430 write 0x01b671 thru 0x01efb5 (0x3945 bytes) 000106 1782481842.519727 fallocate 0x0255b0 thru 0x03009f (0xaaf0 bytes) 000107 1782481842.524444 mapwrite 0x02c240 thru 0x036046 (0x9e07 bytes) 000108 1782481842.591717 MIRROR_RESYNC 5 mirrors 000110 1782481844.072947 trunc from 0x036047 to 0x0259bf (0x10687 bytes) 000111 1782481844.141759 trunc from 0x01785e to 0x01d0fb (0x589e bytes) 000112 1782481844.156674 write 0x02b956 thru 0x03aa43 (0xf0ee bytes) 000113 1782481844.162881 write 0x03c1a7 thru 0x03d571 (0x13cb bytes) 000115 1782481844.170552 trunc from 0x00cb14 to 0x0117b5 (0x4ca2 bytes) 000116 1782481844.185404 punch from 0x02d23c to 0x0335ff (0x63c4 bytes) 000117 1782481844.233550 read 0x0389fa thru 0x03d571 (0x4b78 bytes) 000118 1782481844.246295 trunc from 0x03d572 to 0x000267 (0x3d30a bytes) 000119 1782481844.289873 trunc from 0x000268 to 0x000b16 (0x8af bytes) 000120 1782481844.322316 write 0x01bc98 thru 0x01e384 (0x26ed bytes) 000121 1782481844.323690 write 0x026c61 thru 0x032ceb (0xc08b bytes) 000122 1782481844.337880 trunc from 0x032cec to 0x02ece7 (0x4004 bytes) 000123 1782481844.373799 trunc from 0x02ece8 to 0x034990 (0x5ca9 bytes) 000125 1782481844.405528 fallocate 0x023ef4 thru 0x0301c0 (0xc2cd bytes) 000126 1782481844.414442 mapwrite 0x031889 thru 0x032b07 (0x127f bytes) 000127 1782481844.478097 mapread 0x006ed0 thru 0x01644a (0xf57b bytes) 000128 1782481844.525200 trunc from 0x010f01 to 0x019972 (0x8a72 bytes) 000129 1782481844.546620 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34991): Mirror 2: 0x8bef486f (stale) Mirror 3: 0x8bef486f (stale) Mirror 5: 0x8bef486f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34991): Mirror 2: 0x8bef486f (stale) Mirror 3: 0x8bef486f (stale) Mirror 5: 0x8bef486f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34991: skipped 000130 1782481844.869973 write 0x00d527 thru 0x019a3e (0xc518 bytes) 000131 1782481844.888447 trunc from 0x02720e to 0x033c1e (0xca11 bytes) 000133 1782481844.895820 punch from 0x002965 to 0x00ac85 (0x8321 bytes) 000134 1782481844.906414 read 0x01839d thru 0x01e2f1 (0x5f55 bytes) 000136 1782481844.938323 MIRROR_RESYNC 4 mirrors 000138 1782481846.267291 trunc from 0x021581 to 0x030a23 (0xf4a3 bytes) 000139 1782481846.295989 read 0x02f985 thru 0x034990 (0x500c bytes) 000140 1782481846.304034 fallocate 0x02a8d9 thru 0x02bd09 (0x1431 bytes) 000141 1782481846.309924 write 0x0155ca thru 0x0229b0 (0xd3e7 bytes) 000142 1782481846.322199 read 0x019c33 thru 0x028669 (0xea37 bytes) 000143 1782481846.332878 trunc from 0x034991 to 0x01d6de (0x172b2 bytes) 000144 1782481846.352914 fallocate 0x00d11c thru 0x018c91 (0xbb76 bytes) 000145 1782481846.358376 trunc from 0x01d6df to 0x02e0a4 (0x109c6 bytes) 000146 1782481846.374833 trunc from 0x00665f to 0x0072e5 (0xc87 bytes) 000148 1782481846.381241 read 0x024083 thru 0x029967 (0x58e5 bytes) 000149 1782481846.390717 punch from 0x01ce13 to 0x027351 (0xa53f bytes) 000150 1782481846.410483 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e0a5): Mirror 2: 0x7cffad9a (stale) Mirror 3: 0x7cffad9a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e0a5): Mirror 2: 0x7cffad9a (stale) Mirror 3: 0x7cffad9a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e0a5: skipped 000151 1782481846.737648 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e0a5): Mirror 3: 0x7cffad9a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e0a5): Mirror 3: 0x7cffad9a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e0a5: skipped 000152 1782481847.038659 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4e:0x0]: Invalid argument (22) 000153 1782481847.177799 trunc from 0x00ff73 to 0x01974f (0x97dd bytes) 000154 1782481847.183482 read 0x008854 thru 0x0115cd (0x8d7a bytes) 000155 1782481847.194519 write 0x00d5ca thru 0x0113d6 (0x3e0d bytes) 000156 1782481847.199761 write 0x00def4 thru 0x01a403 (0xc510 bytes) 000157 1782481847.210810 trunc from 0x00b14c to 0x012eef (0x7da4 bytes) 000158 1782481847.230453 mapwrite 0x00bd97 thru 0x016c24 (0xae8e bytes) 000159 1782481847.296346 mapwrite 0x009148 thru 0x014675 (0xb52e bytes) 000160 1782481847.335371 fallocate 0x00d9af thru 0x016096 (0x86e8 bytes) 000161 1782481847.341198 punch from 0x01264f to 0x021715 (0xf0c7 bytes) 000162 1782481847.356326 fallocate 0x03bd09 thru 0x03ffff (0x42f7 bytes) 000163 1782481847.362090 fallocate 0x03cd84 thru 0x03ffff (0x327c bytes) 000164 1782481847.368417 mapwrite 0x037e57 thru 0x03ffff (0x81a9 bytes) 000166 1782481847.411829 fallocate 0x02267b thru 0x02a10e (0x7a94 bytes) 000167 1782481847.431005 trunc from 0x040000 to 0x00ded8 (0x32127 bytes) 000168 1782481847.458373 punch from 0x0049f7 to 0x00aeba (0x64c4 bytes) 000169 1782481847.467283 trunc from 0x00ded9 to 0x01d9b7 (0xfadf bytes) 000170 1782481847.490629 mapread 0x00dd7c thru 0x0157ae (0x7a33 bytes) 000171 1782481847.504342 mapread 0x0022b1 thru 0x002d4e (0xa9e bytes) 000172 1782481847.515695 trunc from 0x01d9b8 to 0x009b60 (0x13e57 bytes) 000174 1782481847.535724 trunc from 0x007a6a to 0x0081ef (0x786 bytes) 000175 1782481847.540698 mapread 0x002603 thru 0x008fbd (0x69bb bytes) 000177 1782481847.553861 write 0x0262c1 thru 0x032333 (0xc073 bytes) 000179 1782481847.567174 mapread 0x00aadc thru 0x0117e7 (0x6d0c bytes) 000180 1782481847.589737 trunc from 0x032334 to 0x0055c3 (0x2cd70 bytes) 000181 1782481847.615184 punch from 0x001a42 to 0x002f64 (0x1523 bytes) 000182 1782481847.626366 trunc from 0x001926 to 0x0023fd (0xad8 bytes) 000183 1782481847.634971 trunc from 0x002b91 to 0x0055c3 (0x2a33 bytes) 000184 1782481847.643591 trunc from 0x004c7a to 0x0055c3 (0x94a bytes) 000185 1782481847.652512 read 0x001778 thru 0x0055c3 (0x3e4c bytes) 000186 1782481847.666960 mapwrite 0x03a437 thru 0x03ffff (0x5bc9 bytes) 000187 1782481847.753440 read 0x027f85 thru 0x02c39a (0x4416 bytes) 000188 1782481847.754280 mapwrite 0x01f0a0 thru 0x0237c3 (0x4724 bytes) 000189 1782481847.781709 mapread 0x00aabb thru 0x01004b (0x5591 bytes) 000190 1782481847.783877 fallocate 0x01d885 thru 0x027023 (0x979f bytes) 000191 1782481847.797049 trunc from 0x01bad0 to 0x02b7cd (0xfcfe bytes) 000192 1782481847.805169 fallocate 0x00f05b thru 0x00feaa (0xe50 bytes) 000193 1782481847.813660 trunc from 0x00433c to 0x011463 (0xd128 bytes) 000194 1782481847.825098 mapwrite 0x0156b0 thru 0x01591e (0x26f bytes) 000196 1782481847.852731 fallocate 0x009f38 thru 0x00f2f3 (0x53bc bytes) 000197 1782481847.867974 read 0x01ad07 thru 0x01d773 (0x2a6d bytes) 000198 1782481847.887876 read 0x015d55 thru 0x01b0f7 (0x53a3 bytes) 000200 1782481847.890504 write 0x02d23b thru 0x03a2f3 (0xd0b9 bytes) 000201 1782481847.894773 trunc from 0x040000 to 0x009ef4 (0x3610b bytes) 000202 1782481847.914375 trunc from 0x009e93 to 0x009ef4 (0x62 bytes) 000205 1782481847.931082 punch from 0x007529 to 0x009ef4 (0x29cc bytes) 000206 1782481847.942750 punch from 0x0078e4 to 0x009ef4 (0x2611 bytes) 000207 1782481847.950654 trunc from 0x0001ef to 0x00114f (0xf61 bytes) 000209 1782481847.958926 write 0x00c1c4 thru 0x010b3e (0x497b bytes) 000210 1782481847.962968 read 0x00e57a thru 0x010b3e (0x25c5 bytes) 000212 1782481847.966860 fallocate 0x03e45a thru 0x03ffff (0x1ba6 bytes) 000213 1782481847.973368 trunc from 0x040000 to 0x03f668 (0x997 bytes) 000215 1782481847.995758 write 0x01c6d7 thru 0x022d6d (0x6697 bytes) 000216 1782481848.012743 write 0x03d7bf thru 0x03ffff (0x2841 bytes) 000217 1782481848.014825 write 0x0058ad thru 0x007808 (0x1f5c bytes) 000219 1782481848.017549 trunc from 0x017375 to 0x01c66a (0x52f6 bytes) 000221 1782481848.035556 mapread 0x0174c4 thru 0x025bb7 (0xe6f4 bytes) 000222 1782481848.068231 read 0x026020 thru 0x026d31 (0xd12 bytes) 000223 1782481848.068957 write 0x03b378 thru 0x03ffff (0x4c88 bytes) 000224 1782481848.070523 trunc from 0x015bf9 to 0x023d38 (0xe140 bytes) 000225 1782481848.075877 write 0x021fa9 thru 0x02e268 (0xc2c0 bytes) 000227 1782481848.089642 MIRROR_EXTEND to 2 mirrors 000228 1782481848.711590 trunc from 0x022252 to 0x02291f (0x6ce bytes) 000229 1782481848.776890 punch from 0x024557 to 0x0315c7 (0xd071 bytes) 000230 1782481848.785882 mapread 0x03b4a5 thru 0x03e296 (0x2df2 bytes) 000231 1782481848.809214 trunc from 0x038319 to 0x03ea85 (0x676d bytes) 000232 1782481848.816871 mapwrite 0x03ae6c thru 0x03ffff (0x5194 bytes) 000233 1782481848.846639 write 0x00abaa thru 0x00d511 (0x2968 bytes) 000234 1782481848.860257 trunc from 0x040000 to 0x03ba29 (0x45d6 bytes) 000235 1782481848.922784 fallocate 0x00aed6 thru 0x00d0ef (0x221a bytes) 000236 1782481848.932105 trunc from 0x00bd05 to 0x01a56c (0xe868 bytes) 000237 1782481848.943222 write 0x02461c thru 0x02c66b (0x8050 bytes) 000238 1782481848.962205 trunc from 0x022ef7 to 0x02deb2 (0xafbc bytes) 000240 1782481848.979736 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ba2a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ba2a): Mirror 2: 0xe54ec90 (stale) CRC-32 checksum value for chunk [0, 0x3ba2a): Mirror 1: 0xde00f6d3 Mirror 3: 0xde00f6d3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ba2a: skipped 000241 1782481849.929741 write 0x0176a0 thru 0x025df2 (0xe753 bytes) 000242 1782481849.991809 mapread 0x00217c thru 0x00ad99 (0x8c1e bytes) 000243 1782481850.024656 read 0x030c51 thru 0x039ffa (0x93aa bytes) 000244 1782481850.037190 punch from 0x01a68d to 0x01cc78 (0x25ec bytes) 000245 1782481850.057385 mapread 0x028093 thru 0x0359fa (0xd968 bytes) 000246 1782481850.078645 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ba2a): Mirror 3: 0xde00f6d3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ba2a): Mirror 3: 0xde00f6d3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ba2a: skipped 000247 1782481850.448630 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ba2a): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ba2a): Mirror 3: 0xde00f6d3 (stale) CRC-32 checksum value for chunk [0, 0x3ba2a): Mirror 1: 0x3bf46c6f Mirror 4: 0x3bf46c6f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ba2a: skipped 000248 1782481851.399050 punch from 0x0263f1 to 0x032fe9 (0xcbf9 bytes) 000249 1782481851.428498 write 0x025b61 thru 0x034d04 (0xf1a4 bytes) 000250 1782481851.446674 trunc from 0x03ba2a to 0x0142ad (0x2777c bytes) 000252 1782481851.484429 MIRROR_RESYNC 3 mirrors 000253 1782481852.305180 punch from 0x0129e9 to 0x0142ad (0x18c5 bytes) 000254 1782481852.334899 punch from 0x00f93c to 0x0142ad (0x4972 bytes) 000256 1782481852.339944 mapread 0x005450 thru 0x009ead (0x4a5e bytes) 000257 1782481852.356539 trunc from 0x0142ae to 0x0039f2 (0x108bb bytes) 000258 1782481852.372922 mapwrite 0x011d52 thru 0x0198be (0x7b6d bytes) 000259 1782481852.424253 punch from 0x003fe4 to 0x011caa (0xdcc7 bytes) 000260 1782481852.431774 fallocate 0x0299dd thru 0x02dfe9 (0x460d bytes) 000261 1782481852.437996 trunc from 0x00f6ed to 0x017215 (0x7b29 bytes) 000262 1782481852.442659 write 0x038863 thru 0x03ffff (0x779d bytes) 000264 1782481852.447972 MIRROR_RESYNC 3 mirrors 000265 1782481853.180723 trunc from 0x02ec50 to 0x03475f (0x5b10 bytes) 000266 1782481853.213062 trunc from 0x040000 to 0x02878b (0x17874 bytes) 000267 1782481853.259345 mapwrite 0x03618e thru 0x03fadf (0x9952 bytes) 000268 1782481853.336793 read 0x000f17 thru 0x001899 (0x983 bytes) 000269 1782481853.343536 trunc from 0x03fae0 to 0x038e8e (0x6c51 bytes) 000270 1782481853.360750 trunc from 0x004374 to 0x0063fe (0x208b bytes) 000271 1782481853.366522 mapread 0x01cd3a thru 0x020529 (0x37f0 bytes) 000272 1782481853.377211 trunc from 0x023da2 to 0x02765e (0x38bd bytes) 000273 1782481853.382486 mapwrite 0x013f4b thru 0x018a81 (0x4b37 bytes) 000274 1782481853.410713 punch from 0x01aab5 to 0x0260e7 (0xb633 bytes) 000276 1782481853.416414 read 0x01ca38 thru 0x0284ad (0xba76 bytes) 000277 1782481853.433205 mapread 0x027254 thru 0x02d25f (0x600c bytes) 000278 1782481853.434746 read 0x010dab thru 0x018d10 (0x7f66 bytes) 000279 1782481853.436061 write 0x00e24e thru 0x01c67b (0xe42e bytes) 000280 1782481853.439145 punch from 0x0164a7 to 0x026464 (0xffbe bytes) 000281 1782481853.455103 trunc from 0x038e8f to 0x0196e7 (0x1f7a7 bytes) 000282 1782481853.479795 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x196e8): Mirror 4: 0x8f47a5c7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x196e8): Mirror 4: 0x8f47a5c7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x196e8: skipped 000283 1782481853.711442 trunc from 0x00adfc to 0x00b207 (0x40c bytes) 000284 1782481853.717976 read 0x0179d6 thru 0x0196e7 (0x1d12 bytes) 000285 1782481853.734952 mapread 0x006a8c thru 0x0100d6 (0x964b bytes) 000286 1782481853.766579 trunc from 0x0196e8 to 0x03110c (0x17a25 bytes) 000287 1782481853.790924 trunc from 0x006f1c to 0x00a37c (0x3461 bytes) 000288 1782481853.796077 fallocate 0x0086bf thru 0x01288d (0xa1cf bytes) 000289 1782481853.801443 trunc from 0x0294fd to 0x030f5e (0x7a62 bytes) 000290 1782481853.805908 read 0x019d0d thru 0x022748 (0x8a3c bytes) 000291 1782481853.829041 trunc from 0x03110d to 0x015c73 (0x1b499 bytes) 000292 1782481853.845080 mapread 0x00e6be thru 0x015c73 (0x75b6 bytes) 000293 1782481853.846445 fallocate 0x00b8ff thru 0x00c813 (0xf15 bytes) 000294 1782481853.851748 trunc from 0x009611 to 0x00a1fc (0xbec bytes) 000295 1782481853.856832 fallocate 0x0010e1 thru 0x00edb8 (0xdcd8 bytes) 000296 1782481853.866853 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4e:0x0]: Invalid argument (22) 000297 1782481854.000980 mapwrite 0x02f9a3 thru 0x03eaac (0xf10a bytes) 000298 1782481854.118730 write 0x011617 thru 0x01178a (0x174 bytes) 000299 1782481854.123747 write 0x03460f thru 0x03ffff (0xb9f1 bytes) 000300 1782481854.131789 MIRROR_EXTEND to 2 mirrors 000301 1782481854.717014 mapread 0x0152af thru 0x0179fa (0x274c bytes) 000302 1782481854.751756 punch from 0x011cca to 0x014997 (0x2cce bytes) 000304 1782481854.784475 mapread 0x016f41 thru 0x01f239 (0x82f9 bytes) 000305 1782481854.798063 trunc from 0x040000 to 0x01da49 (0x225b6 bytes) 000306 1782481854.842625 mapwrite 0x035200 thru 0x03d6ba (0x84bb bytes) 000307 1782481854.877922 mapread 0x03bf21 thru 0x03d6ba (0x179a bytes) 000308 1782481854.880066 write 0x00ed6c thru 0x019df2 (0xb087 bytes) 000309 1782481854.890613 write 0x01f6e9 thru 0x021591 (0x1ea9 bytes) 000310 1782481854.901586 write 0x039d5b thru 0x03ffff (0x62a5 bytes) 000311 1782481854.902748 write 0x036606 thru 0x036ba2 (0x59d bytes) 000312 1782481854.903671 punch from 0x004830 to 0x00e27e (0x9a4f bytes) 000313 1782481854.925845 trunc from 0x026b25 to 0x0302ff (0x97db bytes) 000314 1782481854.931061 write 0x028160 thru 0x02d0e8 (0x4f89 bytes) 000317 1782481854.955757 punch from 0x02c559 to 0x03468c (0x8134 bytes) 000318 1782481854.975480 read 0x00928e thru 0x00f377 (0x60ea bytes) 000320 1782481855.002289 read 0x01dde6 thru 0x02808d (0xa2a8 bytes) 000321 1782481855.018346 read 0x00936f thru 0x0117ff (0x8491 bytes) 000322 1782481855.025015 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4e:0x0]: Invalid argument (22) 000323 1782481855.177182 mapread 0x02f5fc thru 0x03dc9f (0xe6a4 bytes) 000326 1782481855.217572 punch from 0x02a299 to 0x03237e (0x80e6 bytes) 000328 1782481855.226679 trunc from 0x040000 to 0x0350ad (0xaf52 bytes) 000329 1782481855.256603 mapwrite 0x030dd8 thru 0x03b31c (0xa545 bytes) 000330 1782481855.323113 punch from 0x00caec to 0x0146d6 (0x7beb bytes) 000331 1782481855.331220 punch from 0x0381cb to 0x03acdc (0x2b12 bytes) 000332 1782481855.336770 mapread 0x035f61 thru 0x03b31c (0x53bc bytes) 000333 1782481855.348463 MIRROR_EXTEND to 2 mirrors 000334 1782481856.015841 trunc from 0x031917 to 0x036148 (0x4832 bytes) 000335 1782481856.057095 read 0x023a02 thru 0x02d6b1 (0x9cb0 bytes) 000336 1782481856.070897 write 0x024c4a thru 0x031d4e (0xd105 bytes) 000338 1782481856.086992 write 0x010416 thru 0x0172a8 (0x6e93 bytes) 000339 1782481856.109565 trunc from 0x03b31d to 0x00fe93 (0x2b489 bytes) 000340 1782481856.150089 mapwrite 0x037821 thru 0x03ffff (0x87df bytes) 000341 1782481856.218978 mapwrite 0x01a373 thru 0x025543 (0xb1d1 bytes) 000342 1782481856.271170 write 0x010a8b thru 0x01e6b7 (0xdc2d bytes) 000344 1782481856.284996 read 0x022e68 thru 0x02735d (0x44f6 bytes) 000345 1782481856.285704 mapwrite 0x035134 thru 0x03ffff (0xaecc bytes) 000346 1782481856.312274 mapwrite 0x03ae8c thru 0x03ffff (0x5174 bytes) 000347 1782481856.351711 trunc from 0x03ad3b to 0x03ffff (0x52c5 bytes) 000348 1782481856.360772 mapwrite 0x015cf9 thru 0x01851c (0x2824 bytes) 000349 1782481856.378663 read 0x03e2f5 thru 0x03ffff (0x1d0b bytes) 000350 1782481856.389626 write 0x037aec thru 0x03b4b8 (0x39cd bytes) 000351 1782481856.391064 write 0x00eaf6 thru 0x01e3f7 (0xf902 bytes) 000353 1782481856.397115 read 0x002dd7 thru 0x00aaba (0x7ce4 bytes) 000354 1782481856.398026 mapread 0x03a999 thru 0x03b85f (0xec7 bytes) 000355 1782481856.404982 MIRROR_RESYNC 2 mirrors 000356 1782481857.094985 mapwrite 0x00a4a7 thru 0x018b8d (0xe6e7 bytes) 000358 1782481857.226628 read 0x009895 thru 0x00d32b (0x3a97 bytes) 000359 1782481857.231915 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4e:0x0]: Invalid argument (22) 000360 1782481857.365037 trunc from 0x040000 to 0x03818c (0x7e73 bytes) 000361 1782481857.402646 punch from 0x020030 to 0x020b7e (0xb4f bytes) 000362 1782481857.411762 trunc from 0x00e347 to 0x016c2e (0x88e8 bytes) 000364 1782481857.417075 fallocate 0x007e12 thru 0x009482 (0x1671 bytes) 000365 1782481857.424037 trunc from 0x03818d to 0x027f14 (0x10278 bytes) 000366 1782481857.448742 mapwrite 0x013de6 thru 0x01c91a (0x8b35 bytes) 000367 1782481857.505975 mapread 0x00c267 thru 0x014f77 (0x8d11 bytes) 000368 1782481857.520099 write 0x008790 thru 0x016c6d (0xe4de bytes) 000369 1782481857.521968 punch from 0x00495e to 0x012437 (0xdada bytes) 000370 1782481857.542795 read 0x01b2ba thru 0x01f650 (0x4397 bytes) 000371 1782481857.548019 write 0x032931 thru 0x03757d (0x4c4d bytes) 000373 1782481857.549825 write 0x00c30f thru 0x00f1d1 (0x2ec3 bytes) 000375 1782481857.559694 trunc from 0x03757e to 0x0339f1 (0x3b8c bytes) 000376 1782481857.572544 punch from 0x017c5a to 0x01fb73 (0x7f1a bytes) 000377 1782481857.577111 punch from 0x01458e to 0x01f43a (0xaead bytes) 000379 1782481857.583282 write 0x020dbf thru 0x029c86 (0x8ec8 bytes) 000381 1782481857.600548 read 0x0132a8 thru 0x021fa2 (0xecfb bytes) 000382 1782481857.615609 fallocate 0x008ffa thru 0x0100d8 (0x70df bytes) 000385 1782481857.621932 write 0x02617f thru 0x034f41 (0xedc3 bytes) 000387 1782481857.627294 trunc from 0x00cb94 to 0x00e7ad (0x1c1a bytes) 000388 1782481857.645416 MIRROR_EXTEND to 2 mirrors 000389 1782481858.343851 MIRROR_RESYNC 2 mirrors 000390 1782481858.661384 fallocate 0x03135b thru 0x03ae9b (0x9b41 bytes) 000391 1782481858.698183 read 0x013487 thru 0x018270 (0x4dea bytes) 000392 1782481858.710883 mapread 0x0186ea thru 0x01967c (0xf93 bytes) 000393 1782481858.718906 mapread 0x02fc2b thru 0x03ae9b (0xb271 bytes) 000394 1782481858.748096 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ae9c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ae9c): Mirror 2: 0x1217fe8c (stale) CRC-32 checksum value for chunk [0, 0x3ae9c): Mirror 1: 0x1217fe8c Mirror 3: 0x1217fe8c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ae9c: skipped 000395 1782481859.584980 mapread 0x0186f5 thru 0x01c796 (0x40a2 bytes) 000396 1782481859.607200 fallocate 0x029cb0 thru 0x0350de (0xb42f bytes) 000397 1782481859.639199 read 0x00e8e3 thru 0x013be9 (0x5307 bytes) 000400 1782481859.655189 fallocate 0x03126d thru 0x0372f5 (0x6089 bytes) 000401 1782481859.668307 mapwrite 0x012d13 thru 0x0205c8 (0xd8b6 bytes) 000402 1782481859.711157 fallocate 0x02922d thru 0x029368 (0x13c bytes) 000403 1782481859.720443 punch from 0x028dee to 0x02985a (0xa6d bytes) 000404 1782481859.726481 trunc from 0x03ae9c to 0x00cfc3 (0x2ded8 bytes) 000405 1782481859.752631 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xcfc4): Mirror 2: 0x68dd246d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xcfc4): Mirror 2: 0x68dd246d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xcfc4: skipped 000406 1782481860.004939 mapwrite 0x02d6c5 thru 0x03163c (0x3f78 bytes) 000407 1782481860.085930 punch from 0x02b0d3 to 0x03163c (0x656a bytes) 000408 1782481860.094873 MIRROR_RESYNC 2 mirrors 000409 1782481860.653001 trunc from 0x019f03 to 0x026ab0 (0xcbae bytes) 000410 1782481860.701803 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3163d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3163d): Mirror 2: 0x9ea0a0fd (stale) CRC-32 checksum value for chunk [0, 0x3163d): Mirror 1: 0x9ea0a0fd Mirror 3: 0x9ea0a0fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3163d: skipped 000411 1782481861.500889 trunc from 0x03163d to 0x03220e (0xbd2 bytes) 000412 1782481861.568998 trunc from 0x010b88 to 0x0172b3 (0x672c bytes) 000413 1782481861.576695 mapread 0x00306f thru 0x00d423 (0xa3b5 bytes) 000414 1782481861.601007 write 0x0164e4 thru 0x021b0e (0xb62b bytes) 000415 1782481861.611804 trunc from 0x03220f to 0x025ff1 (0xc21d bytes) 000416 1782481861.625963 trunc from 0x025ff2 to 0x03e047 (0x18056 bytes) 000417 1782481861.647319 mapwrite 0x023cb5 thru 0x030358 (0xc6a4 bytes) 000418 1782481861.713538 trunc from 0x03e048 to 0x026d2f (0x17318 bytes) 000419 1782481861.736624 mapwrite 0x00790f thru 0x00f41d (0x7b0f bytes) 000423 1782481861.758046 trunc from 0x026d30 to 0x0257b8 (0x1577 bytes) 000424 1782481861.771535 trunc from 0x0257b9 to 0x02531f (0x499 bytes) 000425 1782481861.783914 trunc from 0x023301 to 0x02531f (0x201f bytes) 000428 1782481861.791496 mapwrite 0x007553 thru 0x00bd23 (0x47d1 bytes) 000429 1782481861.811739 mapwrite 0x0198bb thru 0x028fe6 (0xf72c bytes) 000430 1782481861.866565 read 0x015270 thru 0x01a794 (0x5525 bytes) 000431 1782481861.873327 write 0x020936 thru 0x02b9d8 (0xb0a3 bytes) 000433 1782481861.876765 write 0x025cd2 thru 0x0312e0 (0xb60f bytes) 000434 1782481861.879521 read 0x006d12 thru 0x00c1b4 (0x54a3 bytes) 000435 1782481861.881093 mapread 0x01c0c1 thru 0x02b417 (0xf357 bytes) 000436 1782481861.883598 write 0x004c3a thru 0x0072a3 (0x266a bytes) 000438 1782481861.885777 mapwrite 0x032e60 thru 0x03c50a (0x96ab bytes) 000439 1782481861.936056 punch from 0x036802 to 0x03b1d5 (0x49d4 bytes) 000440 1782481861.941967 fallocate 0x007eca thru 0x00c6df (0x4816 bytes) 000441 1782481861.947009 write 0x0083a4 thru 0x012f51 (0xabae bytes) 000442 1782481861.949343 fallocate 0x0294db thru 0x02b75a (0x2280 bytes) 000443 1782481861.954889 trunc from 0x02a351 to 0x0305d4 (0x6284 bytes) 000444 1782481861.959634 read 0x0164f3 thru 0x0202a4 (0x9db2 bytes) 000445 1782481861.960922 punch from 0x00766d to 0x015a2f (0xe3c3 bytes) 000446 1782481861.975378 write 0x011980 thru 0x013204 (0x1885 bytes) 000447 1782481861.995554 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3c50b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c50b): Mirror 2: 0x99dce502 (stale) Mirror 3: 0x99dce502 (stale) CRC-32 checksum value for chunk [0, 0x3c50b): Mirror 1: 0x5e84505c Mirror 4: 0x5e84505c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c50b: skipped 000449 1782481863.040585 trunc from 0x03c50b to 0x015f9c (0x2656e bytes) 000450 1782481863.085248 read 0x007705 thru 0x015f9c (0xe898 bytes) 000451 1782481863.096896 MIRROR_RESYNC 4 mirrors 000453 1782481864.033507 write 0x036e7c thru 0x03957e (0x2703 bytes) 000454 1782481864.079496 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3957f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3957f): Mirror 2: 0xa2146fdc (stale) Mirror 3: 0xa2146fdc (stale) Mirror 4: 0xa2146fdc (stale) CRC-32 checksum value for chunk [0, 0x3957f): Mirror 1: 0xa3e8c96 Mirror 5: 0xa3e8c96 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3957f: skipped 000455 1782481864.956098 fallocate 0x03d19a thru 0x03ffff (0x2e66 bytes) 000456 1782481864.997320 read 0x009806 thru 0x0126e2 (0x8edd bytes) 000457 1782481865.011975 mapread 0x01aa08 thru 0x027347 (0xc940 bytes) 000458 1782481865.052619 fallocate 0x036212 thru 0x03ffff (0x9dee bytes) 000459 1782481865.058116 read 0x031022 thru 0x03957e (0x855d bytes) 000460 1782481865.075751 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3957f): Mirror 3: 0xa2146fdc (stale) Mirror 4: 0xa2146fdc (stale) Mirror 5: 0xa3e8c96 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3957f): Mirror 3: 0xa2146fdc (stale) Mirror 4: 0xa2146fdc (stale) Mirror 5: 0xa3e8c96 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3957f: skipped 000461 1782481865.440541 trunc from 0x03957f to 0x0253e4 (0x1419a bytes) 000462 1782481865.466713 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x253e5): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x253e5): Mirror 3: 0xd6857dae (stale) Mirror 4: 0xd6857dae (stale) Mirror 5: 0xd6857dae (stale) CRC-32 checksum value for chunk [0, 0x253e5): Mirror 1: 0xd6857dae Mirror 6: 0xd6857dae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x253e5: skipped 000463 1782481866.325563 read 0x016056 thru 0x02212c (0xc0d7 bytes) 000464 1782481866.355648 punch from 0x01b1b1 to 0x01d8c4 (0x2714 bytes) 000465 1782481866.398067 trunc from 0x0253e5 to 0x0369b6 (0x115d2 bytes) 000466 1782481866.450179 write 0x021c1d thru 0x02fd0e (0xe0f2 bytes) 000467 1782481866.462849 mapread 0x00d243 thru 0x01ca70 (0xf82e bytes) 000468 1782481866.492291 mapread 0x02cdc3 thru 0x0369b6 (0x9bf4 bytes) 000469 1782481866.504378 read 0x009555 thru 0x00daec (0x4598 bytes) 000470 1782481866.505534 fallocate 0x01a429 thru 0x0234c9 (0x90a1 bytes) 000471 1782481866.513825 read 0x02eec0 thru 0x0369b6 (0x7af7 bytes) 000472 1782481866.514783 mapread 0x0137a0 thru 0x017f3a (0x479b bytes) 000473 1782481866.516287 trunc from 0x032921 to 0x0369b6 (0x4096 bytes) 000474 1782481866.529272 MIRROR_RESYNC 5 mirrors 000477 1782481868.239656 trunc from 0x0153a9 to 0x0247d1 (0xf429 bytes) 000479 1782481868.291394 read 0x001afa thru 0x003d9b (0x22a2 bytes) 000480 1782481868.300530 punch from 0x032262 to 0x032c16 (0x9b5 bytes) 000482 1782481868.312717 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x369b7): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x369b7): Mirror 3: 0x37dc1c89 (stale) Mirror 4: 0x37dc1c89 (stale) Mirror 5: 0x37dc1c89 (stale) Mirror 6: 0x37dc1c89 (stale) CRC-32 checksum value for chunk [0, 0x369b7): Mirror 1: 0x5f2caea1 Mirror 7: 0x5f2caea1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x369b7: skipped 000483 1782481869.283337 fallocate 0x00d9c2 thru 0x01c028 (0xe667 bytes) 000484 1782481869.323588 trunc from 0x0369b7 to 0x029962 (0xd054 bytes) 000485 1782481869.345295 mapread 0x00a9cf thru 0x01a4a3 (0xfad5 bytes) 000486 1782481869.379927 trunc from 0x002f8f to 0x00680d (0x387f bytes) 000487 1782481869.389805 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29963): Mirror 4: 0x3084dc99 (stale) Mirror 5: 0x3084dc99 (stale) Mirror 6: 0x3084dc99 (stale) Mirror 7: 0x69083b1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29963): Mirror 4: 0x3084dc99 (stale) Mirror 5: 0x3084dc99 (stale) Mirror 6: 0x3084dc99 (stale) Mirror 7: 0x69083b1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29963: skipped 000488 1782481869.820054 write 0x0131d5 thru 0x01bdef (0x8c1b bytes) 000489 1782481869.835606 mapread 0x003ec7 thru 0x011ea8 (0xdfe2 bytes) 000490 1782481869.862808 trunc from 0x01b43c to 0x020d01 (0x58c6 bytes) 000491 1782481869.875918 trunc from 0x024c28 to 0x029962 (0x4d3b bytes) 000492 1782481869.880449 write 0x0318f7 thru 0x03ffff (0xe709 bytes) 000493 1782481869.883081 mapwrite 0x03e1cd thru 0x03e53a (0x36e bytes) 000494 1782481869.897228 mapwrite 0x015766 thru 0x0181ad (0x2a48 bytes) 000497 1782481869.911321 trunc from 0x008ea3 to 0x00a081 (0x11df bytes) 000498 1782481869.919093 fallocate 0x0034ea thru 0x003a64 (0x57b bytes) 000499 1782481869.924315 read 0x0239e0 thru 0x02e2e1 (0xa902 bytes) 000500 1782481869.936901 fallocate 0x025a2d thru 0x03107e (0xb652 bytes) 000501 1782481869.941540 write 0x02960a thru 0x034a30 (0xb427 bytes) 000502 1782481869.943011 read 0x010365 thru 0x01d668 (0xd304 bytes) 000503 1782481869.951657 read 0x0347db thru 0x036b93 (0x23b9 bytes) 000504 1782481869.952574 mapwrite 0x007d72 thru 0x013615 (0xb8a4 bytes) 000505 1782481869.974560 trunc from 0x040000 to 0x03d014 (0x2feb bytes) 000507 1782481869.997589 fallocate 0x02f46a thru 0x036727 (0x72be bytes) 000509 1782481870.006760 read 0x00e96e thru 0x01843f (0x9ad2 bytes) 000511 1782481870.009917 fallocate 0x01cf36 thru 0x028976 (0xba41 bytes) 000512 1782481870.023013 read 0x00a488 thru 0x01094c (0x64c5 bytes) 000513 1782481870.025754 mapwrite 0x000673 thru 0x0073a6 (0x6d34 bytes) 000514 1782481870.060388 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d015): Mirror 5: 0x751f76ec (stale) Mirror 6: 0x751f76ec (stale) Mirror 7: 0x4d699660 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d015): Mirror 5: 0x751f76ec (stale) Mirror 6: 0x751f76ec (stale) Mirror 7: 0x4d699660 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d015: skipped 000515 1782481870.443681 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3d015): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d015): Mirror 5: 0x751f76ec (stale) Mirror 6: 0x751f76ec (stale) Mirror 7: 0x4d699660 (stale) CRC-32 checksum value for chunk [0, 0x3d015): Mirror 1: 0x15e7c33b Mirror 8: 0x15e7c33b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d015: skipped 000516 1782481871.369751 mapread 0x000073 thru 0x00fd0a (0xfc98 bytes) 000517 1782481871.420700 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3d015): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d015): Mirror 5: 0x751f76ec (stale) Mirror 6: 0x751f76ec (stale) Mirror 7: 0x4d699660 (stale) CRC-32 checksum value for chunk [0, 0x3d015): Mirror 1: 0x15e7c33b Mirror 8: 0x15e7c33b Mirror 9: 0x15e7c33b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d015: skipped 000518 1782481872.175298 write 0x01ea48 thru 0x02de11 (0xf3ca bytes) 000519 1782481872.236590 MIRROR_RESYNC 6 mirrors 000520 1782481873.318965 mapread 0x0339b8 thru 0x038c47 (0x5290 bytes) 000521 1782481873.342814 mapread 0x01f264 thru 0x02c71e (0xd4bb bytes) 000522 1782481873.363862 mapread 0x00471a thru 0x004d9a (0x681 bytes) 000523 1782481873.373959 read 0x017a29 thru 0x018767 (0xd3f bytes) 000524 1782481873.380575 fallocate 0x02f5e4 thru 0x038afc (0x9519 bytes) 000525 1782481873.424411 fallocate 0x0146fc thru 0x022b0a (0xe40f bytes) 000526 1782481873.433485 mapread 0x01d7c6 thru 0x025a9d (0x82d8 bytes) 000527 1782481873.450814 punch from 0x00099c to 0x010520 (0xfb85 bytes) 000528 1782481873.456319 punch from 0x0104c3 to 0x01abc7 (0xa705 bytes) 000529 1782481873.461287 punch from 0x00e787 to 0x017db8 (0x9632 bytes) 000530 1782481873.465316 mapread 0x0209ef thru 0x029a77 (0x9089 bytes) 000532 1782481873.474906 fallocate 0x0304b8 thru 0x03ffa3 (0xfaec bytes) 000533 1782481873.479861 trunc from 0x03d015 to 0x020b69 (0x1c4ab bytes) 000534 1782481873.503557 read 0x016103 thru 0x017498 (0x1396 bytes) 000535 1782481873.517357 trunc from 0x020b6a to 0x010da9 (0xfdc0 bytes) 000536 1782481873.534053 mapwrite 0x01e0e2 thru 0x02a707 (0xc626 bytes) 000538 1782481873.613817 mapwrite 0x023ec2 thru 0x029f0a (0x6049 bytes) 000539 1782481873.640741 read 0x00136e thru 0x0041fc (0x2e8f bytes) 000540 1782481873.642721 fallocate 0x01f3ff thru 0x02741b (0x801d bytes) 000542 1782481873.650054 trunc from 0x0047be to 0x00af31 (0x6774 bytes) 000543 1782481873.654639 trunc from 0x02a708 to 0x025aae (0x4c59 bytes) 000544 1782481873.668804 fallocate 0x03dd11 thru 0x03ffff (0x22ef bytes) 000545 1782481873.675952 mapread 0x0169db thru 0x02033d (0x9963 bytes) 000546 1782481873.684320 mapread 0x0218cf thru 0x025aae (0x41e0 bytes) 000547 1782481873.685352 trunc from 0x025aaf to 0x0311d9 (0xb72b bytes) 000548 1782481873.698519 fallocate 0x03673e thru 0x03ffff (0x98c2 bytes) 000549 1782481873.703697 fallocate 0x020f3e thru 0x029fe4 (0x90a7 bytes) 000550 1782481873.708420 punch from 0x004e76 to 0x008190 (0x331b bytes) 000551 1782481873.713201 fallocate 0x02a0b4 thru 0x037c34 (0xdb81 bytes) 000552 1782481873.717568 mapwrite 0x03c4dc thru 0x03ffff (0x3b24 bytes) 000553 1782481873.758809 mapwrite 0x0040a4 thru 0x00d6be (0x961b bytes) 000554 1782481873.844842 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x350d994 (stale) Mirror 7: 0x350d994 (stale) Mirror 8: 0x350d994 (stale) Mirror 9: 0x350d994 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x350d994 (stale) Mirror 7: 0x350d994 (stale) Mirror 8: 0x350d994 (stale) Mirror 9: 0x350d994 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000555 1782481874.372091 mapwrite 0x01c529 thru 0x0250f9 (0x8bd1 bytes) 000556 1782481874.429842 punch from 0x020e4e to 0x02d9a7 (0xcb5a bytes) 000557 1782481874.434721 fallocate 0x00345c thru 0x00b7ea (0x838f bytes) 000558 1782481874.442485 write 0x01b0ec thru 0x027d68 (0xcc7d bytes) 000559 1782481874.462987 mapread 0x0085bd thru 0x014b3c (0xc580 bytes) 000560 1782481874.501390 mapread 0x01fce5 thru 0x026cdb (0x6ff7 bytes) 000561 1782481874.502820 trunc from 0x040000 to 0x03c555 (0x3aaa bytes) 000562 1782481874.535413 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c556): Mirror 7: 0xf099d833 (stale) Mirror 8: 0xf099d833 (stale) Mirror 9: 0xf099d833 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c556): Mirror 7: 0xf099d833 (stale) Mirror 8: 0xf099d833 (stale) Mirror 9: 0xf099d833 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c556: skipped 000563 1782481874.915987 read 0x0019a1 thru 0x009768 (0x7dc8 bytes) 000564 1782481874.934627 trunc from 0x03c556 to 0x00010b (0x3c44a bytes) 000565 1782481874.959249 write 0x023d5c thru 0x02eac6 (0xad6b bytes) 000566 1782481874.960908 trunc from 0x01ba9d to 0x020e8c (0x53f0 bytes) 000568 1782481874.974480 mapwrite 0x021843 thru 0x0262ef (0x4aad bytes) 000569 1782481875.013231 trunc from 0x02eac7 to 0x00e989 (0x2013d bytes) 000570 1782481875.051516 mapwrite 0x0201c1 thru 0x02a437 (0xa277 bytes) 000571 1782481875.142990 trunc from 0x01776f to 0x024845 (0xd0d7 bytes) 000572 1782481875.154360 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a438): Mirror 7: 0x51ff9803 (stale) Mirror 8: 0x51ff9803 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a438): Mirror 7: 0x51ff9803 (stale) Mirror 8: 0x51ff9803 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a438: skipped 000573 1782481875.488052 punch from 0x00cc00 to 0x011abc (0x4ebd bytes) 000574 1782481875.494457 trunc from 0x01e2cc to 0x0241c6 (0x5efb bytes) 000575 1782481875.522942 write 0x038634 thru 0x03ffff (0x79cc bytes) 000576 1782481875.536662 mapread 0x01f482 thru 0x02aa8c (0xb60b bytes) 000577 1782481875.569569 trunc from 0x01b178 to 0x026846 (0xb6cf bytes) 000578 1782481875.577718 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x350d994 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x350d994 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000579 1782481875.892374 trunc from 0x040000 to 0x013e51 (0x2c1ae bytes) 000581 1782481875.913454 write 0x03cf56 thru 0x03ffff (0x30aa bytes) 000582 1782481875.914665 read 0x03b3dc thru 0x03ffff (0x4c24 bytes) 000583 1782481875.930049 read 0x00228b thru 0x003453 (0x11c9 bytes) 000584 1782481875.944098 trunc from 0x040000 to 0x007dcb (0x38234 bytes) 000585 1782481875.977488 read 0x0013a6 thru 0x0051b0 (0x3e0b bytes) 000586 1782481875.978486 fallocate 0x033190 thru 0x0381e5 (0x5056 bytes) 000587 1782481875.985583 trunc from 0x007dcc to 0x03bf6d (0x341a2 bytes) 000589 1782481876.003685 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3bf6e): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bf6e): Mirror 8: 0x6f1bd101 (stale) CRC-32 checksum value for chunk [0, 0x3bf6e): Mirror 1: 0x7e1a4dea Mirror 9: 0x7e1a4dea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bf6e: skipped 000590 1782481876.779095 trunc from 0x03bf6e to 0x018956 (0x23617 bytes) 000591 1782481876.854986 write 0x009ec6 thru 0x00b108 (0x1243 bytes) 000592 1782481876.866551 read 0x00fcd8 thru 0x015a98 (0x5dc1 bytes) 000593 1782481876.882655 trunc from 0x018957 to 0x020d99 (0x8443 bytes) 000594 1782481876.911935 mapread 0x00294d thru 0x00fa86 (0xd13a bytes) 000596 1782481876.947293 punch from 0x0074cc to 0x00e5e9 (0x711e bytes) 000597 1782481876.958012 mapread 0x013a3b thru 0x020d99 (0xd35f bytes) 000598 1782481876.973675 read 0x00292c thru 0x0036b2 (0xd87 bytes) 000599 1782481876.974467 punch from 0x00883d to 0x0149b2 (0xc176 bytes) 000600 1782481876.980106 trunc from 0x020d9a to 0x003dd0 (0x1cfc9 bytes) 000601 1782481876.999182 mapwrite 0x02d10b thru 0x039a70 (0xc966 bytes) 000602 1782481877.061067 mapwrite 0x029fec thru 0x02d020 (0x3035 bytes) 000603 1782481877.084334 trunc from 0x039a71 to 0x03f0cd (0x565d bytes) 000604 1782481877.119562 trunc from 0x03f0ce to 0x02505d (0x1a070 bytes) 000605 1782481877.152186 fallocate 0x0027aa thru 0x00b755 (0x8fac bytes) 000606 1782481877.164614 trunc from 0x02505e to 0x03bcca (0x16c6d bytes) 000607 1782481877.193570 write 0x00352d thru 0x00fe77 (0xc94b bytes) 000608 1782481877.195083 punch from 0x004d5b to 0x0145c9 (0xf86f bytes) 000610 1782481877.213957 read 0x0238ad thru 0x02fe5e (0xc5b2 bytes) 000611 1782481877.229075 fallocate 0x00e3f5 thru 0x01211a (0x3d26 bytes) 000613 1782481877.235719 write 0x03a954 thru 0x03e569 (0x3c16 bytes) 000614 1782481877.237763 fallocate 0x019f2a thru 0x023fcf (0xa0a6 bytes) 000615 1782481877.243286 write 0x02f714 thru 0x0350e8 (0x59d5 bytes) 000616 1782481877.245794 fallocate 0x016dcb thru 0x01c457 (0x568d bytes) 000618 1782481877.251102 punch from 0x02386b to 0x028814 (0x4faa bytes) 000619 1782481877.256100 read 0x025c07 thru 0x035753 (0xfb4d bytes) 000621 1782481877.265339 trunc from 0x03e56a to 0x03c817 (0x1d52 bytes) 000622 1782481877.277682 fallocate 0x01f6df thru 0x02cdcc (0xd6ee bytes) 000623 1782481877.285397 fallocate 0x01e09a thru 0x02be40 (0xdda7 bytes) 000624 1782481877.294275 mapwrite 0x022499 thru 0x02eb02 (0xc66a bytes) 000625 1782481877.319970 trunc from 0x02da4b to 0x039ccc (0xc282 bytes) 000626 1782481877.327714 mapread 0x003601 thru 0x0117e3 (0xe1e3 bytes) 000628 1782481877.332077 trunc from 0x03c818 to 0x021658 (0x1b1bf bytes) 000630 1782481877.355619 punch from 0x0155cd to 0x0156d9 (0x10d bytes) 000631 1782481877.361720 mapwrite 0x00ee7c thru 0x01d9c9 (0xeb4e bytes) 000632 1782481877.389333 trunc from 0x00b4d0 to 0x016b78 (0xb6a9 bytes) 000633 1782481877.402297 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x21659): Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21659): Mirror 8: 0x86c8c7f3 (stale) Mirror 9: 0x20dcd55a (stale) CRC-32 checksum value for chunk [0, 0x21659): Mirror 1: 0x254d56e3 Mirror 10: 0x254d56e3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21659: skipped 000634 1782481878.212994 punch from 0x0101e1 to 0x013378 (0x3198 bytes) 000635 1782481878.248927 mapwrite 0x01707f thru 0x021b0d (0xaa8f bytes) 000636 1782481878.374898 trunc from 0x021b0e to 0x02ed22 (0xd215 bytes) 000637 1782481878.394017 trunc from 0x02ed23 to 0x029811 (0x5511 bytes) 000638 1782481878.414451 punch from 0x013a25 to 0x0188af (0x4e8b bytes) 000639 1782481878.423755 punch from 0x00a31b to 0x00ae10 (0xaf6 bytes) 000640 1782481878.430009 punch from 0x01f50f to 0x029272 (0x9d64 bytes) 000642 1782481878.443843 mapwrite 0x00b104 thru 0x0179d3 (0xc8d0 bytes) 000644 1782481878.477990 mapwrite 0x015c46 thru 0x019bdf (0x3f9a bytes) 000645 1782481878.492318 mapread 0x0206d2 thru 0x029811 (0x9140 bytes) 000646 1782481878.507323 trunc from 0x01bed1 to 0x0216e8 (0x5818 bytes) 000647 1782481878.513339 mapwrite 0x038a18 thru 0x03ffff (0x75e8 bytes) 000648 1782481878.571669 trunc from 0x032171 to 0x03ffff (0xde8f bytes) 000649 1782481878.580057 write 0x00b41d thru 0x0170d6 (0xbcba bytes) 000651 1782481878.582919 read 0x003b67 thru 0x00e0f3 (0xa58d bytes) 000652 1782481878.589221 punch from 0x00b374 to 0x00b493 (0x120 bytes) 000653 1782481878.607583 write 0x0203b2 thru 0x0229de (0x262d bytes) 000654 1782481878.616298 mapread 0x02f51e thru 0x031718 (0x21fb bytes) 000655 1782481878.644552 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x350d994 (stale) Mirror 10: 0x4c6da53d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x350d994 (stale) Mirror 10: 0x4c6da53d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000656 1782481878.915299 trunc from 0x0362cf to 0x03ac7f (0x49b1 bytes) 000657 1782481878.924561 mapwrite 0x004283 thru 0x00972d (0x54ab bytes) 000658 1782481878.984034 mapwrite 0x02f67b thru 0x039b1c (0xa4a2 bytes) 000660 1782481879.028420 mapwrite 0x009dc7 thru 0x0108fb (0x6b35 bytes) 000661 1782481879.047030 punch from 0x006453 to 0x015015 (0xebc3 bytes) 000663 1782481879.053016 trunc from 0x040000 to 0x025d00 (0x1a2ff bytes) 000664 1782481879.073733 trunc from 0x02281d to 0x0254d4 (0x2cb8 bytes) 000665 1782481879.079613 punch from 0x00171a to 0x00e5a5 (0xce8c bytes) 000666 1782481879.084948 trunc from 0x023895 to 0x025d00 (0x246c bytes) 000667 1782481879.092562 fallocate 0x03e42b thru 0x03f289 (0xe5f bytes) 000669 1782481879.102007 mapread 0x03e539 thru 0x03f289 (0xd51 bytes) 000670 1782481879.112758 mapwrite 0x0023b1 thru 0x008486 (0x60d6 bytes) 000671 1782481879.143770 trunc from 0x0158ce to 0x01dae6 (0x8219 bytes) 000672 1782481879.150409 read 0x02741f thru 0x02776a (0x34c bytes) 000673 1782481879.157892 trunc from 0x03f28a to 0x0141cb (0x2b0be bytes) 000674 1782481879.175082 mapread 0x003afc thru 0x006e6b (0x3370 bytes) 000675 1782481879.177091 mapwrite 0x024a6b thru 0x0349cb (0xff61 bytes) 000676 1782481879.263707 punch from 0x00653f to 0x010feb (0xaaad bytes) 000677 1782481879.268485 trunc from 0x02ef71 to 0x0349cb (0x5a5b bytes) 000678 1782481879.273627 trunc from 0x0349cc to 0x009d28 (0x2aca3 bytes) 000680 1782481879.290074 read 0x000b01 thru 0x0036e3 (0x2be3 bytes) 000681 1782481879.297787 punch from 0x0099c9 to 0x009d28 (0x360 bytes) 000682 1782481879.302153 trunc from 0x00480a to 0x00632f (0x1b26 bytes) 000683 1782481879.307030 punch from 0x0001d8 to 0x004238 (0x4061 bytes) 000686 1782481879.313683 mapread 0x003b41 thru 0x009d28 (0x61e8 bytes) 000687 1782481879.330849 mapwrite 0x013d65 thru 0x01a908 (0x6ba4 bytes) 000689 1782481879.378335 trunc from 0x01a909 to 0x012d32 (0x7bd6 bytes) 000690 1782481879.392712 trunc from 0x009555 to 0x00ed09 (0x57b5 bytes) 000691 1782481879.398443 punch from 0x0088c0 to 0x012973 (0xa0b4 bytes) 000692 1782481879.404796 mapread 0x011f47 thru 0x012d32 (0xdec bytes) 000693 1782481879.417814 mapread 0x00dd5a thru 0x012d32 (0x4fd9 bytes) 000694 1782481879.440067 trunc from 0x00bb83 to 0x012d32 (0x71b0 bytes) 000695 1782481879.444627 fallocate 0x00c893 thru 0x0147c5 (0x7f33 bytes) 000697 1782481879.449458 read 0x007854 thru 0x012d32 (0xb4df bytes) 000698 1782481879.457298 trunc from 0x00f8b7 to 0x012d32 (0x347c bytes) 000699 1782481879.461249 mapread 0x0129e7 thru 0x012d32 (0x34c bytes) 000700 1782481879.467524 trunc from 0x00b06e to 0x012d32 (0x7cc5 bytes) 000701 1782481879.471884 trunc from 0x006ac3 to 0x0120e7 (0xb625 bytes) 000702 1782481879.477551 mapwrite 0x01c5df thru 0x01d5f3 (0x1015 bytes) 000703 1782481879.529988 mapread 0x017505 thru 0x017a60 (0x55c bytes) 000704 1782481879.532854 mapwrite 0x01a7b8 thru 0x01a82e (0x77 bytes) 000706 1782481879.549159 mapread 0x00d77e thru 0x0177cb (0xa04e bytes) 000707 1782481879.587029 mapread 0x019fde thru 0x01d5f3 (0x3616 bytes) 000708 1782481879.591148 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d5f4): Mirror 8: 0x3a5fed8f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d5f4): Mirror 8: 0x3a5fed8f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d5f4: skipped 000709 1782481879.827803 mapwrite 0x01c8a2 thru 0x028331 (0xba90 bytes) 000710 1782481879.911752 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x28332): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28332): Mirror 8: 0xb5c0c22d (stale) CRC-32 checksum value for chunk [0, 0x28332): Mirror 1: 0x4875fa61 Mirror 9: 0x4875fa61 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28332: skipped 000711 1782481880.674559 mapread 0x014244 thru 0x017ae1 (0x389e bytes) 000712 1782481880.702501 write 0x0045c6 thru 0x00feec (0xb927 bytes) 000713 1782481880.754878 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28332): Mirror 9: 0x4875fa61 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28332): Mirror 9: 0x4875fa61 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28332: skipped 000714 1782481881.004632 mapwrite 0x03f52f thru 0x03ffff (0xad1 bytes) 000715 1782481881.040661 punch from 0x032240 to 0x03c1d5 (0x9f96 bytes) 000716 1782481881.044841 trunc from 0x004790 to 0x011bce (0xd43f bytes) 000717 1782481881.053265 punch from 0x036788 to 0x037356 (0xbcf bytes) 000718 1782481881.058344 mapread 0x01755b thru 0x021937 (0xa3dd bytes) 000720 1782481881.086798 mapread 0x0241e4 thru 0x0319d6 (0xd7f3 bytes) 000721 1782481881.097297 write 0x001ce9 thru 0x00b96a (0x9c82 bytes) 000722 1782481881.107757 mapwrite 0x033f5e thru 0x03ee5c (0xaeff bytes) 000724 1782481881.149229 punch from 0x020c4e to 0x026891 (0x5c44 bytes) 000725 1782481881.155836 trunc from 0x023ca0 to 0x025a32 (0x1d93 bytes) 000726 1782481881.165104 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x81b1a8f9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4018cd8b Mirror 10: 0x4018cd8b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000728 1782481881.854995 trunc from 0x040000 to 0x00b9bc (0x34643 bytes) 000729 1782481881.937207 punch from 0x00b5a9 to 0x00b9bc (0x414 bytes) 000730 1782481881.942366 read 0x0087f2 thru 0x00a665 (0x1e74 bytes) 000731 1782481881.947071 write 0x03ffb7 thru 0x03ffff (0x49 bytes) 000732 1782481881.949391 trunc from 0x040000 to 0x00db1d (0x324e2 bytes) 000733 1782481881.965382 read 0x00bac8 thru 0x00db1d (0x2056 bytes) 000735 1782481881.976849 trunc from 0x00db1e to 0x02b7d9 (0x1dcbc bytes) 000737 1782481881.990606 trunc from 0x02b7da to 0x008d95 (0x22a44 bytes) 000738 1782481882.006679 mapread 0x005e5b thru 0x008d95 (0x2f3b bytes) 000739 1782481882.010963 mapread 0x00095e thru 0x001d06 (0x13a9 bytes) 000740 1782481882.016223 mapwrite 0x019742 thru 0x01a5d7 (0xe96 bytes) 000741 1782481882.056107 read 0x00938e thru 0x014b8e (0xb801 bytes) 000742 1782481882.070002 trunc from 0x01a5d8 to 0x01b245 (0xc6e bytes) 000743 1782481882.090823 trunc from 0x01b246 to 0x0056cf (0x15b76 bytes) 000744 1782481882.116984 fallocate 0x01c228 thru 0x01fd60 (0x3b39 bytes) 000745 1782481882.129418 MIRROR_RESYNC 3 mirrors 000746 1782481882.709933 trunc from 0x01fd61 to 0x036bf2 (0x16e92 bytes) 000747 1782481882.775493 trunc from 0x036bf3 to 0x01e2e6 (0x1890c bytes) 000748 1782481882.803473 trunc from 0x01e2e7 to 0x013dcc (0xa51a bytes) 000749 1782481882.822305 read 0x002579 thru 0x004206 (0x1c8e bytes) 000750 1782481882.835023 MIRROR_RESYNC 3 mirrors 000751 1782481883.415056 mapwrite 0x00d936 thru 0x015070 (0x773b bytes) 000752 1782481883.503912 read 0x000403 thru 0x00bada (0xb6d8 bytes) 000753 1782481883.521211 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15071): Mirror 9: 0xf5c8bd40 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15071): Mirror 9: 0xf5c8bd40 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15071: skipped 000754 1782481883.786962 fallocate 0x00b8cf thru 0x01697c (0xb0ae bytes) 000755 1782481883.800178 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4e:0x0]: Invalid argument (22) 000758 1782481883.948840 trunc from 0x0021a5 to 0x00baf7 (0x9953 bytes) 000759 1782481883.958673 mapread 0x014e87 thru 0x015070 (0x1ea bytes) 000761 1782481883.979951 read 0x008fdd thru 0x00f766 (0x678a bytes) 000762 1782481883.992436 trunc from 0x015071 to 0x0118de (0x3792 bytes) 000763 1782481884.014557 write 0x028c9a thru 0x0364cb (0xd832 bytes) 000764 1782481884.019749 read 0x00ec16 thru 0x011ea4 (0x328f bytes) 000765 1782481884.030187 mapread 0x00ac4d thru 0x019cc7 (0xf07b bytes) 000766 1782481884.031905 trunc from 0x0364cc to 0x022216 (0x142b5 bytes) 000767 1782481884.063254 trunc from 0x0090b4 to 0x00d68f (0x45dc bytes) 000768 1782481884.072583 trunc from 0x022217 to 0x018112 (0xa104 bytes) 000769 1782481884.094801 punch from 0x00ee30 to 0x01468a (0x585b bytes) 000770 1782481884.105757 mapwrite 0x0313e7 thru 0x033c9e (0x28b8 bytes) 000771 1782481884.169738 mapwrite 0x03e48f thru 0x03ffff (0x1b71 bytes) 000772 1782481884.227715 mapwrite 0x03d0b2 thru 0x03ffff (0x2f4e bytes) 000773 1782481884.258024 read 0x035279 thru 0x03ffff (0xad87 bytes) 000774 1782481884.275466 fallocate 0x01d9f4 thru 0x028691 (0xac9e bytes) 000775 1782481884.286374 trunc from 0x033ed1 to 0x0363b7 (0x24e7 bytes) 000776 1782481884.299226 trunc from 0x040000 to 0x02d55b (0x12aa4 bytes) 000777 1782481884.335849 trunc from 0x01b30d to 0x02a225 (0xef19 bytes) 000780 1782481884.345990 punch from 0x016c04 to 0x025a94 (0xee91 bytes) 000781 1782481884.363981 MIRROR_EXTEND to 2 mirrors 000782 1782481884.942251 read 0x00857a thru 0x01836a (0xfdf1 bytes) 000783 1782481884.959856 read 0x00e2c0 thru 0x01b2aa (0xcfeb bytes) 000784 1782481884.979856 write 0x0049f9 thru 0x011a40 (0xd048 bytes) 000785 1782481885.038805 punch from 0x01717a to 0x025d0b (0xeb92 bytes) 000786 1782481885.049242 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2d55c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d55c): Mirror 2: 0x90292e3e (stale) CRC-32 checksum value for chunk [0, 0x2d55c): Mirror 1: 0x8027ec14 Mirror 3: 0x8027ec14 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d55c: skipped 000787 1782481885.854000 trunc from 0x002ae7 to 0x00a2d1 (0x77eb bytes) 000788 1782481885.901183 mapread 0x011a32 thru 0x01aa95 (0x9064 bytes) 000789 1782481885.922721 read 0x02cbbc thru 0x02d55b (0x9a0 bytes) 000790 1782481885.931879 MIRROR_RESYNC 3 mirrors 000791 1782481886.622260 trunc from 0x02d55c to 0x01241a (0x1b141 bytes) 000792 1782481886.693046 trunc from 0x00e447 to 0x01241a (0x3fd4 bytes) 000793 1782481886.699604 read 0x006959 thru 0x01241a (0xbac2 bytes) 000794 1782481886.724018 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1241b): Mirror 2: 0x23f9a6d3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1241b): Mirror 2: 0x23f9a6d3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1241b: skipped 000795 1782481886.990292 fallocate 0x028186 thru 0x02df01 (0x5d7c bytes) 000796 1782481886.997702 write 0x01759b thru 0x0229e7 (0xb44d bytes) 000798 1782481887.006383 read 0x00b8be thru 0x0103e8 (0x4b2b bytes) 000799 1782481887.012485 mapread 0x01e60f thru 0x0229e7 (0x43d9 bytes) 000801 1782481887.014518 mapread 0x0020fc thru 0x00af8b (0x8e90 bytes) 000802 1782481887.041209 trunc from 0x01c9f8 to 0x0229e7 (0x5ff0 bytes) 000803 1782481887.064019 trunc from 0x0229e8 to 0x00237a (0x2066d bytes) 000804 1782481887.084467 read 0x0020bf thru 0x00237a (0x2bc bytes) 000805 1782481887.086498 mapwrite 0x02742f thru 0x02a471 (0x3043 bytes) 000807 1782481887.134723 read 0x01dc72 thru 0x024a7e (0x6e0d bytes) 000808 1782481887.135648 write 0x02b4c6 thru 0x0351e3 (0x9d1e bytes) 000810 1782481887.144785 MIRROR_RESYNC 2 mirrors 000811 1782481887.639349 read 0x01acb3 thru 0x0223b1 (0x76ff bytes) 000813 1782481887.648197 mapread 0x004aa4 thru 0x0143db (0xf938 bytes) 000815 1782481887.677649 trunc from 0x034efc to 0x0351e3 (0x2e8 bytes) 000816 1782481887.720627 trunc from 0x033b00 to 0x0351e3 (0x16e4 bytes) 000817 1782481887.727919 read 0x001c28 thru 0x008e42 (0x721b bytes) 000818 1782481887.740281 mapread 0x023746 thru 0x02ea3b (0xb2f6 bytes) 000819 1782481887.742446 trunc from 0x0351e4 to 0x02f531 (0x5cb2 bytes) 000820 1782481887.786004 read 0x010baa thru 0x01d2ca (0xc721 bytes) 000821 1782481887.790103 write 0x01ec8e thru 0x020641 (0x19b4 bytes) 000822 1782481887.792750 write 0x018ddd thru 0x01c80c (0x3a30 bytes) 000823 1782481887.795092 trunc from 0x00267c to 0x009570 (0x6ef5 bytes) 000824 1782481887.805085 write 0x020ccb thru 0x023b30 (0x2e66 bytes) 000825 1782481887.808928 write 0x035905 thru 0x03ed87 (0x9483 bytes) 000827 1782481887.814605 mapread 0x035c81 thru 0x03ed87 (0x9107 bytes) 000828 1782481887.817217 mapwrite 0x00d11c thru 0x014269 (0x714e bytes) 000829 1782481887.845377 write 0x032e18 thru 0x03e9de (0xbbc7 bytes) 000830 1782481887.859272 trunc from 0x034232 to 0x03b38b (0x715a bytes) 000831 1782481887.882494 mapwrite 0x01eecd thru 0x02830b (0x943f bytes) 000832 1782481887.910682 mapread 0x01e81a thru 0x02a835 (0xc01c bytes) 000833 1782481887.911922 trunc from 0x0142e9 to 0x01ef3f (0xac57 bytes) 000834 1782481887.920293 write 0x013175 thru 0x014e6b (0x1cf7 bytes) 000835 1782481887.931434 trunc from 0x03ed88 to 0x01937b (0x25a0c bytes) 000836 1782481887.947563 punch from 0x014246 to 0x01937b (0x5136 bytes) 000837 1782481887.958821 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1937c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1937c): Mirror 2: 0xc9f6dfca (stale) CRC-32 checksum value for chunk [0, 0x1937c): Mirror 1: 0xc988976a Mirror 3: 0xc988976a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1937c: skipped 000838 1782481888.746113 mapread 0x00b4cc thru 0x011d4c (0x6881 bytes) 000839 1782481888.774014 fallocate 0x02f0f7 thru 0x03053f (0x1449 bytes) 000840 1782481888.806890 fallocate 0x00265b thru 0x008589 (0x5f2f bytes) 000841 1782481888.810940 trunc from 0x01937c to 0x03c4a1 (0x23126 bytes) 000842 1782481888.839965 mapread 0x001589 thru 0x004158 (0x2bd0 bytes) 000843 1782481888.853001 mapread 0x0071bc thru 0x0120a6 (0xaeeb bytes) 000844 1782481888.862792 mapwrite 0x0210c9 thru 0x02fe6b (0xeda3 bytes) 000845 1782481888.915099 punch from 0x029df9 to 0x02d1d2 (0x33da bytes) 000847 1782481888.920988 write 0x002982 thru 0x00ccc8 (0xa347 bytes) 000848 1782481888.922884 mapread 0x00f34d thru 0x01dff9 (0xecad bytes) 000849 1782481888.932956 mapread 0x027476 thru 0x027eef (0xa7a bytes) 000850 1782481888.935216 write 0x0089d6 thru 0x00c819 (0x3e44 bytes) 000851 1782481888.937728 fallocate 0x0151d8 thru 0x0244a2 (0xf2cb bytes) 000852 1782481888.945623 mapwrite 0x019b78 thru 0x01dcbd (0x4146 bytes) 000853 1782481888.973358 write 0x03ab8d thru 0x03ffff (0x5473 bytes) 000855 1782481888.981353 trunc from 0x040000 to 0x01d151 (0x22eae bytes) 000856 1782481889.010817 read 0x0037cf thru 0x00e96e (0xb1a0 bytes) 000857 1782481889.020272 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d152): Mirror 2: 0x5b52e7c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d152): Mirror 2: 0x5b52e7c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d152: skipped 000859 1782481889.280710 punch from 0x006595 to 0x015f1a (0xf986 bytes) 000860 1782481889.288632 write 0x028487 thru 0x02f29c (0x6e16 bytes) 000861 1782481889.300591 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2f29d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f29d): Mirror 2: 0x2d781347 (stale) CRC-32 checksum value for chunk [0, 0x2f29d): Mirror 1: 0x97014aea Mirror 3: 0x97014aea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f29d: skipped 000862 1782481890.060299 punch from 0x002c21 to 0x006def (0x41cf bytes) 000863 1782481890.101105 punch from 0x010308 to 0x01306d (0x2d66 bytes) 000864 1782481890.109244 fallocate 0x00ccb3 thru 0x00f809 (0x2b57 bytes) 000865 1782481890.116000 mapwrite 0x00382b thru 0x00b853 (0x8029 bytes) 000866 1782481890.171670 write 0x005ea7 thru 0x0138b5 (0xda0f bytes) 000867 1782481890.185694 write 0x01d69a thru 0x01f971 (0x22d8 bytes) 000869 1782481890.204663 trunc from 0x02f29d to 0x02a7ad (0x4aef bytes) 000870 1782481890.231631 punch from 0x01e22f to 0x0281a7 (0x9f79 bytes) 000871 1782481890.250473 mapread 0x011f3e thru 0x01e08a (0xc14d bytes) 000872 1782481890.280224 MIRROR_RESYNC 3 mirrors 000873 1782481891.009366 mapwrite 0x00a0cb thru 0x013bac (0x9ae2 bytes) 000874 1782481891.094484 fallocate 0x018087 thru 0x01957c (0x14f6 bytes) 000875 1782481891.105623 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2a7ae): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a7ae): Mirror 2: 0x99cc1ac0 (stale) Mirror 3: 0x99cc1ac0 (stale) CRC-32 checksum value for chunk [0, 0x2a7ae): Mirror 1: 0x463c4aa5 Mirror 4: 0x463c4aa5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a7ae: skipped 000876 1782481891.976359 write 0x029ac3 thru 0x02d0d1 (0x360f bytes) 000879 1782481892.028862 write 0x03a4e8 thru 0x03ffff (0x5b18 bytes) 000880 1782481892.030955 punch from 0x02a09d to 0x03680e (0xc772 bytes) 000881 1782481892.043895 fallocate 0x037e01 thru 0x03ffff (0x81ff bytes) 000882 1782481892.049759 punch from 0x014b0b to 0x019c56 (0x514c bytes) 000883 1782481892.058019 write 0x022c1c thru 0x02d269 (0xa64e bytes) 000884 1782481892.068729 mapread 0x0014b8 thru 0x00321a (0x1d63 bytes) 000885 1782481892.088655 mapwrite 0x025022 thru 0x028744 (0x3723 bytes) 000886 1782481892.113514 punch from 0x02e401 to 0x035597 (0x7197 bytes) 000887 1782481892.118696 trunc from 0x006355 to 0x010434 (0xa0e0 bytes) 000888 1782481892.125440 write 0x014742 thru 0x01a898 (0x6157 bytes) 000889 1782481892.135424 mapwrite 0x032cf7 thru 0x03501e (0x2328 bytes) 000890 1782481892.165545 trunc from 0x00b9a1 to 0x0175dc (0xbc3c bytes) 000891 1782481892.170737 punch from 0x004327 to 0x012d91 (0xea6b bytes) 000892 1782481892.175727 mapwrite 0x00203a thru 0x00a712 (0x86d9 bytes) 000893 1782481892.201056 read 0x0014a5 thru 0x009a27 (0x8583 bytes) 000895 1782481892.203211 trunc from 0x014cab to 0x021e59 (0xd1af bytes) 000896 1782481892.208717 write 0x01ca6d thru 0x01f1d0 (0x2764 bytes) 000897 1782481892.218785 mapread 0x011750 thru 0x01f330 (0xdbe1 bytes) 000898 1782481892.243110 read 0x00c1e7 thru 0x0129cd (0x67e7 bytes) 000899 1782481892.247007 write 0x03b2b6 thru 0x03ffff (0x4d4a bytes) 000900 1782481892.250073 trunc from 0x011de7 to 0x01634e (0x4568 bytes) 000901 1782481892.257801 trunc from 0x019ab0 to 0x02151d (0x7a6e bytes) 000902 1782481892.286813 MIRROR_RESYNC 4 mirrors 000903 1782481893.406284 punch from 0x018076 to 0x023159 (0xb0e4 bytes) 000904 1782481893.452094 trunc from 0x040000 to 0x02b7f7 (0x14808 bytes) 000905 1782481893.485707 punch from 0x015ce5 to 0x023573 (0xd88f bytes) 000906 1782481893.495288 punch from 0x01e248 to 0x0227ef (0x45a8 bytes) 000908 1782481893.507016 fallocate 0x0344cc thru 0x03b4d4 (0x7009 bytes) 000909 1782481893.514788 punch from 0x01c633 to 0x026ed2 (0xa8a0 bytes) 000910 1782481893.521920 mapread 0x01b137 thru 0x01f926 (0x47f0 bytes) 000911 1782481893.561457 mapwrite 0x0224b4 thru 0x02c31a (0x9e67 bytes) 000914 1782481893.622589 trunc from 0x011c75 to 0x0160d6 (0x4462 bytes) 000915 1782481893.628602 fallocate 0x0216cc thru 0x031063 (0xf998 bytes) 000916 1782481893.638486 read 0x008f3c thru 0x0170d4 (0xe199 bytes) 000917 1782481893.646327 trunc from 0x031064 to 0x02f71c (0x1947 bytes) 000918 1782481893.663032 fallocate 0x0204a7 thru 0x02b26e (0xadc8 bytes) 000919 1782481893.672478 trunc from 0x0030b1 to 0x00ec98 (0xbbe8 bytes) 000920 1782481893.678987 write 0x03c101 thru 0x03ffff (0x3eff bytes) 000921 1782481893.680491 mapread 0x029ac8 thru 0x0391db (0xf714 bytes) 000922 1782481893.713540 write 0x035457 thru 0x035e3c (0x9e6 bytes) 000923 1782481893.716683 read 0x025c68 thru 0x035223 (0xf5bc bytes) 000924 1782481893.717630 mapread 0x0106f0 thru 0x01ebe1 (0xe4f2 bytes) 000925 1782481893.721415 mapwrite 0x013feb thru 0x01b950 (0x7966 bytes) 000926 1782481893.742452 mapwrite 0x003a3e thru 0x0041e2 (0x7a5 bytes) 000927 1782481893.783230 MIRROR_RESYNC 4 mirrors 000929 1782481895.001331 fallocate 0x0346db thru 0x03ffff (0xb925 bytes) 000930 1782481895.054367 fallocate 0x030cc2 thru 0x03660d (0x594c bytes) 000931 1782481895.062378 read 0x020052 thru 0x02a4f8 (0xa4a7 bytes) 000932 1782481895.072229 mapwrite 0x0091f0 thru 0x0155d7 (0xc3e8 bytes) 000933 1782481895.124654 MIRROR_RESYNC 4 mirrors 000934 1782481896.329835 write 0x015fc9 thru 0x01ee87 (0x8ebf bytes) 000935 1782481896.364907 read 0x01359b thru 0x013856 (0x2bc bytes) 000936 1782481896.369578 write 0x021227 thru 0x02cd9e (0xbb78 bytes) 000937 1782481896.382430 mapread 0x023b7f thru 0x03397b (0xfdfd bytes) 000938 1782481896.400876 write 0x00274a thru 0x002c9c (0x553 bytes) 000939 1782481896.406948 trunc from 0x040000 to 0x02a82b (0x157d4 bytes) 000940 1782481896.428779 punch from 0x013b14 to 0x01d757 (0x9c44 bytes) 000941 1782481896.446757 punch from 0x006a61 to 0x00c3c2 (0x5962 bytes) 000942 1782481896.453535 read 0x00ac34 thru 0x0146ca (0x9a97 bytes) 000944 1782481896.474919 trunc from 0x02a82c to 0x02c13e (0x1913 bytes) 000945 1782481896.491554 mapread 0x003779 thru 0x011921 (0xe1a9 bytes) 000946 1782481896.492972 mapwrite 0x01ad76 thru 0x020715 (0x59a0 bytes) 000947 1782481896.514802 trunc from 0x019fd2 to 0x01bf24 (0x1f53 bytes) 000948 1782481896.520408 fallocate 0x00c964 thru 0x01686e (0x9f0b bytes) 000949 1782481896.526662 trunc from 0x0051d4 to 0x00c151 (0x6f7e bytes) 000950 1782481896.532396 punch from 0x025f44 to 0x02bc30 (0x5ced bytes) 000951 1782481896.541404 fallocate 0x01e040 thru 0x0217f1 (0x37b2 bytes) 000952 1782481896.546174 mapwrite 0x00e950 thru 0x016808 (0x7eb9 bytes) 000953 1782481896.571529 write 0x011906 thru 0x01e80e (0xcf09 bytes) 000955 1782481896.574720 trunc from 0x0285e3 to 0x028b77 (0x595 bytes) 000958 1782481896.590829 punch from 0x0187fc to 0x024908 (0xc10d bytes) 000959 1782481896.617566 mapread 0x024fda thru 0x02c13e (0x7165 bytes) 000960 1782481896.679167 read 0x000e91 thru 0x004f27 (0x4097 bytes) 000961 1782481896.680374 trunc from 0x02c13f to 0x0127f7 (0x19947 bytes) 000962 1782481896.702999 write 0x0102ce thru 0x019504 (0x9237 bytes) 000963 1782481896.704925 punch from 0x00a9c2 to 0x00ae4c (0x48b bytes) 000964 1782481896.712028 trunc from 0x009fd7 to 0x012048 (0x8072 bytes) 000966 1782481896.743886 mapread 0x00c24b thru 0x018f3e (0xccf4 bytes) 000967 1782481896.763059 mapwrite 0x013b4b thru 0x021710 (0xdbc6 bytes) 000968 1782481896.822723 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21711): Mirror 2: 0x43a0ab5a (stale) Mirror 4: 0x43a0ab5a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21711): Mirror 2: 0x43a0ab5a (stale) Mirror 4: 0x43a0ab5a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21711: skipped 000969 1782481897.160199 punch from 0x007e22 to 0x011b0a (0x9ce9 bytes) 000970 1782481897.168043 mapread 0x01affe thru 0x021710 (0x6713 bytes) 000971 1782481897.188990 mapread 0x0095be thru 0x016832 (0xd275 bytes) 000972 1782481897.210087 trunc from 0x01aeaf to 0x01c5d6 (0x1728 bytes) 000973 1782481897.215922 punch from 0x01dd6a to 0x021710 (0x39a7 bytes) 000974 1782481897.221259 fallocate 0x03e883 thru 0x03ffff (0x177d bytes) 000975 1782481897.226735 fallocate 0x02f9da thru 0x0380a8 (0x86cf bytes) 000976 1782481897.232965 trunc from 0x00e5b0 to 0x010f51 (0x29a2 bytes) 000977 1782481897.241234 mapread 0x013b08 thru 0x01e1bd (0xa6b6 bytes) 000978 1782481897.256576 trunc from 0x021711 to 0x01c8ce (0x4e42 bytes) 000979 1782481897.282116 mapwrite 0x00124f thru 0x005e2f (0x4be1 bytes) 000980 1782481897.315558 read 0x01336b thru 0x0135ad (0x243 bytes) 000981 1782481897.316334 read 0x008084 thru 0x00fe9b (0x7e18 bytes) 000982 1782481897.316967 trunc from 0x01c8cf to 0x01f9cf (0x3101 bytes) 000983 1782481897.341094 trunc from 0x01f9d0 to 0x01db2f (0x1ea0 bytes) 000984 1782481897.360273 write 0x009c80 thru 0x0102b6 (0x6637 bytes) 000985 1782481897.369312 punch from 0x006401 to 0x015d26 (0xf926 bytes) 000986 1782481897.390828 mapread 0x0190b9 thru 0x01b431 (0x2379 bytes) 000987 1782481897.392334 trunc from 0x018edf to 0x01db2f (0x4c51 bytes) 000988 1782481897.398251 read 0x004484 thru 0x013746 (0xf2c3 bytes) 000990 1782481897.412323 trunc from 0x00f512 to 0x015960 (0x644f bytes) 000991 1782481897.418439 mapwrite 0x02ac84 thru 0x03a453 (0xf7d0 bytes) 000992 1782481897.490726 fallocate 0x00be38 thru 0x00f52a (0x36f3 bytes) 000993 1782481897.498883 mapwrite 0x0023f3 thru 0x00d8c5 (0xb4d3 bytes) 000994 1782481897.519040 read 0x02cd09 thru 0x038e3b (0xc133 bytes) 000995 1782481897.523303 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a454): Mirror 4: 0x8d19e685 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a454): Mirror 4: 0x8d19e685 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a454: skipped 000996 1782481897.818376 MIRROR_RESYNC 2 mirrors 000997 1782481898.503025 trunc from 0x03a454 to 0x008b9f (0x318b4 bytes) 000998 1782481898.549784 punch from 0x0037bc to 0x008b9f (0x53e4 bytes) 000999 1782481898.558525 punch from 0x0056f9 to 0x008b9f (0x34a7 bytes) 001000 1782481898.564322 trunc from 0x008ba0 to 0x00e5fd (0x5a5e bytes) All operations completed A-OK! PASS 70a (70s) == sanity-flr test 71: check mirror extend parent fid ==== 09:51:42 (1782481902) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x234:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (8s) == sanity-flr test 72: check append on FLR file ========== 09:51:50 (1782481910) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.16407 s, 12.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100114 s, 10.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x135:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (5s) == sanity-flr test 200a: stress test ===================== 09:51:55 (1782481915) Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre2 Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4657696 .. Extending file size to 2026560 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 2532768 .. Extending file size to 7873760 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7249440 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8046048 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9252672 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8651328 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9223328 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9263296 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61526 61527 61528 61530 61532 done 192.168.204.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg437-client.virtnet /mnt/lustre2 (opts:) 192.168.204.137@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg437-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (73s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:53:08 (1782481988) Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre2 Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 2240256 .. Extending file size to 5548256 .. Extending file size to 4131168 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6713536 .. Extending file size to 6319520 .. done Extending file size to 6875456 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8008800 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7605120 .. Extending file size to 8161184 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8415488 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8464000 .. Extending file size to 8549088 .. failed Extending file size to 8928448 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8798560 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8996064 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9192704 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 63497 63498 63499 63502 63504 done 192.168.204.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg437-client.virtnet /mnt/lustre2 (opts:) 192.168.204.137@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg437-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (72s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:54:20 (1782482060) Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2456 1268848 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2164 1269140 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17824 3589196 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25828 3581192 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43652 7170388 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2456 1268848 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2164 1269140 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17824 3589196 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25828 3581192 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43652 7170388 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0783915 s, 26.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.10351 s, 20.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xf5d50f33 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg437-client.virtnet /mnt/lustre2 (opts:) PASS 200c (10s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:54:31 (1782482071) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0710493 s, 29.5 MB/s PASS 202 (4s) == sanity-flr test 203: mirror file preserve mirror ID === 09:54:35 (1782482075) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x95:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x149:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x149:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14a:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14a:0x0] } PASS 203 (5s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:54:40 (1782482080) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0610528 s, 34.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (4s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:54:45 (1782482085) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0713917 s, 29.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14e:0x0] } PASS 204b (4s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:54:49 (1782482089) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0654773 s, 32.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x150:0x0] } PASS 204c (29s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:55:18 (1782482118) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg437-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101829 s, 10.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2e7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2e7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2e7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2e7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2e8:0x0] } PASS 204d (44s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:56:03 (1782482163) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg437-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2ea:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0776937 s, 27.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2ea:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2e9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2ea:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2e9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (45s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:56:48 (1782482208) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg437-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg437-server: Pool lustre.test_204f created Adding targets to pool oleg437-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg437-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0749407 s, 28.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x151:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x151:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg437-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg437-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg437-server: Pool lustre.test_204f destroyed PASS 204f (60s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:57:49 (1782482269) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:57:54 (1782482274) PASS 205b (5s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:58:00 (1782482280) oleg437-server: Pool lustre.test_206 created Adding targets to pool oleg437-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg437-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg437-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg437-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg437-server: Pool lustre.test_206 destroyed PASS 206 (21s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:58:21 (1782482301) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595144 s, 17.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356741 s, 29.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xd432e9d2 Mirror 3: 0xd432e9d2 PASS 207 (8s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:58:29 (1782482309) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:58:31 (1782482311) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:58:34 (1782482314) Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0582669 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x850000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1a00000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1a00000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1a00000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x15c0000 Read this (4 bytes): BBBB 192.168.204.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg437-client.virtnet /mnt/lustre2 (opts:) PASS 209a (28s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:59:02 (1782482342) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0155029 s, 528 kB/s the very first read read with warm cache PASS 209b (5s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:59:07 (1782482347) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051436 s, 20.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb6 lmm_fid: [0x200000405:0xb6:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb6 lmm_fid: [0x200000405:0xb6:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15c:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611318 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-26 09:59:09.000000000 -0400 Modify: 2026-06-26 09:59:09.000000000 -0400 Change: 2026-06-26 09:59:10.000000000 -0400 Birth: 2026-06-26 09:59:08.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:59:14 (1782482354) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370484 s, 28.3 MB/s oleg437-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg437-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg437-server: O/280000401/d25/159: File not found by ext2_lookup PASS 210b (17s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:59:31 (1782482371) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.13068 s, 313 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0152313 s, 269 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 26 09:59 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbb:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 26 09:59 /mnt/lustre/f211.sanity-flr PASS 211 (6s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 09:59:37 (1782482377) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (6s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 09:59:43 (1782482383) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (6s) == sanity-flr test complete, duration 1601 sec =========== 09:59:49 (1782482389) === sanity-flr: start cleanup 09:59:51 (1782482391) === === sanity-flr: finish cleanup 09:59:58 (1782482398) ===