-----============= acceptance-small: sanity-flr ============----- Fri Sep 4 07:57:59 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg654-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 07:58:18 (1788523098) === oleg654-client.virtnet: executing check_config_client /mnt/lustre oleg654-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg654-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff94bcc74c3800.idle_timeout=debug osc.lustre-OST0001-osc-ffff94bcc74c3800.idle_timeout=debug disable quota as required oleg654-server: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 07:59:02 (1788523142) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8273] == sanity-flr test 0a: lfs mirror create with -N option == 07:59:05 (1788523145) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 07:59:23 (1788523163) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 07:59:27 (1788523167) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 07:59:30 (1788523170) PASS 0d (21s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 07:59:51 (1788523191) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 07:59:54 (1788523194) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 07:59:57 (1788523197) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff94bcc74c3800.stats=clear osc.lustre-OST0001-osc-ffff94bcc74c3800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 08:00:05 (1788523205) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 08:00:14 (1788523214) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 08:00:22 (1788523222) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 08:00:25 (1788523225) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 08:00:32 (1788523232) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (6s) == sanity-flr test 2: create components from existing files ========================================================== 08:00:39 (1788523239) PASS 2 (5s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 08:00:44 (1788523244) PASS 3 (6s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 08:00:50 (1788523250) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (5s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 08:00:56 (1788523256) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 08:01:05 (1788523265) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 08:01:11 (1788523271) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.717074 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619665 s, 16.9 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 08:01:17 (1788523277) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.391356 s, 40.2 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 08:01:24 (1788523284) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (8s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 08:01:32 (1788523292) Stopping /mnt/lustre-ost1 (opts:) on oleg654-server oleg654-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94bcc74c3800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94bcc74c3800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg654-server: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg654-server: oleg654-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg654-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg654-server oleg654-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff94bcc74c3800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94bcc74c3800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg654-server: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg654-server: oleg654-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg654-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (61s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 08:02:33 (1788523353) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.289249 s, 25.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg654-server oleg654-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94bcc74c3800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94bcc74c3800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg654-server: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg654-server: oleg654-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg654-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 08:03:15 (1788523395) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg654-server oleg654-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94bcc74c3800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94bcc74c3800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg654-server: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg654-server: oleg654-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg654-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg654-server oleg654-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff94bcc74c3800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94bcc74c3800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg654-server: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg654-server: oleg654-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg654-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (76s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 08:04:31 (1788523471) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.04341 s, 17.2 MB/s -rw-r--r-- 1 root root 52428800 Sep 4 08:04 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.823s user 0m0.000s sys 0m0.154s Stopping /mnt/lustre-ost1 (opts:) on oleg654-server oleg654-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94bcc74c3800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94bcc74c3800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.991s user 0m0.000s sys 0m0.209s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg654-server: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg654-server: oleg654-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg654-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg654-server oleg654-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff94bcc74c3800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94bcc74c3800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.982s user 0m0.002s sys 0m0.190s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg654-server: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg654-server: oleg654-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg654-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg654-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff94bcc74c3800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94bcc74c3800.ost_server_uuid in FULL state after 0 sec PASS 33b (90s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 08:06:01 (1788523561) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 08:06:04 (1788523564) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 08:06:07 (1788523567) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 08:06:09 (1788523569) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 08:06:16 (1788523576) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.593466 s, 21.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 08:06:28 (1788523588) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026952 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206457 s, 50.8 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 08:06:35 (1788523595) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 08:06:43 (1788523603) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00100588 s, 1.0 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00141304 s, 725 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00142634 s, 718 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00113897 s, 899 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 08:06:52 (1788523612) create test files with size 15758027 .. bc54f70d0a355589556968afe9add268 - 156a3bf3d052f3b9cf84c4099ff92f86 - 8dd0233a00666bbf2da7313f332d6507 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg654-client.virtnet /mnt/lustre (opts:) Starting client: oleg654-client.virtnet: -o user_xattr,flock 192.168.206.154@tcp:/lustre /mnt/lustre PASS 37 (21s) == sanity-flr test 38: resync ============================ 08:07:13 (1788523633) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (18s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 08:07:31 (1788523651) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 08:07:38 (1788523658) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982301 s, 10.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040672 s, 25.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 08:07:50 (1788523670) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0762933 s, 27.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.369045 s, 11.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.167949 s, 12.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 08:08:09 (1788523689) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 08:08:12 (1788523692) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 08:08:16 (1788523696) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478064 s, 21.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 08:08:24 (1788523704) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.464122 s, 6.8 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 08:08:36 (1788523716) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd2:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd2:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 08:08:46 (1788523726) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.281785 s, 37.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg654-client.virtnet /mnt/lustre (opts:) Starting client: oleg654-client.virtnet: -o user_xattr,flock 192.168.206.154@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 08:08:57 (1788523737) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff94bcc97b9000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.294125 s, 35.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg654-client.virtnet /mnt/lustre (opts:) Starting client: oleg654-client.virtnet: -o user_xattr,flock 192.168.206.154@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff94bcc8929000.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 08:09:11 (1788523751) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (33s) == sanity-flr test 46: Verify setstripe --copy option ==== 08:09:44 (1788523784) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (31s) == sanity-flr test 47: Verify mirror obj alloc =========== 08:10:15 (1788523815) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 08:10:18 (1788523818) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.205194 s, 15.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.170684 s, 18.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x3f38af84 (nosync,stale) Mirror 2: 0x3f38af84 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 389101ab46c53be7db94034758338ede - mirror 1 checksum: 401ac1eb9dd20c14ba31e8958791fce2 - mirror 2 checksum: 389101ab46c53be7db94034758338ede - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-04T08:10:20-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-04T08:10:20-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 389101ab46c53be7db94034758338ede - mirror 1 checksum: 401ac1eb9dd20c14ba31e8958791fce2 - mirror 2 checksum: 401ac1eb9dd20c14ba31e8958791fce2 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-04T08:10:20-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-04T08:10:20-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } PASS 48 (8s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 08:10:26 (1788523826) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360962 s, 29.0 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (6s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 08:10:32 (1788523832) Failing mds1 on oleg654-server Stopping /mnt/lustre-mds1 (opts:) on oleg654-server 08:10:44 (1788523844) shut down facet: mds1 facet_host: oleg654-server facet_failover_host: oleg654-server Failover mds1 to oleg654-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg654-server: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-MDT0000 08:11:15 (1788523875) targets are mounted 08:11:15 (1788523875) facet_failover done oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x11:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425658 s, 24.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x11 lmm_fid: [0x200000405:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } PASS 50A (58s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 08:11:30 (1788523890) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0235688 s, 869 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0119263 s, 1.7 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (8s) == sanity-flr test 50b: mirror rsync handles sparseness == 08:11:38 (1788523898) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0559227 s, 366 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.031515 s, 650 kB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (8s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 08:11:46 (1788523906) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0045464 s, 3.6 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 08:11:52 (1788523912) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0756383 s, 271 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (7s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 08:11:59 (1788523919) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00554173 s, 739 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0358342 s, 114 kB/s PASS 60a (6s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 08:12:05 (1788523925) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) PASS 60b (17s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 08:12:22 (1788523942) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (26s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 08:12:49 (1788523969) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 08:13:12 (1788523992) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 08:13:36 (1788524016) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 08:13:44 (1788524024) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 57611 mirror split pid 57613 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57611 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57613 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 08:14:51 (1788524091) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 4093 fallocating to largest ever: 0x34d25 000001 1788524093.691687 fallocate 0x03127a thru 0x034d24 (0x3aab bytes) 000002 1788524093.699654 read 0x0035cd thru 0x00c58a (0x8fbe bytes) 000003 1788524093.708264 mapwrite 0x00da07 thru 0x011914 (0x3f0e bytes) 000004 1788524093.737415 fallocate 0x008d5e thru 0x0111d3 (0x8476 bytes) 000005 1788524093.741781 mapread 0x002b24 thru 0x0124ac (0xf989 bytes) 000006 1788524093.747522 trunc from 0x034d25 to 0x02a5cc (0xa758 bytes) 000007 1788524093.761771 trunc from 0x02a5cd to 0x01ac92 (0xf93a bytes) 000008 1788524093.781079 MIRROR_EXTEND to 2 mirrors 000009 1788524094.465873 mapread 0x019931 thru 0x01ac92 (0x1362 bytes) 000010 1788524094.482904 mapwrite 0x004c40 thru 0x00b88e (0x6c4f bytes) 000011 1788524094.647051 read 0x0057e4 thru 0x00acd6 (0x54f3 bytes) 000012 1788524094.647630 write 0x02b8d4 thru 0x0383d6 (0xcb03 bytes) 000013 1788524094.662345 write 0x010bda thru 0x01e513 (0xd93a bytes) 000014 1788524094.669107 fallocate 0x0154c7 thru 0x024df2 (0xf92c bytes) fallocating to largest ever: 0x40000 000015 1788524094.674912 fallocate 0x038bf3 thru 0x03ffff (0x740d bytes) 000016 1788524094.679551 write 0x013bb6 thru 0x018638 (0x4a83 bytes) 000017 1788524094.684246 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1bf:0x0]: Invalid argument (22) 000018 1788524094.932034 read 0x027724 thru 0x035650 (0xdf2d bytes) 000022 1788524094.952437 trunc from 0x0383d7 to 0x02cbe2 (0xb7f4 bytes) 000023 1788524094.969624 fallocate 0x0211a1 thru 0x02f84c (0xe6ac bytes) 000024 1788524094.975291 trunc from 0x02cbe3 to 0x00f58e (0x1d654 bytes) 000025 1788524094.990086 read 0x008815 thru 0x00f58e (0x6d7a bytes) 000026 1788524095.004592 trunc from 0x00f58f to 0x036159 (0x26bcb bytes) 000027 1788524095.019657 read 0x01c2fd thru 0x022481 (0x6185 bytes) 000028 1788524095.031681 read 0x028784 thru 0x036159 (0xd9d6 bytes) 000033 1788524095.036469 mapwrite 0x018020 thru 0x018764 (0x745 bytes) 000036 1788524095.050476 read 0x01464c thru 0x01eef4 (0xa8a9 bytes) 000037 1788524095.051376 punch from 0x035506 to 0x036159 (0xc54 bytes) 000038 1788524095.055445 read 0x026316 thru 0x02e7b2 (0x849d bytes) 000039 1788524095.056640 trunc from 0x01f474 to 0x02f3c9 (0xff56 bytes) 000042 1788524095.063671 trunc from 0x03615a to 0x02efe0 (0x7179 bytes) 000044 1788524095.080555 read 0x01b9bc thru 0x023c9d (0x82e2 bytes) 000047 1788524095.092049 mapread 0x02eee5 thru 0x02efe0 (0xfc bytes) 000048 1788524095.093204 mapread 0x02df53 thru 0x02efe0 (0x108e bytes) 000049 1788524095.094250 fallocate 0x039109 thru 0x03d260 (0x4158 bytes) 000050 1788524095.098638 fallocate 0x022fdb thru 0x02af81 (0x7fa7 bytes) 000051 1788524095.102734 write 0x01c17a thru 0x027fd7 (0xbe5e bytes) 000052 1788524095.103907 mapwrite 0x0294d3 thru 0x032fc3 (0x9af1 bytes) 000053 1788524095.142314 trunc from 0x01cf0c to 0x02b04a (0xe13f bytes) 000054 1788524095.147267 trunc from 0x032fc4 to 0x02a9a5 (0x861e bytes) 000055 1788524095.164484 read 0x007f7c thru 0x00b8ec (0x3971 bytes) 000056 1788524095.165095 mapread 0x0292f2 thru 0x02a9a5 (0x16b4 bytes) 000057 1788524095.188103 read 0x004df7 thru 0x014b8d (0xfd97 bytes) 000058 1788524095.195090 MIRROR_EXTEND to 2 mirrors 000060 1788524096.397922 trunc from 0x01c922 to 0x01fa16 (0x30f5 bytes) 000061 1788524096.467937 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1bf:0x0]: Invalid argument (22) 000062 1788524096.696681 mapwrite 0x00b6ec thru 0x00d608 (0x1f1d bytes) 000063 1788524096.777902 write 0x037549 thru 0x03ffff (0x8ab7 bytes) 000064 1788524096.791545 trunc from 0x040000 to 0x02e703 (0x118fc bytes) 000065 1788524096.866032 trunc from 0x02e704 to 0x0390da (0xa9d7 bytes) 000066 1788524096.942395 write 0x03147a thru 0x03a2b6 (0x8e3d bytes) 000067 1788524096.953266 fallocate 0x02d371 thru 0x02feed (0x2b7d bytes) 000068 1788524096.969333 write 0x02e8bf thru 0x02fa35 (0x1177 bytes) 000069 1788524096.982269 fallocate 0x0338bf thru 0x03ffff (0xc741 bytes) 000071 1788524097.016130 trunc from 0x0053e9 to 0x00541b (0x33 bytes) 000072 1788524097.035118 read 0x02925d thru 0x02ded2 (0x4c76 bytes) 000073 1788524097.066538 trunc from 0x040000 to 0x035212 (0xaded bytes) 000074 1788524097.095275 punch from 0x023eb3 to 0x02894d (0x4a9b bytes) 000075 1788524097.102592 write 0x010f36 thru 0x0196b3 (0x877e bytes) 000077 1788524097.126361 MIRROR_EXTEND to 2 mirrors 000078 1788524098.148416 trunc from 0x035213 to 0x002da7 (0x3246b bytes) 000079 1788524098.264302 write 0x007146 thru 0x0072d4 (0x18f bytes) 000080 1788524098.266089 read 0x001dbc thru 0x003204 (0x1449 bytes) 000081 1788524098.274079 punch from 0x0015e2 to 0x0072d4 (0x5cf3 bytes) 000082 1788524098.290669 mapread 0x00605d thru 0x0072d4 (0x1278 bytes) 000083 1788524098.305610 punch from 0x006c0c to 0x0072d4 (0x6c9 bytes) 000084 1788524098.315153 fallocate 0x036366 thru 0x03a283 (0x3f1e bytes) 000086 1788524098.324571 trunc from 0x00137f to 0x007060 (0x5ce2 bytes) 000087 1788524098.336449 fallocate 0x01f3ab thru 0x02ed7c (0xf9d2 bytes) 000088 1788524098.350609 mapread 0x021fd4 thru 0x02d268 (0xb295 bytes) 000089 1788524098.377577 mapwrite 0x00d509 thru 0x01a0e0 (0xcbd8 bytes) 000090 1788524098.428840 fallocate 0x0294b1 thru 0x02f69a (0x61ea bytes) 000091 1788524098.435938 read 0x01886f thru 0x01fda8 (0x753a bytes) 000092 1788524098.436529 read 0x037c07 thru 0x03a283 (0x267d bytes) 000093 1788524098.447571 mapwrite 0x01bc74 thru 0x029939 (0xdcc6 bytes) 000094 1788524098.490896 fallocate 0x0080d0 thru 0x013425 (0xb356 bytes) 000095 1788524098.504478 read 0x00d892 thru 0x01bced (0xe45c bytes) 000096 1788524098.506551 trunc from 0x03a284 to 0x01f254 (0x1b02f bytes) 000097 1788524098.562211 trunc from 0x01f255 to 0x034fd7 (0x15d83 bytes) 000099 1788524098.615281 punch from 0x032a40 to 0x034fd7 (0x2598 bytes) 000100 1788524098.620330 trunc from 0x034fd8 to 0x01080e (0x247c9 bytes) 000101 1788524098.658204 read 0x00f93e thru 0x01080e (0xed1 bytes) 000102 1788524098.658852 trunc from 0x01080f to 0x019a65 (0x9257 bytes) 000103 1788524098.685190 punch from 0x0035b5 to 0x00a747 (0x7193 bytes) 000104 1788524098.692064 mapread 0x014fa9 thru 0x019a65 (0x4abd bytes) 000107 1788524098.718280 mapwrite 0x017f8b thru 0x026041 (0xe0b7 bytes) 000108 1788524098.804064 trunc from 0x01a0c0 to 0x021890 (0x77d1 bytes) 000109 1788524098.820055 write 0x00c282 thru 0x01c244 (0xffc3 bytes) 000110 1788524098.846411 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1bf:0x0]: Invalid argument (22) 000111 1788524099.143217 mapwrite 0x024610 thru 0x0339a8 (0xf399 bytes) 000113 1788524099.219377 punch from 0x01b735 to 0x025cfa (0xa5c6 bytes) 000114 1788524099.225885 read 0x00edec thru 0x014b53 (0x5d68 bytes) 000115 1788524099.233979 mapwrite 0x03f9bc thru 0x03ffff (0x644 bytes) 000116 1788524099.271434 fallocate 0x02c55e thru 0x03adf7 (0xe89a bytes) 000117 1788524099.276407 mapwrite 0x0038ae thru 0x00de6e (0xa5c1 bytes) 000118 1788524099.308474 fallocate 0x01f629 thru 0x02847c (0x8e54 bytes) 000120 1788524099.319553 trunc from 0x040000 to 0x005797 (0x3a868 bytes) 000121 1788524099.342785 mapwrite 0x036e0d thru 0x03ffff (0x91f3 bytes) 000122 1788524099.403652 trunc from 0x040000 to 0x033ddb (0xc224 bytes) 000123 1788524099.415923 mapwrite 0x03f952 thru 0x03ffff (0x6ae bytes) 000124 1788524099.447306 punch from 0x024172 to 0x0251a1 (0x1030 bytes) 000125 1788524099.451971 trunc from 0x02529c to 0x033424 (0xe189 bytes) 000126 1788524099.456076 trunc from 0x040000 to 0x001b93 (0x3e46c bytes) 000127 1788524099.469069 punch from 0x0000d2 to 0x001b93 (0x1ac2 bytes) 000128 1788524099.473756 mapwrite 0x02685a thru 0x031556 (0xacfd bytes) 000129 1788524099.519505 punch from 0x00ad52 to 0x00ea6b (0x3d1a bytes) 000130 1788524099.526597 fallocate 0x03e0fd thru 0x03ffff (0x1f03 bytes) 000131 1788524099.536305 punch from 0x029364 to 0x02fc7e (0x691b bytes) 000133 1788524099.554371 write 0x025120 thru 0x02f48d (0xa36e bytes) 000134 1788524099.561603 trunc from 0x03fcbc to 0x03ffff (0x344 bytes) 000136 1788524099.568097 read 0x02cea1 thru 0x033542 (0x66a2 bytes) 000137 1788524099.580518 write 0x024aae thru 0x02edbd (0xa310 bytes) 000138 1788524099.605966 trunc from 0x040000 to 0x0312b5 (0xed4a bytes) 000139 1788524099.628511 MIRROR_EXTEND to 2 mirrors 000140 1788524100.233738 punch from 0x02e985 to 0x0312b5 (0x2931 bytes) 000143 1788524100.285226 trunc from 0x01fc01 to 0x02c28a (0xc68a bytes) 000144 1788524100.295127 trunc from 0x011ebd to 0x01a4ec (0x8630 bytes) 000145 1788524100.306466 trunc from 0x0312b6 to 0x0188fa (0x189bb bytes) 000146 1788524100.389645 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x188fb): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x188fb): Mirror 2: 0xce117cdd (stale) CRC-32 checksum value for chunk [0, 0x188fb): Mirror 1: 0xce117cdd Mirror 3: 0xce117cdd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x188fb: skipped 000147 1788524101.081339 fallocate 0x022c31 thru 0x02c16e (0x953e bytes) 000148 1788524101.120330 mapread 0x00b847 thru 0x00cb39 (0x12f3 bytes) 000151 1788524101.129643 trunc from 0x0149a8 to 0x016a36 (0x208f bytes) 000152 1788524101.135449 mapread 0x016b83 thru 0x0180fb (0x1579 bytes) 000153 1788524101.154242 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x188fb): Mirror 3: 0xce117cdd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x188fb): Mirror 3: 0xce117cdd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x188fb: skipped 000154 1788524101.579843 punch from 0x0151f8 to 0x0188fa (0x3703 bytes) 000155 1788524101.596595 fallocate 0x00ec37 thru 0x0178da (0x8ca4 bytes) 000157 1788524101.608653 read 0x017c26 thru 0x0188fa (0xcd5 bytes) 000158 1788524101.622067 write 0x03ef2f thru 0x03ffff (0x10d1 bytes) 000159 1788524101.630439 trunc from 0x011015 to 0x01235e (0x134a bytes) 000160 1788524101.634459 punch from 0x01b774 to 0x01c747 (0xfd4 bytes) 000162 1788524101.641744 read 0x0369b8 thru 0x037a3b (0x1084 bytes) 000163 1788524101.674336 read 0x00f784 thru 0x01b6e0 (0xbf5d bytes) 000164 1788524101.674993 mapread 0x0274f1 thru 0x02c922 (0x5432 bytes) 000167 1788524101.677645 fallocate 0x02e1e2 thru 0x02eab1 (0x8d0 bytes) 000168 1788524101.686819 punch from 0x0087d9 to 0x01523d (0xca65 bytes) 000169 1788524101.691609 mapwrite 0x0201ce thru 0x023b10 (0x3943 bytes) 000170 1788524101.730482 trunc from 0x035aec to 0x03ffff (0xa514 bytes) 000171 1788524101.744443 trunc from 0x03dbca to 0x03ed80 (0x11b7 bytes) 000172 1788524101.749148 mapwrite 0x020d23 thru 0x02a8d2 (0x9bb0 bytes) 000173 1788524101.768800 mapwrite 0x0362f3 thru 0x03b3c5 (0x50d3 bytes) 000174 1788524101.803612 punch from 0x001439 to 0x009991 (0x8559 bytes) 000176 1788524101.808618 write 0x0294e2 thru 0x02a81e (0x133d bytes) 000177 1788524101.809568 trunc from 0x01bc1c to 0x028cc9 (0xd0ae bytes) 000178 1788524101.816595 mapwrite 0x00c9b4 thru 0x015e78 (0x94c5 bytes) 000179 1788524101.836600 read 0x014ed6 thru 0x01da58 (0x8b83 bytes) 000180 1788524101.864875 trunc from 0x040000 to 0x02a27f (0x15d80 bytes) 000181 1788524101.889329 mapwrite 0x00ae08 thru 0x018154 (0xd34d bytes) 000182 1788524101.922576 mapwrite 0x00cfbf thru 0x0142a8 (0x72ea bytes) 000183 1788524101.977599 read 0x01592c thru 0x0245fa (0xeccf bytes) 000184 1788524101.978751 write 0x003fad thru 0x00ad37 (0x6d8b bytes) 000185 1788524101.980340 mapread 0x001af6 thru 0x0041ec (0x26f7 bytes) 000186 1788524101.983200 trunc from 0x02a280 to 0x0120db (0x181a4 bytes) 000187 1788524102.015824 punch from 0x00b66c to 0x0120db (0x6a70 bytes) 000188 1788524102.024375 mapwrite 0x01ca54 thru 0x028796 (0xbd43 bytes) 000189 1788524102.110783 read 0x02597c thru 0x028796 (0x2e1b bytes) 000190 1788524102.111685 write 0x002b60 thru 0x01237d (0xf81e bytes) 000191 1788524102.116513 punch from 0x00d190 to 0x013571 (0x63e2 bytes) 000192 1788524102.148020 trunc from 0x0009df to 0x0084db (0x7afd bytes) 000193 1788524102.159780 trunc from 0x028797 to 0x00a4a8 (0x1e2ee bytes) 000194 1788524102.182886 punch from 0x0057e9 to 0x00a0a3 (0x48bb bytes) 000195 1788524102.190560 mapread 0x009453 thru 0x00a4a8 (0x1056 bytes) 000196 1788524102.200329 fallocate 0x0032dc thru 0x00c20f (0x8f34 bytes) 000197 1788524102.204383 trunc from 0x00a4a9 to 0x00b94b (0x14a3 bytes) 000200 1788524102.219991 write 0x0344d7 thru 0x03ffff (0xbb29 bytes) 000201 1788524102.222948 trunc from 0x0186a0 to 0x018a70 (0x3d1 bytes) 000202 1788524102.231358 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1bf:0x0]: Invalid argument (22) 000203 1788524102.411288 punch from 0x031396 to 0x032ee8 (0x1b53 bytes) 000205 1788524102.416964 punch from 0x011a76 to 0x0193b3 (0x793e bytes) 000206 1788524102.423604 punch from 0x0039dd to 0x00d87d (0x9ea1 bytes) 000207 1788524102.429230 punch from 0x0318f7 to 0x038688 (0x6d92 bytes) 000208 1788524102.434453 mapwrite 0x00f8ca thru 0x01c4f3 (0xcc2a bytes) 000211 1788524102.520178 read 0x00b3c5 thru 0x00bcf5 (0x931 bytes) 000212 1788524102.521284 punch from 0x0250a3 to 0x0343f8 (0xf356 bytes) 000213 1788524102.526141 fallocate 0x02aef7 thru 0x02d483 (0x258d bytes) 000214 1788524102.533528 mapread 0x036032 thru 0x0398c4 (0x3893 bytes) 000215 1788524102.545429 fallocate 0x02bd98 thru 0x02c3b0 (0x619 bytes) 000216 1788524102.550566 punch from 0x00e38c to 0x018828 (0xa49d bytes) 000217 1788524102.564811 punch from 0x036cba to 0x03bf81 (0x52c8 bytes) 000218 1788524102.573188 write 0x03037c thru 0x036b14 (0x6799 bytes) 000219 1788524102.590584 read 0x03eaff thru 0x03ffff (0x1501 bytes) 000220 1788524102.604997 trunc from 0x040000 to 0x00b918 (0x346e7 bytes) 000221 1788524102.628249 mapread 0x0040b6 thru 0x00afa6 (0x6ef1 bytes) 000223 1788524102.630457 read 0x0065e5 thru 0x009c89 (0x36a5 bytes) 000224 1788524102.639533 trunc from 0x00b919 to 0x018f91 (0xd679 bytes) 000225 1788524102.675836 read 0x0049f9 thru 0x007d9b (0x33a3 bytes) 000226 1788524102.676424 trunc from 0x018f92 to 0x03e7c6 (0x25835 bytes) 000228 1788524102.724611 trunc from 0x006508 to 0x0117ba (0xb2b3 bytes) 000230 1788524102.755446 trunc from 0x03e7c7 to 0x025549 (0x1927d bytes) 000231 1788524102.786291 fallocate 0x0093ee thru 0x017d40 (0xe953 bytes) 000232 1788524102.795069 read 0x01bffc thru 0x023ee1 (0x7ee6 bytes) 000234 1788524102.831156 fallocate 0x013ecf thru 0x014f7f (0x10b1 bytes) 000235 1788524102.851631 trunc from 0x00bdf7 to 0x00de51 (0x205b bytes) 000236 1788524102.872460 mapwrite 0x011044 thru 0x01f318 (0xe2d5 bytes) 000237 1788524102.929199 punch from 0x0225b5 to 0x02328b (0xcd7 bytes) 000238 1788524102.951015 trunc from 0x01a221 to 0x025549 (0xb329 bytes) 000239 1788524102.960992 punch from 0x00eadd to 0x011161 (0x2685 bytes) 000240 1788524102.977124 trunc from 0x02554a to 0x0153f7 (0x10152 bytes) 000241 1788524103.028235 mapread 0x00b00a thru 0x0153f7 (0xa3ee bytes) 000242 1788524103.113377 read 0x0129f2 thru 0x0153f7 (0x2a06 bytes) 000243 1788524103.114052 mapread 0x00d33e thru 0x01258d (0x5250 bytes) 000245 1788524103.116459 fallocate 0x005c89 thru 0x00a076 (0x43ee bytes) 000248 1788524103.123424 read 0x00c6fb thru 0x0141ff (0x7b05 bytes) 000249 1788524103.124453 read 0x0109b3 thru 0x0153f7 (0x4a45 bytes) 000251 1788524103.125904 fallocate 0x00de14 thru 0x015052 (0x723f bytes) 000252 1788524103.130026 write 0x00289f thru 0x005ef3 (0x3655 bytes) 000253 1788524103.132082 read 0x005b17 thru 0x00f283 (0x976d bytes) 000254 1788524103.132944 read 0x00aec8 thru 0x012b02 (0x7c3b bytes) 000256 1788524103.134436 punch from 0x002fab to 0x011a55 (0xeaab bytes) 000257 1788524103.147188 trunc from 0x0153f8 to 0x0085b8 (0xce3f bytes) 000258 1788524103.164090 punch from 0x007bf4 to 0x0085b8 (0x9c5 bytes) 000259 1788524103.168226 write 0x0337a7 thru 0x0399fa (0x6254 bytes) 000261 1788524103.170415 write 0x006488 thru 0x009848 (0x33c1 bytes) 000262 1788524103.180892 write 0x030cbd thru 0x03c160 (0xb4a4 bytes) 000263 1788524103.187265 MIRROR_EXTEND to 2 mirrors 000264 1788524104.061524 mapwrite 0x019fb0 thru 0x01c9e3 (0x2a34 bytes) 000265 1788524104.156946 trunc from 0x03c161 to 0x026cc1 (0x1549f bytes) 000266 1788524104.222231 trunc from 0x026cc2 to 0x0074b9 (0x1f808 bytes) 000267 1788524104.260915 read 0x000f54 thru 0x001666 (0x713 bytes) 000268 1788524104.268945 trunc from 0x00587c to 0x0074b9 (0x1c3e bytes) 000269 1788524104.278604 read 0x006e69 thru 0x0074b9 (0x651 bytes) 000270 1788524104.298521 write 0x03f198 thru 0x03ffff (0xe68 bytes) 000271 1788524104.300601 write 0x010542 thru 0x01b986 (0xb445 bytes) 000273 1788524104.345228 trunc from 0x040000 to 0x01f66c (0x20993 bytes) 000274 1788524104.392110 fallocate 0x00e666 thru 0x019ef0 (0xb88b bytes) 000275 1788524104.408781 trunc from 0x01f66d to 0x014621 (0xb04b bytes) 000276 1788524104.450765 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x14622): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14622): Mirror 2: 0xc42fc5a9 (stale) CRC-32 checksum value for chunk [0, 0x14622): Mirror 1: 0xf6415269 Mirror 3: 0xf6415269 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14622: skipped 000278 1788524105.442767 mapread 0x00bd10 thru 0x014621 (0x8912 bytes) 000279 1788524105.472445 mapwrite 0x036183 thru 0x03ffff (0x9e7d bytes) 000280 1788524105.626645 fallocate 0x01ae5f thru 0x02294a (0x7aec bytes) 000281 1788524105.631921 trunc from 0x0122a2 to 0x01e8f6 (0xc655 bytes) 000282 1788524105.639542 read 0x0138b2 thru 0x01a07f (0x67ce bytes) 000284 1788524105.650626 trunc from 0x013ce9 to 0x021c2f (0xdf47 bytes) 000285 1788524105.657276 trunc from 0x040000 to 0x0152aa (0x2ad55 bytes) 000286 1788524105.681146 mapread 0x0124fd thru 0x0152aa (0x2dae bytes) 000287 1788524105.691159 mapwrite 0x01b65c thru 0x01b7e2 (0x187 bytes) 000288 1788524105.739301 mapwrite 0x02189e thru 0x02cdae (0xb511 bytes) 000289 1788524105.797991 mapwrite 0x02a25e thru 0x02fb9f (0x5942 bytes) 000290 1788524105.833635 punch from 0x02a1f2 to 0x02cbb0 (0x29bf bytes) 000292 1788524105.842508 MIRROR_RESYNC 3 mirrors 000293 1788524106.807246 punch from 0x01c565 to 0x0211df (0x4c7b bytes) 000294 1788524106.859730 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fba0): Mirror 3: 0xd4939c19 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fba0): Mirror 3: 0xd4939c19 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fba0: skipped 000295 1788524107.077623 fallocate 0x022659 thru 0x0314c5 (0xee6d bytes) 000296 1788524107.081970 write 0x03dd9b thru 0x03ffff (0x2265 bytes) 000298 1788524107.092034 MIRROR_RESYNC 2 mirrors 000299 1788524107.898655 trunc from 0x03907e to 0x03ceea (0x3e6d bytes) 000300 1788524107.939623 mapread 0x03e268 thru 0x03ffff (0x1d98 bytes) 000301 1788524107.953541 write 0x00fc2b thru 0x01faf2 (0xfec8 bytes) 000302 1788524107.973178 trunc from 0x0269be to 0x0295a1 (0x2be4 bytes) 000303 1788524107.985008 mapread 0x02d9b7 thru 0x02ef67 (0x15b1 bytes) 000304 1788524107.993751 mapwrite 0x02e9a8 thru 0x034a43 (0x609c bytes) 000305 1788524108.011501 mapread 0x004e79 thru 0x00e2d6 (0x945e bytes) 000306 1788524108.025018 fallocate 0x03d62e thru 0x03ffff (0x29d2 bytes) 000308 1788524108.030253 write 0x02cecb thru 0x03aac3 (0xdbf9 bytes) 000309 1788524108.035283 read 0x02c220 thru 0x03bc9d (0xfa7e bytes) 000310 1788524108.039888 mapread 0x02721f thru 0x034e55 (0xdc37 bytes) 000311 1788524108.041269 mapwrite 0x03a9a3 thru 0x03ffff (0x565d bytes) 000312 1788524108.059591 mapwrite 0x02065f thru 0x023619 (0x2fbb bytes) 000313 1788524108.078445 trunc from 0x02b93f to 0x03872e (0xcdf0 bytes) 000314 1788524108.082539 trunc from 0x023a54 to 0x028559 (0x4b06 bytes) 000315 1788524108.086977 mapread 0x035b51 thru 0x03c1ec (0x669c bytes) 000316 1788524108.094366 mapwrite 0x01ab50 thru 0x0284bd (0xd96e bytes) 000317 1788524108.119442 read 0x03c723 thru 0x03ffff (0x38dd bytes) 000318 1788524108.120438 mapread 0x001503 thru 0x004728 (0x3226 bytes) 000319 1788524108.121533 read 0x024c2d thru 0x028421 (0x37f5 bytes) 000320 1788524108.122160 write 0x039316 thru 0x03ffff (0x6cea bytes) 000321 1788524108.123735 mapread 0x0263f1 thru 0x028be5 (0x27f5 bytes) 000322 1788524108.125279 write 0x028bab thru 0x03895e (0xfdb4 bytes) 000323 1788524108.127407 write 0x0117d0 thru 0x012e4f (0x1680 bytes) 000324 1788524108.132333 MIRROR_RESYNC 2 mirrors 000326 1788524108.663615 read 0x0233fb thru 0x02ef67 (0xbb6d bytes) 000327 1788524108.671932 mapwrite 0x01d54a thru 0x02c2cc (0xed83 bytes) 000328 1788524108.779319 fallocate 0x00ef07 thru 0x00f2ac (0x3a6 bytes) 000329 1788524108.797879 mapread 0x03c323 thru 0x03e654 (0x2332 bytes) 000330 1788524108.809541 mapwrite 0x010fee thru 0x01eab7 (0xdaca bytes) 000331 1788524108.841207 trunc from 0x040000 to 0x02bb7b (0x14484 bytes) 000332 1788524108.882903 read 0x00aaf8 thru 0x018a8c (0xdf95 bytes) 000333 1788524108.886888 MIRROR_RESYNC 2 mirrors 000334 1788524109.617049 fallocate 0x006100 thru 0x011f8e (0xbe8f bytes) 000335 1788524109.652869 mapread 0x010255 thru 0x01ef3c (0xece8 bytes) 000336 1788524109.699281 trunc from 0x009e79 to 0x00e6ea (0x4872 bytes) 000337 1788524109.704561 mapread 0x00c499 thru 0x0125a3 (0x610b bytes) 000338 1788524109.715857 write 0x036d2a thru 0x03ffff (0x92d6 bytes) 000339 1788524109.720835 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1bf:0x0]: Invalid argument (22) 000340 1788524109.863145 read 0x013a6b thru 0x01f061 (0xb5f7 bytes) 000344 1788524109.881274 mapread 0x01944c thru 0x01f74a (0x62ff bytes) 000345 1788524109.883418 read 0x00b77a thru 0x0181da (0xca61 bytes) 000347 1788524109.894391 read 0x03b0d4 thru 0x03bbc3 (0xaf0 bytes) 000348 1788524109.894960 read 0x008fc1 thru 0x00d2af (0x42ef bytes) 000349 1788524109.895482 write 0x017566 thru 0x0271b1 (0xfc4c bytes) 000351 1788524109.904222 read 0x011bd7 thru 0x018460 (0x688a bytes) 000352 1788524109.905001 trunc from 0x040000 to 0x012da3 (0x2d25c bytes) 000353 1788524109.922090 punch from 0x002db9 to 0x005a6d (0x2cb5 bytes) 000354 1788524109.927894 trunc from 0x0053b1 to 0x0097bb (0x440b bytes) 000356 1788524109.941188 trunc from 0x012da4 to 0x0326b3 (0x1f910 bytes) 000357 1788524109.959604 mapwrite 0x03af3c thru 0x03ffff (0x50c4 bytes) 000358 1788524110.014693 mapwrite 0x02437c thru 0x02c157 (0x7ddc bytes) 000359 1788524110.039322 mapwrite 0x019966 thru 0x01c44f (0x2aea bytes) 000361 1788524110.049787 write 0x002cf3 thru 0x00c97a (0x9c88 bytes) 000362 1788524110.051821 write 0x021265 thru 0x0231d4 (0x1f70 bytes) 000363 1788524110.053427 read 0x01ff6d thru 0x0283ba (0x844e bytes) 000364 1788524110.054432 punch from 0x033218 to 0x03e2ae (0xb097 bytes) 000365 1788524110.065091 MIRROR_EXTEND to 2 mirrors 000366 1788524110.706510 punch from 0x007e04 to 0x016e1f (0xf01c bytes) 000367 1788524110.738764 write 0x034a89 thru 0x03b621 (0x6b99 bytes) 000368 1788524110.753388 read 0x001558 thru 0x007822 (0x62cb bytes) 000369 1788524110.775521 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1bf:0x0]: Invalid argument (22) 000370 1788524110.903158 write 0x01d5c0 thru 0x02a3c2 (0xce03 bytes) 000371 1788524110.916658 fallocate 0x026aa5 thru 0x02b376 (0x48d2 bytes) 000372 1788524110.923690 punch from 0x023eba to 0x026a36 (0x2b7d bytes) 000373 1788524110.934514 mapwrite 0x031672 thru 0x036285 (0x4c14 bytes) 000375 1788524110.976253 read 0x002652 thru 0x00422d (0x1bdc bytes) 000376 1788524110.980551 write 0x00126a thru 0x006eb0 (0x5c47 bytes) 000378 1788524110.990363 write 0x01cdd6 thru 0x021d7a (0x4fa5 bytes) 000379 1788524110.995613 read 0x00cd46 thru 0x01bc82 (0xef3d bytes) 000380 1788524111.004572 trunc from 0x040000 to 0x000a13 (0x3f5ec bytes) 000381 1788524111.052042 fallocate 0x03d086 thru 0x03ffff (0x2f7a bytes) 000382 1788524111.057787 mapread 0x000a0b thru 0x000a13 (0x9 bytes) 000383 1788524111.058848 trunc from 0x000092 to 0x000a13 (0x982 bytes) 000384 1788524111.063405 fallocate 0x011385 thru 0x01f1bc (0xde38 bytes) 000386 1788524111.068815 fallocate 0x03e030 thru 0x03ffff (0x1fd0 bytes) 000387 1788524111.074776 mapread 0x00043d thru 0x0005a6 (0x16a bytes) 000390 1788524111.081542 trunc from 0x000a14 to 0x02f0e6 (0x2e6d3 bytes) 000391 1788524111.092341 trunc from 0x02ebb6 to 0x02f0e6 (0x531 bytes) 000393 1788524111.098506 trunc from 0x029ff2 to 0x02f0e6 (0x50f5 bytes) 000394 1788524111.102623 read 0x00fabe thru 0x015679 (0x5bbc bytes) 000396 1788524111.113553 fallocate 0x03acba thru 0x03ffff (0x5346 bytes) 000397 1788524111.118681 fallocate 0x00ce15 thru 0x014541 (0x772d bytes) 000399 1788524111.125643 mapread 0x01048e thru 0x01cf92 (0xcb05 bytes) 000400 1788524111.127471 trunc from 0x040000 to 0x01b1b4 (0x24e4b bytes) 000401 1788524111.139515 punch from 0x01487e to 0x0185c1 (0x3d44 bytes) 000402 1788524111.142961 mapread 0x014d52 thru 0x01b1b4 (0x6463 bytes) 000403 1788524111.163739 trunc from 0x01b1b5 to 0x03a24c (0x1f098 bytes) 000404 1788524111.178547 trunc from 0x00830e to 0x0167ec (0xe4df bytes) 000405 1788524111.182476 punch from 0x01fa0a to 0x02a782 (0xad79 bytes) 000406 1788524111.186221 trunc from 0x03a24d to 0x002272 (0x37fda bytes) 000407 1788524111.202032 mapread 0x001035 thru 0x002272 (0x123e bytes) 000408 1788524111.203409 trunc from 0x000157 to 0x002272 (0x211c bytes) 000409 1788524111.210400 mapread 0x00067a thru 0x002272 (0x1bf9 bytes) 000410 1788524111.237982 trunc from 0x002273 to 0x027e7b (0x25c09 bytes) 000411 1788524111.254559 trunc from 0x027e7c to 0x01df70 (0x9f0b bytes) 000412 1788524111.281109 MIRROR_EXTEND to 2 mirrors 000413 1788524111.812447 trunc from 0x01df71 to 0x01e88f (0x91f bytes) 000414 1788524111.873889 punch from 0x00b7db to 0x01045c (0x4c82 bytes) 000415 1788524111.877332 mapwrite 0x03331b thru 0x033681 (0x367 bytes) 000416 1788524111.911521 trunc from 0x00c6f4 to 0x00e92f (0x223c bytes) 000417 1788524111.920082 read 0x014529 thru 0x018677 (0x414f bytes) 000418 1788524111.942118 trunc from 0x033682 to 0x01affa (0x18687 bytes) 000419 1788524111.967479 mapwrite 0x0220f4 thru 0x028929 (0x6836 bytes) 000420 1788524112.022235 trunc from 0x011c59 to 0x017969 (0x5d11 bytes) 000422 1788524112.030889 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2892a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2892a): Mirror 2: 0x3dcd2d5f (stale) CRC-32 checksum value for chunk [0, 0x2892a): Mirror 1: 0xb7430722 Mirror 3: 0xb7430722 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2892a: skipped 000423 1788524112.642233 mapread 0x01d052 thru 0x02400a (0x6fb9 bytes) 000424 1788524112.659127 MIRROR_RESYNC 3 mirrors 000425 1788524113.113284 read 0x011a16 thru 0x01af94 (0x957f bytes) 000426 1788524113.124872 mapwrite 0x03670b thru 0x039de0 (0x36d6 bytes) 000427 1788524113.222937 punch from 0x0100df to 0x01b2cd (0xb1ef bytes) 000428 1788524113.231188 write 0x00af01 thru 0x01301f (0x811f bytes) 000429 1788524113.259096 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39de1): Mirror 3: 0x7a2d2fe8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39de1): Mirror 3: 0x7a2d2fe8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39de1: skipped 000430 1788524113.614212 trunc from 0x039de1 to 0x00a798 (0x2f648 bytes) 000431 1788524113.643341 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xa799): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa799): Mirror 3: 0x97fe4850 (stale) CRC-32 checksum value for chunk [0, 0xa799): Mirror 1: 0x97fe4850 Mirror 4: 0x97fe4850 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa799: skipped 000432 1788524114.343203 mapwrite 0x02a6f4 thru 0x02cb86 (0x2493 bytes) 000434 1788524114.441128 trunc from 0x0011eb to 0x007df9 (0x6c0f bytes) 000436 1788524114.449022 write 0x022faa thru 0x03276e (0xf7c5 bytes) 000437 1788524114.471225 read 0x032152 thru 0x03276e (0x61d bytes) 000438 1788524114.471864 write 0x0168f8 thru 0x022258 (0xb961 bytes) 000439 1788524114.481374 mapread 0x0313ee thru 0x03276e (0x1381 bytes) 000440 1788524114.482268 trunc from 0x020503 to 0x022253 (0x1d51 bytes) 000441 1788524114.502546 mapread 0x02cbba thru 0x03276e (0x5bb5 bytes) 000442 1788524114.514532 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3276f): Mirror 3: 0x8d62167c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3276f): Mirror 3: 0x8d62167c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3276f: skipped 000443 1788524114.814083 write 0x01cc0d thru 0x02943f (0xc833 bytes) 000444 1788524114.838149 MIRROR_RESYNC 2 mirrors 000445 1788524115.372253 punch from 0x02f633 to 0x03276e (0x313c bytes) 000446 1788524115.436885 mapwrite 0x0186fe thru 0x024bc5 (0xc4c8 bytes) 000447 1788524115.496645 trunc from 0x01a61f to 0x026eca (0xc8ac bytes) 000448 1788524115.503612 mapread 0x014f53 thru 0x01af7c (0x602a bytes) 000449 1788524115.527812 punch from 0x0064f2 to 0x00e360 (0x7e6f bytes) 000450 1788524115.536394 mapread 0x0237d9 thru 0x028a1e (0x5246 bytes) 000452 1788524115.549806 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3276f): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3276f): Mirror 3: 0x8dd810 (stale) CRC-32 checksum value for chunk [0, 0x3276f): Mirror 1: 0xe2b998e1 Mirror 4: 0xe2b998e1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3276f: skipped 000455 1788524116.514427 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3276f): Mirror 3: 0x8dd810 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3276f): Mirror 3: 0x8dd810 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3276f: skipped 000456 1788524116.809518 mapwrite 0x00d330 thru 0x01ab5e (0xd82f bytes) 000457 1788524116.881956 MIRROR_RESYNC 2 mirrors 000458 1788524117.398414 trunc from 0x0258c6 to 0x0271ae (0x18e9 bytes) 000459 1788524117.432172 fallocate 0x01d674 thru 0x0255be (0x7f4b bytes) 000460 1788524117.446540 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1bf:0x0]: Invalid argument (22) 000462 1788524117.586666 mapread 0x024eef thru 0x025587 (0x699 bytes) 000463 1788524117.596335 fallocate 0x033963 thru 0x03ffff (0xc69d bytes) 000464 1788524117.609680 read 0x030d93 thru 0x03ef21 (0xe18f bytes) 000465 1788524117.617842 trunc from 0x015a3a to 0x01f1b1 (0x9778 bytes) 000466 1788524117.621610 write 0x039f9d thru 0x03ffff (0x6063 bytes) 000467 1788524117.626381 mapwrite 0x025735 thru 0x02e3e1 (0x8cad bytes) 000468 1788524117.652439 read 0x0071de thru 0x00b248 (0x406b bytes) 000469 1788524117.666209 punch from 0x02454b to 0x031e35 (0xd8eb bytes) 000470 1788524117.676776 fallocate 0x00105a thru 0x004f26 (0x3ecd bytes) 000472 1788524117.684410 write 0x03cfa9 thru 0x03d501 (0x559 bytes) 000473 1788524117.685533 write 0x001e61 thru 0x003716 (0x18b6 bytes) 000474 1788524117.687722 trunc from 0x040000 to 0x0377c1 (0x883e bytes) 000475 1788524117.708695 punch from 0x026c14 to 0x02ae8b (0x4278 bytes) 000476 1788524117.714494 read 0x014336 thru 0x018442 (0x410d bytes) 000478 1788524117.715971 punch from 0x001a86 to 0x00e16a (0xc6e5 bytes) 000479 1788524117.728436 punch from 0x008ef4 to 0x01650c (0xd619 bytes) 000480 1788524117.733123 mapwrite 0x02f43c thru 0x031939 (0x24fe bytes) 000482 1788524117.754002 mapwrite 0x010150 thru 0x01116e (0x101f bytes) 000484 1788524117.770667 mapwrite 0x01d32b thru 0x022e93 (0x5b69 bytes) 000486 1788524117.789426 read 0x0058b4 thru 0x012f0c (0xd659 bytes) 000487 1788524117.790319 trunc from 0x0377c2 to 0x027c7d (0xfb44 bytes) 000488 1788524117.808298 mapwrite 0x03ca17 thru 0x03ffff (0x35e9 bytes) 000490 1788524117.856982 trunc from 0x03ae61 to 0x03f09c (0x423c bytes) 000492 1788524117.862462 mapwrite 0x00a632 thru 0x017c70 (0xd63f bytes) 000493 1788524117.889039 write 0x0010ee thru 0x005953 (0x4866 bytes) 000494 1788524117.890307 fallocate 0x027fbd thru 0x033353 (0xb397 bytes) 000495 1788524117.894863 write 0x02e3de thru 0x03cdc6 (0xe9e9 bytes) 000496 1788524117.905141 write 0x03935d thru 0x03e719 (0x53bd bytes) 000498 1788524117.912510 read 0x01d727 thru 0x01fdb7 (0x2691 bytes) 000500 1788524117.913624 read 0x00346c thru 0x012947 (0xf4dc bytes) 000501 1788524117.915023 punch from 0x00f83b to 0x01f014 (0xf7da bytes) 000502 1788524117.919429 punch from 0x01234f to 0x01bf11 (0x9bc3 bytes) 000503 1788524117.923106 fallocate 0x0384ee thru 0x03a8c9 (0x23dc bytes) 000504 1788524117.927197 mapwrite 0x00d38c thru 0x01473e (0x73b3 bytes) 000505 1788524117.956615 MIRROR_EXTEND to 2 mirrors 000506 1788524118.684032 fallocate 0x002314 thru 0x00f9f8 (0xd6e5 bytes) 000507 1788524118.731263 punch from 0x033c43 to 0x03ffff (0xc3bd bytes) 000508 1788524118.754165 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x2f8253a2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd6f214ad Mirror 3: 0xd6f214ad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000509 1788524119.613733 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x2f8253a2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd6f214ad Mirror 3: 0xd6f214ad Mirror 4: 0xd6f214ad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000510 1788524120.460009 read 0x02b1ea thru 0x0316eb (0x6502 bytes) 000511 1788524120.468521 mapread 0x03afe2 thru 0x03ffff (0x501e bytes) 000512 1788524120.503308 MIRROR_RESYNC 4 mirrors 000513 1788524121.287213 MIRROR_RESYNC 4 mirrors 000514 1788524121.641637 read 0x029123 thru 0x02d1ab (0x4089 bytes) 000517 1788524121.649238 fallocate 0x00f0ee thru 0x0105c0 (0x14d3 bytes) 000518 1788524121.697269 write 0x028597 thru 0x032d24 (0xa78e bytes) 000519 1788524121.708866 trunc from 0x00dfeb to 0x01a927 (0xc93d bytes) 000520 1788524121.713732 mapread 0x01bc5c thru 0x01c2b0 (0x655 bytes) 000521 1788524121.722247 read 0x00eb7e thru 0x011e52 (0x32d5 bytes) 000522 1788524121.736034 fallocate 0x03b433 thru 0x03ffff (0x4bcd bytes) 000523 1788524121.743764 mapread 0x03e87e thru 0x03ffff (0x1782 bytes) 000524 1788524121.746578 fallocate 0x0236ee thru 0x026c49 (0x355c bytes) 000525 1788524121.766591 MIRROR_RESYNC 4 mirrors 000526 1788524122.977764 read 0x007153 thru 0x00b0fa (0x3fa8 bytes) 000527 1788524123.004648 write 0x021a61 thru 0x02a413 (0x89b3 bytes) 000528 1788524123.044620 MIRROR_RESYNC 4 mirrors 000531 1788524124.392563 fallocate 0x0317e9 thru 0x03ffff (0xe817 bytes) 000532 1788524124.471077 read 0x023a7c thru 0x02994a (0x5ecf bytes) 000533 1788524124.485529 write 0x03d2d8 thru 0x03ffff (0x2d28 bytes) 000536 1788524124.494472 read 0x031f65 thru 0x035f50 (0x3fec bytes) 000537 1788524124.508103 fallocate 0x013f5c thru 0x0203b0 (0xc455 bytes) 000538 1788524124.518536 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe400bbef (stale) Mirror 3: 0xe400bbef (stale) Mirror 4: 0xe400bbef (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1c75f749 Mirror 5: 0x1c75f749 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000539 1788524125.494671 punch from 0x002d81 to 0x009441 (0x66c1 bytes) 000540 1788524125.534674 trunc from 0x040000 to 0x00a5af (0x35a50 bytes) 000541 1788524125.577875 trunc from 0x003606 to 0x00a5af (0x6faa bytes) 000542 1788524125.595082 trunc from 0x00a5b0 to 0x03cf0a (0x3295b bytes) 000543 1788524125.611404 fallocate 0x039b3b thru 0x03ffff (0x64c5 bytes) 000545 1788524125.620311 trunc from 0x03ce1f to 0x03ffff (0x31e1 bytes) 000546 1788524125.625601 trunc from 0x040000 to 0x02442d (0x1bbd2 bytes) 000548 1788524125.644684 read 0x0162d2 thru 0x01d849 (0x7578 bytes) 000549 1788524125.660498 MIRROR_RESYNC 5 mirrors 000550 1788524126.635048 punch from 0x01f142 to 0x02442d (0x52ec bytes) 000551 1788524126.663500 read 0x00dd00 thru 0x01bbe1 (0xdee2 bytes) 000553 1788524126.675558 trunc from 0x02442e to 0x02947a (0x504d bytes) 000554 1788524126.702115 mapread 0x0081ca thru 0x0150f3 (0xcf2a bytes) 000555 1788524126.719035 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2947b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2947b): Mirror 2: 0xaad9eb7e (stale) Mirror 3: 0xaad9eb7e (stale) Mirror 4: 0xaad9eb7e (stale) Mirror 5: 0xaad9eb7e (stale) CRC-32 checksum value for chunk [0, 0x2947b): Mirror 1: 0xaad9eb7e Mirror 6: 0xaad9eb7e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2947b: skipped 000556 1788524127.557115 trunc from 0x02947b to 0x02e96e (0x54f4 bytes) 000557 1788524127.627451 fallocate 0x00ab03 thru 0x0162a3 (0xb7a1 bytes) 000558 1788524127.635548 trunc from 0x00cb23 to 0x0108bd (0x3d9b bytes) 000559 1788524127.643101 trunc from 0x02e96f to 0x03f2d2 (0x10964 bytes) 000561 1788524127.661925 trunc from 0x001375 to 0x00b474 (0xa100 bytes) 000562 1788524127.671895 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3f2d3): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f2d3): Mirror 2: 0xbd282ea3 (stale) Mirror 3: 0xbd282ea3 (stale) Mirror 4: 0xbd282ea3 (stale) Mirror 5: 0xbd282ea3 (stale) Mirror 6: 0xbd282ea3 (stale) CRC-32 checksum value for chunk [0, 0x3f2d3): Mirror 1: 0x3119b9b2 Mirror 7: 0x3119b9b2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f2d3: skipped 000563 1788524128.438125 punch from 0x019283 to 0x028495 (0xf213 bytes) 000564 1788524128.472821 mapwrite 0x008cb2 thru 0x00f35b (0x66aa bytes) 000565 1788524128.515688 mapread 0x02145d thru 0x029c78 (0x881c bytes) 000567 1788524128.532583 fallocate 0x009267 thru 0x0156fa (0xc494 bytes) 000568 1788524128.536670 write 0x0115e2 thru 0x0199f1 (0x8410 bytes) 000569 1788524128.544652 mapread 0x0141db thru 0x01428a (0xb0 bytes) 000570 1788524128.545968 punch from 0x00bec5 to 0x017021 (0xb15d bytes) 000571 1788524128.561825 mapread 0x02156f thru 0x02a60c (0x909e bytes) 000573 1788524128.572025 read 0x029bfc thru 0x036c41 (0xd046 bytes) 000574 1788524128.580312 trunc from 0x016c13 to 0x01ff56 (0x9344 bytes) 000575 1788524128.584516 trunc from 0x03f2d3 to 0x006e07 (0x384cb bytes) 000576 1788524128.607493 fallocate 0x038657 thru 0x03ca2a (0x43d4 bytes) 000577 1788524128.614592 punch from 0x006aba to 0x006e07 (0x34e bytes) 000579 1788524128.620099 trunc from 0x005ff2 to 0x006e07 (0xe16 bytes) 000580 1788524128.633672 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x6e08): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6e08): Mirror 2: 0xd3a56ca8 (stale) Mirror 3: 0xd3a56ca8 (stale) Mirror 4: 0xd3a56ca8 (stale) Mirror 5: 0xd3a56ca8 (stale) Mirror 6: 0xd3a56ca8 (stale) Mirror 7: 0xf4baa2b8 (stale) CRC-32 checksum value for chunk [0, 0x6e08): Mirror 1: 0xf4baa2b8 Mirror 8: 0xf4baa2b8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6e08: skipped 000582 1788524129.380063 fallocate 0x01b27a thru 0x0252fc (0xa083 bytes) 000583 1788524129.424999 mapread 0x01012d thru 0x0200dc (0xffb0 bytes) 000584 1788524129.468009 trunc from 0x0252fd to 0x036696 (0x1139a bytes) 000585 1788524129.507153 fallocate 0x032902 thru 0x038cbb (0x63ba bytes) 000586 1788524129.515813 mapwrite 0x02bdde thru 0x0374fe (0xb721 bytes) 000587 1788524129.575282 mapwrite 0x022cd3 thru 0x02d717 (0xaa45 bytes) 000588 1788524129.600351 punch from 0x016455 to 0x01a47d (0x4029 bytes) 000589 1788524129.604857 read 0x02d4da thru 0x038cbb (0xb7e2 bytes) 000590 1788524129.605689 mapwrite 0x036497 thru 0x038ca3 (0x280d bytes) 000591 1788524129.620386 fallocate 0x00aba9 thru 0x010cae (0x6106 bytes) 000592 1788524129.624844 fallocate 0x000c51 thru 0x0066ad (0x5a5d bytes) 000593 1788524129.629066 punch from 0x00de14 to 0x012b12 (0x4cff bytes) 000594 1788524129.633380 fallocate 0x029d08 thru 0x038cf9 (0xeff2 bytes) 000595 1788524129.636974 write 0x01103b thru 0x017208 (0x61ce bytes) 000596 1788524129.645380 trunc from 0x038cbc to 0x0164d5 (0x227e6 bytes) 000597 1788524129.666325 punch from 0x00dc0a to 0x0164d5 (0x88cc bytes) 000598 1788524129.696580 trunc from 0x0164d6 to 0x007e0d (0xe6c8 bytes) 000599 1788524129.719098 read 0x004380 thru 0x007e0d (0x3a8e bytes) 000600 1788524129.731330 trunc from 0x007e0e to 0x00ddaf (0x5fa2 bytes) 000601 1788524129.747889 trunc from 0x00ddb0 to 0x01990f (0xbb60 bytes) 000603 1788524129.763247 trunc from 0x019910 to 0x0250fe (0xb7ef bytes) 000604 1788524129.778031 mapwrite 0x029a14 thru 0x038479 (0xea66 bytes) 000605 1788524129.823201 trunc from 0x02dfcc to 0x038479 (0xa4ae bytes) 000606 1788524129.832561 mapread 0x0298ff thru 0x037b28 (0xe22a bytes) 000607 1788524129.906479 mapread 0x029ba9 thru 0x038479 (0xe8d1 bytes) 000608 1788524129.915692 trunc from 0x011293 to 0x0120c4 (0xe32 bytes) 000609 1788524129.922665 read 0x0106c3 thru 0x01f53f (0xee7d bytes) 000610 1788524129.932669 mapwrite 0x00060e thru 0x00a891 (0xa284 bytes) 000611 1788524129.957378 punch from 0x01e106 to 0x0283cb (0xa2c6 bytes) 000612 1788524129.961623 mapwrite 0x02b8b8 thru 0x02cd9b (0x14e4 bytes) 000613 1788524129.972503 trunc from 0x03847a to 0x0385b3 (0x13a bytes) 000614 1788524129.988645 punch from 0x01e344 to 0x027747 (0x9404 bytes) 000615 1788524129.994238 mapwrite 0x004550 thru 0x006451 (0x1f02 bytes) 000616 1788524130.005195 read 0x006cba thru 0x00e05d (0x73a4 bytes) 000617 1788524130.006054 write 0x01a198 thru 0x0246c4 (0xa52d bytes) 000619 1788524130.012345 write 0x008508 thru 0x00fc80 (0x7779 bytes) 000620 1788524130.013618 mapwrite 0x025cdd thru 0x02f613 (0x9937 bytes) 000621 1788524130.042574 read 0x01ed19 thru 0x020ca9 (0x1f91 bytes) 000622 1788524130.043295 punch from 0x020b77 to 0x022c32 (0x20bc bytes) 000623 1788524130.053077 MIRROR_RESYNC 8 mirrors 000624 1788524132.046009 trunc from 0x0385b4 to 0x019581 (0x1f032 bytes) 000625 1788524132.081022 mapwrite 0x019e5d thru 0x023bec (0x9d90 bytes) 000626 1788524132.148542 mapread 0x014cd8 thru 0x023bec (0xef15 bytes) 000627 1788524132.151019 punch from 0x016279 to 0x023bec (0xd974 bytes) 000628 1788524132.159519 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23bed): Mirror 2: 0x964cb14 (stale) Mirror 3: 0x964cb14 (stale) Mirror 5: 0x964cb14 (stale) Mirror 6: 0x964cb14 (stale) Mirror 7: 0x964cb14 (stale) Mirror 8: 0x964cb14 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23bed): Mirror 2: 0x964cb14 (stale) Mirror 3: 0x964cb14 (stale) Mirror 5: 0x964cb14 (stale) Mirror 6: 0x964cb14 (stale) Mirror 7: 0x964cb14 (stale) Mirror 8: 0x964cb14 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23bed: skipped 000629 1788524132.676727 write 0x039087 thru 0x03ef70 (0x5eea bytes) 000630 1788524132.684521 read 0x01dab4 thru 0x029c9f (0xc1ec bytes) 000631 1788524132.693508 punch from 0x0291fe to 0x029460 (0x263 bytes) 000632 1788524132.699350 trunc from 0x00a27e to 0x017733 (0xd4b6 bytes) 000633 1788524132.709005 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ef71): Mirror 2: 0x3facbe96 (stale) Mirror 5: 0x3facbe96 (stale) Mirror 6: 0x3facbe96 (stale) Mirror 7: 0x3facbe96 (stale) Mirror 8: 0x3facbe96 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ef71): Mirror 2: 0x3facbe96 (stale) Mirror 5: 0x3facbe96 (stale) Mirror 6: 0x3facbe96 (stale) Mirror 7: 0x3facbe96 (stale) Mirror 8: 0x3facbe96 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ef71: skipped 000634 1788524133.129970 write 0x00f9cb thru 0x01de72 (0xe4a8 bytes) 000635 1788524133.165537 mapwrite 0x03da3b thru 0x03ffff (0x25c5 bytes) 000636 1788524133.224008 read 0x019ade thru 0x01a03c (0x55f bytes) 000638 1788524133.226504 mapread 0x024aa8 thru 0x027372 (0x28cb bytes) 000639 1788524133.244525 write 0x03145b thru 0x03ffff (0xeba5 bytes) 000641 1788524133.254437 mapwrite 0x01076e thru 0x0173cb (0x6c5e bytes) 000642 1788524133.276271 punch from 0x009521 to 0x01487c (0xb35c bytes) 000643 1788524133.283443 punch from 0x0285ad to 0x034126 (0xbb7a bytes) 000644 1788524133.287518 fallocate 0x017307 thru 0x0253f8 (0xe0f2 bytes) 000645 1788524133.299362 fallocate 0x02f32f thru 0x03c85f (0xd531 bytes) 000646 1788524133.306260 fallocate 0x01e5cd thru 0x021ac8 (0x34fc bytes) 000647 1788524133.310084 trunc from 0x040000 to 0x029456 (0x16ba9 bytes) 000649 1788524133.323443 fallocate 0x024be8 thru 0x030307 (0xb720 bytes) 000650 1788524133.329975 punch from 0x029f15 to 0x030307 (0x63f3 bytes) 000651 1788524133.333871 trunc from 0x030308 to 0x0011fd (0x2f10a bytes) 000652 1788524133.350645 mapwrite 0x037f2b thru 0x03e7c3 (0x6899 bytes) 000653 1788524133.405496 write 0x00f118 thru 0x01c4ca (0xd3b3 bytes) 000654 1788524133.419963 mapwrite 0x02421c thru 0x03035d (0xc142 bytes) 000655 1788524133.451178 trunc from 0x004cc8 to 0x00bbed (0x6f26 bytes) 000656 1788524133.460436 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3e7c4): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e7c4): Mirror 2: 0xf7eeb8b8 (stale) Mirror 5: 0xf7eeb8b8 (stale) Mirror 6: 0xf7eeb8b8 (stale) Mirror 7: 0xf7eeb8b8 (stale) Mirror 8: 0xf7eeb8b8 (stale) CRC-32 checksum value for chunk [0, 0x3e7c4): Mirror 1: 0x885b0d24 Mirror 9: 0x885b0d24 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e7c4: skipped 000657 1788524134.343247 fallocate 0x0018f8 thru 0x00cf8c (0xb695 bytes) 000658 1788524134.382116 write 0x03b975 thru 0x03ffff (0x468b bytes) 000659 1788524134.390209 mapread 0x01b126 thru 0x01f9ac (0x4887 bytes) 000660 1788524134.403496 trunc from 0x040000 to 0x02250b (0x1daf4 bytes) 000661 1788524134.442200 write 0x02c4f1 thru 0x02f658 (0x3168 bytes) 000662 1788524134.444554 fallocate 0x02f915 thru 0x0387b3 (0x8e9f bytes) 000663 1788524134.453034 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2f659): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f659): Mirror 2: 0x64909b53 (stale) Mirror 5: 0x64909b53 (stale) Mirror 6: 0x64909b53 (stale) Mirror 7: 0x64909b53 (stale) Mirror 8: 0x64909b53 (stale) Mirror 9: 0x83d70a10 (stale) CRC-32 checksum value for chunk [0, 0x2f659): Mirror 1: 0xcb2d6e46 Mirror 10: 0xcb2d6e46 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f659: skipped 000665 1788524135.436037 mapread 0x00de1a thru 0x014fe3 (0x71ca bytes) 000666 1788524135.471307 write 0x0141e3 thru 0x01d5b6 (0x93d4 bytes) 000667 1788524135.568415 mapread 0x017c9d thru 0x024704 (0xca68 bytes) 000668 1788524135.595457 mapread 0x005988 thru 0x00f499 (0x9b12 bytes) 000669 1788524135.615636 mapwrite 0x02643e thru 0x027bf3 (0x17b6 bytes) 000670 1788524135.649375 punch from 0x006b75 to 0x00850d (0x1999 bytes) 000671 1788524135.657439 punch from 0x0217d8 to 0x029dea (0x8613 bytes) 000672 1788524135.671580 trunc from 0x02f659 to 0x03369d (0x4045 bytes) 000673 1788524135.718668 fallocate 0x019478 thru 0x024c90 (0xb819 bytes) 000674 1788524135.729735 read 0x00dcff thru 0x0177f3 (0x9af5 bytes) 000675 1788524135.738541 trunc from 0x03369e to 0x002358 (0x31345 bytes) 000676 1788524135.766299 write 0x0369a4 thru 0x03ffff (0x965c bytes) 000677 1788524135.770488 mapread 0x0190d3 thru 0x01edba (0x5ce8 bytes) 000678 1788524135.780802 read 0x02d1ea thru 0x02fc43 (0x2a5a bytes) 000679 1788524135.793878 trunc from 0x040000 to 0x01cba8 (0x23457 bytes) 000680 1788524135.840194 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1cba9): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cba9): Mirror 2: 0x53131969 (stale) Mirror 5: 0x53131969 (stale) Mirror 6: 0x53131969 (stale) Mirror 7: 0x53131969 (stale) Mirror 8: 0x53131969 (stale) Mirror 9: 0x4fe6f0f0 (stale) Mirror 10: 0x4fe6f0f0 (stale) CRC-32 checksum value for chunk [0, 0x1cba9): Mirror 1: 0xe83d0b3e Mirror 11: 0xe83d0b3e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cba9: skipped 000681 1788524136.922001 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1cba9): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cba9): Mirror 2: 0x53131969 (stale) Mirror 5: 0x53131969 (stale) Mirror 6: 0x53131969 (stale) Mirror 7: 0x53131969 (stale) Mirror 8: 0x53131969 (stale) Mirror 9: 0x4fe6f0f0 (stale) Mirror 10: 0x4fe6f0f0 (stale) CRC-32 checksum value for chunk [0, 0x1cba9): Mirror 1: 0xe83d0b3e Mirror 11: 0xe83d0b3e Mirror 12: 0xe83d0b3e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cba9: skipped 000682 1788524137.704960 punch from 0x000ecd to 0x0095f4 (0x8728 bytes) 000683 1788524137.755038 fallocate 0x039499 thru 0x0398cc (0x434 bytes) 000684 1788524137.760499 read 0x018a56 thru 0x0284fb (0xfaa6 bytes) 000685 1788524137.769866 mapread 0x017371 thru 0x022afd (0xb78d bytes) 000686 1788524137.774619 mapwrite 0x0031c9 thru 0x012d3c (0xfb74 bytes) 000687 1788524137.819532 trunc from 0x0398cd to 0x03ec59 (0x538d bytes) 000689 1788524137.842522 mapwrite 0x01d7f7 thru 0x02bd0a (0xe514 bytes) 000690 1788524137.870028 write 0x00df1d thru 0x017e13 (0x9ef7 bytes) 000691 1788524137.872390 trunc from 0x03ec5a to 0x01bcdf (0x22f7a bytes) 000692 1788524137.890502 write 0x01458f thru 0x0227c9 (0xe23b bytes) 000693 1788524137.893653 trunc from 0x01266c to 0x017b5f (0x54f4 bytes) 000694 1788524137.912454 mapwrite 0x027aff thru 0x02d3a8 (0x58aa bytes) 000695 1788524137.963877 MIRROR_RESYNC 10 mirrors 000696 1788524140.206965 fallocate 0x031b85 thru 0x03dc92 (0xc10e bytes) 000697 1788524140.246201 trunc from 0x03dc93 to 0x03db44 (0x14e bytes) 000698 1788524140.270225 trunc from 0x03db45 to 0x02d4aa (0x1069a bytes) 000699 1788524140.286446 trunc from 0x02d4ab to 0x038d9b (0xb8f1 bytes) 000701 1788524140.307050 punch from 0x021441 to 0x02c948 (0xb508 bytes) 000702 1788524140.317234 trunc from 0x02e105 to 0x030c7b (0x2b77 bytes) 000703 1788524140.321201 punch from 0x01a0ab to 0x028aac (0xea02 bytes) 000704 1788524140.325213 mapwrite 0x006f80 thru 0x00fe67 (0x8ee8 bytes) 000705 1788524140.353500 mapread 0x00f10a thru 0x01100c (0x1f03 bytes) 000706 1788524140.360292 read 0x0194e0 thru 0x01c128 (0x2c49 bytes) 000707 1788524140.367636 trunc from 0x004c6a to 0x00e340 (0x96d7 bytes) 000708 1788524140.371996 mapread 0x024779 thru 0x029b05 (0x538d bytes) 000709 1788524140.386068 write 0x03c6be thru 0x03ffff (0x3942 bytes) 000710 1788524140.387614 trunc from 0x040000 to 0x031769 (0xe896 bytes) 000711 1788524140.420242 mapwrite 0x00a103 thru 0x00b8e0 (0x17de bytes) 000712 1788524140.467517 trunc from 0x03176a to 0x022f7c (0xe7ed bytes) 000713 1788524140.488362 punch from 0x003181 to 0x011e1b (0xec9b bytes) 000714 1788524140.492599 mapwrite 0x01dd4e thru 0x027f58 (0xa20b bytes) 000715 1788524140.534415 punch from 0x0153c7 to 0x020754 (0xb38e bytes) 000716 1788524140.539773 fallocate 0x0251e5 thru 0x032d7a (0xdb96 bytes) 000717 1788524140.543580 mapread 0x020fc6 thru 0x027f58 (0x6f93 bytes) 000718 1788524140.551460 punch from 0x0194e8 to 0x022709 (0x9222 bytes) 000719 1788524140.558253 MIRROR_RESYNC 10 mirrors 000720 1788524143.158615 mapread 0x002058 thru 0x00c8f6 (0xa89f bytes) 000721 1788524143.178551 punch from 0x00c8f5 to 0x0160c6 (0x97d2 bytes) 000722 1788524143.210751 fallocate 0x013f05 thru 0x0191c5 (0x52c1 bytes) 000723 1788524143.215515 trunc from 0x027f59 to 0x008649 (0x1f90f bytes) 000724 1788524143.230966 mapwrite 0x017d43 thru 0x01ec0e (0x6ecc bytes) 000725 1788524143.270432 trunc from 0x01ec0f to 0x02b2ba (0xc6ac bytes) 000726 1788524143.287277 trunc from 0x0084f2 to 0x011bed (0x96fc bytes) 000727 1788524143.291611 write 0x036d0a thru 0x03899a (0x1c91 bytes) 000728 1788524143.293091 trunc from 0x03899b to 0x033d29 (0x4c71 bytes) 000730 1788524143.322319 trunc from 0x01f96d to 0x02e787 (0xee1b bytes) 000731 1788524143.329497 punch from 0x02eb96 to 0x033cd5 (0x5140 bytes) 000732 1788524143.339020 mapread 0x02072f thru 0x027119 (0x69eb bytes) 000733 1788524143.355614 trunc from 0x033d2a to 0x018f01 (0x1ae28 bytes) 000734 1788524143.381114 MIRROR_RESYNC 10 mirrors 000735 1788524145.404092 mapwrite 0x0223ca thru 0x026141 (0x3d78 bytes) 000736 1788524145.505484 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26142): Mirror 5: 0xc17e60c0 (stale) Mirror 6: 0xc17e60c0 (stale) Mirror 7: 0xc17e60c0 (stale) Mirror 8: 0xc17e60c0 (stale) Mirror 9: 0xc17e60c0 (stale) Mirror 10: 0xc17e60c0 (stale) Mirror 11: 0xc17e60c0 (stale) Mirror 12: 0xc17e60c0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26142): Mirror 5: 0xc17e60c0 (stale) Mirror 6: 0xc17e60c0 (stale) Mirror 7: 0xc17e60c0 (stale) Mirror 8: 0xc17e60c0 (stale) Mirror 9: 0xc17e60c0 (stale) Mirror 10: 0xc17e60c0 (stale) Mirror 11: 0xc17e60c0 (stale) Mirror 12: 0xc17e60c0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26142: skipped 000737 1788524146.032975 fallocate 0x037ad3 thru 0x03ffff (0x852d bytes) 000739 1788524146.038385 fallocate 0x00bf90 thru 0x017953 (0xb9c4 bytes) 000741 1788524146.046758 write 0x01d615 thru 0x02b34f (0xdd3b bytes) 000742 1788524146.061966 punch from 0x000f7a to 0x003661 (0x26e8 bytes) 000744 1788524146.070740 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2b350): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b350): Mirror 5: 0x41dab5a4 (stale) Mirror 6: 0x41dab5a4 (stale) Mirror 7: 0x41dab5a4 (stale) Mirror 8: 0x41dab5a4 (stale) Mirror 9: 0x41dab5a4 (stale) Mirror 10: 0x41dab5a4 (stale) Mirror 11: 0x41dab5a4 (stale) Mirror 12: 0x41dab5a4 (stale) CRC-32 checksum value for chunk [0, 0x2b350): Mirror 1: 0x600148cc Mirror 13: 0x600148cc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b350: skipped 000745 1788524147.092012 read 0x00b26c thru 0x019a58 (0xe7ed bytes) 000746 1788524147.106069 trunc from 0x02b350 to 0x008897 (0x22ab8 bytes) 000747 1788524147.161143 read 0x005255 thru 0x008897 (0x3643 bytes) 000748 1788524147.180793 mapread 0x005faf thru 0x008897 (0x28e9 bytes) 000749 1788524147.181758 punch from 0x0058be to 0x008897 (0x2fda bytes) 000750 1788524147.199519 mapread 0x00077e thru 0x00468d (0x3f10 bytes) 000751 1788524147.201433 mapread 0x001ada thru 0x008897 (0x6dbe bytes) 000753 1788524147.233046 trunc from 0x008898 to 0x01e64c (0x15db5 bytes) 000754 1788524147.254810 punch from 0x00bdfc to 0x00c896 (0xa9b bytes) 000755 1788524147.267252 write 0x01af8c thru 0x0236d9 (0x874e bytes) 000756 1788524147.288449 trunc from 0x022410 to 0x0236d9 (0x12ca bytes) 000757 1788524147.304507 mapread 0x001131 thru 0x00cc19 (0xbae9 bytes) 000758 1788524147.306324 write 0x013bf5 thru 0x022f98 (0xf3a4 bytes) 000759 1788524147.312229 trunc from 0x00ebda to 0x00ef0a (0x331 bytes) 000760 1788524147.317015 fallocate 0x029fb6 thru 0x02dfc0 (0x400b bytes) 000761 1788524147.320993 punch from 0x02cbba to 0x02dfc0 (0x1407 bytes) 000762 1788524147.325098 trunc from 0x02dfc1 to 0x020e92 (0xd12e bytes) 000763 1788524147.339801 trunc from 0x001daf to 0x007981 (0x5bd3 bytes) 000764 1788524147.345489 write 0x01e701 thru 0x02b714 (0xd014 bytes) 000765 1788524147.347373 trunc from 0x02b715 to 0x01998b (0x11d89 bytes) 000766 1788524147.368110 fallocate 0x01ef8a thru 0x02166a (0x26e1 bytes) 000767 1788524147.374777 fallocate 0x01d8b1 thru 0x023126 (0x5876 bytes) 000768 1788524147.380464 trunc from 0x01ccad to 0x01df22 (0x1276 bytes) 000770 1788524147.387337 trunc from 0x00e8f9 to 0x00f615 (0xd1d bytes) 000771 1788524147.391762 fallocate 0x035f15 thru 0x03ad9e (0x4e8a bytes) 000772 1788524147.395930 trunc from 0x022394 to 0x023126 (0xd93 bytes) 000773 1788524147.406042 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x23127): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23127): Mirror 5: 0x313ba812 (stale) Mirror 6: 0x313ba812 (stale) Mirror 7: 0x313ba812 (stale) Mirror 8: 0x313ba812 (stale) Mirror 9: 0x313ba812 (stale) Mirror 10: 0x313ba812 (stale) Mirror 11: 0x313ba812 (stale) Mirror 12: 0x313ba812 (stale) Mirror 13: 0x1ec99613 (stale) CRC-32 checksum value for chunk [0, 0x23127): Mirror 1: 0xa0a68d39 Mirror 14: 0xa0a68d39 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23127: skipped 000774 1788524148.563041 write 0x006e6a thru 0x015218 (0xe3af bytes) 000775 1788524148.622941 trunc from 0x00e075 to 0x019093 (0xb01f bytes) 000776 1788524148.636134 write 0x01783f thru 0x0235a2 (0xbd64 bytes) 000777 1788524148.644568 trunc from 0x00266e to 0x007de8 (0x577b bytes) 000778 1788524148.649399 punch from 0x01b0a7 to 0x01e6d4 (0x362e bytes) 000779 1788524148.661194 trunc from 0x010aa5 to 0x01a48a (0x99e6 bytes) 000780 1788524148.665435 mapread 0x007371 thru 0x00b56f (0x41ff bytes) 000781 1788524148.673584 fallocate 0x007822 thru 0x008a06 (0x11e5 bytes) 000782 1788524148.678084 trunc from 0x0235a3 to 0x02d166 (0x9bc4 bytes) 000783 1788524148.717420 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2d167): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d167): Mirror 5: 0x325e3291 (stale) Mirror 6: 0x325e3291 (stale) Mirror 7: 0x325e3291 (stale) Mirror 8: 0x325e3291 (stale) Mirror 9: 0x325e3291 (stale) Mirror 10: 0x325e3291 (stale) Mirror 11: 0x325e3291 (stale) Mirror 12: 0x325e3291 (stale) Mirror 13: 0x681b5da0 (stale) Mirror 14: 0xe76f3085 (stale) CRC-32 checksum value for chunk [0, 0x2d167): Mirror 1: 0xfb12d1e0 Mirror 15: 0xfb12d1e0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d167: skipped 000785 1788524149.684079 mapread 0x02b986 thru 0x02d166 (0x17e1 bytes) 000786 1788524149.706338 trunc from 0x02d167 to 0x00c6a9 (0x20abd bytes) 000787 1788524149.763186 mapwrite 0x01c972 thru 0x028b8b (0xc21a bytes) 000788 1788524149.816306 mapread 0x0167d7 thru 0x016995 (0x1bf bytes) 000789 1788524149.817407 trunc from 0x028b8c to 0x026772 (0x2419 bytes) 000790 1788524149.835197 trunc from 0x026773 to 0x01b251 (0xb521 bytes) 000791 1788524149.849634 punch from 0x015868 to 0x01b251 (0x59ea bytes) 000792 1788524149.853284 trunc from 0x00b2dd to 0x00b514 (0x238 bytes) 000793 1788524149.857276 mapread 0x006d07 thru 0x011bae (0xaea8 bytes) 000794 1788524149.874269 trunc from 0x00cce0 to 0x016afc (0x9e1d bytes) 000795 1788524149.878693 punch from 0x0163b1 to 0x01b251 (0x4ea1 bytes) 000797 1788524149.883944 trunc from 0x010a84 to 0x01a2e2 (0x985f bytes) 000798 1788524149.891119 MIRROR_SPLIT mirror 5 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b252): Mirror 5: 0x8a2938d0 (stale) Mirror 6: 0x8a2938d0 (stale) Mirror 7: 0x8a2938d0 (stale) Mirror 8: 0x8a2938d0 (stale) Mirror 10: 0x8a2938d0 (stale) Mirror 11: 0x8a2938d0 (stale) Mirror 12: 0x8a2938d0 (stale) Mirror 13: 0x8a2938d0 (stale) Mirror 14: 0x21cf1d8b (stale) Mirror 15: 0x40d2f98a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b252): Mirror 5: 0x8a2938d0 (stale) Mirror 6: 0x8a2938d0 (stale) Mirror 7: 0x8a2938d0 (stale) Mirror 8: 0x8a2938d0 (stale) Mirror 10: 0x8a2938d0 (stale) Mirror 11: 0x8a2938d0 (stale) Mirror 12: 0x8a2938d0 (stale) Mirror 13: 0x8a2938d0 (stale) Mirror 14: 0x21cf1d8b (stale) Mirror 15: 0x40d2f98a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b252: skipped 000799 1788524150.555832 mapwrite 0x015390 thru 0x01adcc (0x5a3d bytes) 000800 1788524150.585171 trunc from 0x016c5c to 0x017b0a (0xeaf bytes) 000801 1788524150.589383 trunc from 0x01b252 to 0x010d69 (0xa4e8 bytes) 000802 1788524150.604425 trunc from 0x010d6a to 0x03ade5 (0x2a07c bytes) 000803 1788524150.615569 trunc from 0x01c715 to 0x022083 (0x596f bytes) 000804 1788524150.620347 trunc from 0x03ade6 to 0x0229f8 (0x183ed bytes) 000805 1788524150.633439 fallocate 0x01183f thru 0x0201a4 (0xe966 bytes) 000806 1788524150.638479 trunc from 0x0167f9 to 0x01ec5d (0x8465 bytes) 000807 1788524150.642416 read 0x01c1fe thru 0x0229f8 (0x67fb bytes) 000808 1788524150.652502 trunc from 0x0229f9 to 0x016c03 (0xbdf5 bytes) 000809 1788524150.669933 write 0x000e78 thru 0x008e03 (0x7f8c bytes) 000810 1788524150.678572 read 0x00da34 thru 0x016c03 (0x91d0 bytes) 000811 1788524150.685880 trunc from 0x0133ed to 0x016c03 (0x3817 bytes) 000812 1788524150.690295 punch from 0x005fcb to 0x00cd4a (0x6d80 bytes) 000813 1788524150.705086 punch from 0x00f633 to 0x016c03 (0x75d1 bytes) 000814 1788524150.712745 MIRROR_RESYNC 11 mirrors 000815 1788524152.699260 write 0x023a53 thru 0x02d38e (0x993c bytes) 000816 1788524152.758630 mapwrite 0x0181eb thru 0x01abff (0x2a15 bytes) 000817 1788524152.851596 trunc from 0x02d38f to 0x03ca4b (0xf6bd bytes) 000818 1788524152.896777 mapwrite 0x03fb4c thru 0x03ffff (0x4b4 bytes) 000819 1788524152.965097 read 0x01b3ab thru 0x023da5 (0x89fb bytes) 000820 1788524152.971638 MIRROR_RESYNC 11 mirrors 000821 1788524156.152086 mapwrite 0x014a9e thru 0x01fcf5 (0xb258 bytes) 000822 1788524156.300370 punch from 0x00234d to 0x0038bf (0x1573 bytes) 000823 1788524156.307153 fallocate 0x00b9b1 thru 0x00ca6d (0x10bd bytes) 000824 1788524156.316825 fallocate 0x023fd7 thru 0x02f2ab (0xb2d5 bytes) 000825 1788524156.322054 fallocate 0x02e894 thru 0x03372d (0x4e9a bytes) 000826 1788524156.327616 punch from 0x01cbc0 to 0x01f3e6 (0x2827 bytes) 000827 1788524156.338428 trunc from 0x02d343 to 0x0392f7 (0xbfb5 bytes) 000828 1788524156.347236 write 0x00903c thru 0x00fcbf (0x6c84 bytes) 000829 1788524156.358042 punch from 0x03f871 to 0x03ffff (0x78f bytes) 000830 1788524156.371018 mapread 0x002d92 thru 0x0045eb (0x185a bytes) 000831 1788524156.390578 trunc from 0x0103b1 to 0x013dfb (0x3a4b bytes) 000832 1788524156.414624 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x67d4d4b6 (stale) Mirror 6: 0x67d4d4b6 (stale) Mirror 7: 0x67d4d4b6 (stale) Mirror 8: 0x67d4d4b6 (stale) Mirror 10: 0x67d4d4b6 (stale) Mirror 11: 0x67d4d4b6 (stale) Mirror 12: 0x67d4d4b6 (stale) Mirror 13: 0x67d4d4b6 (stale) Mirror 14: 0x67d4d4b6 (stale) Mirror 15: 0x67d4d4b6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x32d86f55 Mirror 16: 0x32d86f55 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000833 1788524158.127090 write 0x03e2ae thru 0x03e981 (0x6d4 bytes) 000834 1788524158.213449 mapread 0x02d9b9 thru 0x03bd01 (0xe349 bytes) 000835 1788524158.270537 trunc from 0x040000 to 0x017c27 (0x283d8 bytes) 000836 1788524158.310872 mapread 0x0153bf thru 0x017c27 (0x2869 bytes) 000837 1788524158.326104 trunc from 0x001a21 to 0x009f12 (0x84f2 bytes) 000838 1788524158.334312 mapread 0x01686b thru 0x017c27 (0x13bd bytes) 000839 1788524158.340905 MIRROR_RESYNC 12 mirrors 000840 1788524160.928227 trunc from 0x00db26 to 0x00f521 (0x19fc bytes) 000841 1788524160.998772 mapread 0x012dc8 thru 0x017c27 (0x4e60 bytes) 000842 1788524161.021632 write 0x01aadd thru 0x01d28c (0x27b0 bytes) 000843 1788524161.028089 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x1d28d): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d28d): Mirror 5: 0xdef34cf8 (stale) Mirror 6: 0xdef34cf8 (stale) Mirror 7: 0xdef34cf8 (stale) Mirror 8: 0xdef34cf8 (stale) Mirror 10: 0xdef34cf8 (stale) Mirror 11: 0xdef34cf8 (stale) Mirror 12: 0xdef34cf8 (stale) Mirror 13: 0xdef34cf8 (stale) Mirror 14: 0xdef34cf8 (stale) Mirror 15: 0xdef34cf8 (stale) Mirror 16: 0xdef34cf8 (stale) CRC-32 checksum value for chunk [0, 0x1d28d): Mirror 1: 0xb5eb9df4 Mirror 17: 0xb5eb9df4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d28d: skipped 000844 1788524162.063521 write 0x0000de thru 0x00981a (0x973d bytes) 000845 1788524162.117759 MIRROR_RESYNC 13 mirrors 000847 1788524165.036122 mapread 0x009d8c thru 0x010caa (0x6f1f bytes) 000848 1788524165.052894 write 0x009cd8 thru 0x00f2a9 (0x55d2 bytes) 000849 1788524165.090914 punch from 0x0183df to 0x0194ca (0x10ec bytes) 000850 1788524165.095415 mapread 0x016f8e thru 0x01d28c (0x62ff bytes) 000852 1788524165.105258 read 0x018ae5 thru 0x01c597 (0x3ab3 bytes) 000853 1788524165.105955 write 0x003360 thru 0x0060e4 (0x2d85 bytes) 000854 1788524165.116988 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x1d28d): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d28d): Mirror 5: 0xf1015e7b (stale) Mirror 6: 0xf1015e7b (stale) Mirror 7: 0xf1015e7b (stale) Mirror 8: 0xf1015e7b (stale) Mirror 10: 0xf1015e7b (stale) Mirror 11: 0xf1015e7b (stale) Mirror 12: 0xf1015e7b (stale) Mirror 13: 0xf1015e7b (stale) Mirror 14: 0xf1015e7b (stale) Mirror 15: 0xf1015e7b (stale) Mirror 16: 0xf1015e7b (stale) Mirror 17: 0xf1015e7b (stale) CRC-32 checksum value for chunk [0, 0x1d28d): Mirror 1: 0xe46891fc Mirror 18: 0xe46891fc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d28d: skipped 000856 1788524166.149678 MIRROR_RESYNC 14 mirrors 000859 1788524168.775350 punch from 0x004657 to 0x00810c (0x3ab6 bytes) 000861 1788524168.832615 mapwrite 0x01ddd3 thru 0x02da27 (0xfc55 bytes) 000862 1788524168.936475 punch from 0x0127d5 to 0x019dcb (0x75f7 bytes) 000863 1788524168.948896 MIRROR_RESYNC 14 mirrors 000865 1788524171.664111 trunc from 0x028d38 to 0x02c431 (0x36fa bytes) 000866 1788524171.707076 mapread 0x00dbcc thru 0x01b93c (0xdd71 bytes) 000867 1788524171.796488 mapread 0x0218c1 thru 0x028ab5 (0x71f5 bytes) 000868 1788524171.826149 trunc from 0x00285f to 0x012797 (0xff39 bytes) 000869 1788524171.838449 punch from 0x012661 to 0x017b37 (0x54d7 bytes) 000870 1788524171.847630 write 0x0239e7 thru 0x02f46d (0xba87 bytes) 000871 1788524171.849367 trunc from 0x00a68c to 0x00fc40 (0x55b5 bytes) 000872 1788524171.859386 trunc from 0x0217ad to 0x02f46d (0xdcc1 bytes) 000873 1788524171.889975 mapread 0x0062e9 thru 0x00bff8 (0x5d10 bytes) 000875 1788524171.902054 punch from 0x0244a3 to 0x02831a (0x3e78 bytes) 000876 1788524171.907394 punch from 0x005566 to 0x00b693 (0x612e bytes) 000877 1788524171.911554 write 0x02d9dd thru 0x03742b (0x9a4f bytes) 000878 1788524171.917337 mapread 0x007206 thru 0x012ad6 (0xb8d1 bytes) 000879 1788524171.933945 fallocate 0x005d99 thru 0x0061e8 (0x450 bytes) 000880 1788524171.943209 punch from 0x025daa to 0x035660 (0xf8b7 bytes) 000881 1788524171.962216 trunc from 0x01b92a to 0x02b1e4 (0xf8bb bytes) 000882 1788524171.970328 mapwrite 0x017d19 thru 0x0211d9 (0x94c1 bytes) 000883 1788524172.032442 mapwrite 0x00a0dc thru 0x00cf9e (0x2ec3 bytes) 000884 1788524172.060868 fallocate 0x0041af thru 0x00dd1f (0x9b71 bytes) 000885 1788524172.067151 punch from 0x0363a1 to 0x03742b (0x108b bytes) 000886 1788524172.073351 trunc from 0x03742c to 0x02f61c (0x7e0f bytes) 000887 1788524172.097905 punch from 0x006266 to 0x008133 (0x1ece bytes) 000888 1788524172.106694 punch from 0x00f84f to 0x01e908 (0xf0ba bytes) 000889 1788524172.112646 write 0x0066e2 thru 0x0092d9 (0x2bf8 bytes) 000890 1788524172.119345 mapread 0x012a66 thru 0x01ad3e (0x82d9 bytes) 000891 1788524172.144854 write 0x03a2c4 thru 0x03ffff (0x5d3c bytes) 000892 1788524172.146871 read 0x023c30 thru 0x02990d (0x5cde bytes) 000893 1788524172.159883 write 0x028d7f thru 0x037b5b (0xeddd bytes) 000894 1788524172.177429 MIRROR_SPLIT mirror 4 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x92562f20 (stale) Mirror 6: 0x92562f20 (stale) Mirror 7: 0x92562f20 (stale) Mirror 10: 0x92562f20 (stale) Mirror 11: 0x92562f20 (stale) Mirror 12: 0x92562f20 (stale) Mirror 13: 0x92562f20 (stale) Mirror 14: 0x92562f20 (stale) Mirror 15: 0x92562f20 (stale) Mirror 16: 0x92562f20 (stale) Mirror 17: 0x92562f20 (stale) Mirror 18: 0x92562f20 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x92562f20 (stale) Mirror 6: 0x92562f20 (stale) Mirror 7: 0x92562f20 (stale) Mirror 10: 0x92562f20 (stale) Mirror 11: 0x92562f20 (stale) Mirror 12: 0x92562f20 (stale) Mirror 13: 0x92562f20 (stale) Mirror 14: 0x92562f20 (stale) Mirror 15: 0x92562f20 (stale) Mirror 16: 0x92562f20 (stale) Mirror 17: 0x92562f20 (stale) Mirror 18: 0x92562f20 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000895 1788524172.788189 MIRROR_RESYNC 13 mirrors 000896 1788524176.482534 mapread 0x00f2c4 thru 0x01cb8c (0xd8c9 bytes) 000897 1788524176.513294 MIRROR_SPLIT mirror 10 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1bf:0x0]: Invalid argument (22) 000898 1788524177.094752 mapread 0x03b461 thru 0x03ffff (0x4b9f bytes) 000899 1788524177.121774 read 0x0166db thru 0x01a5ff (0x3f25 bytes) 000900 1788524177.137147 write 0x026092 thru 0x027984 (0x18f3 bytes) 000901 1788524177.261300 trunc from 0x040000 to 0x00d8fe (0x32701 bytes) 000902 1788524177.297409 fallocate 0x022678 thru 0x031d81 (0xf70a bytes) 000903 1788524177.302835 trunc from 0x003541 to 0x008d87 (0x5847 bytes) 000904 1788524177.316431 punch from 0x0064da to 0x00d8fe (0x7425 bytes) 000905 1788524177.321147 trunc from 0x00d016 to 0x00d8fe (0x8e9 bytes) 000906 1788524177.337844 fallocate 0x02dbc4 thru 0x03b956 (0xdd93 bytes) 000907 1788524177.344791 mapwrite 0x005811 thru 0x00947b (0x3c6b bytes) 000908 1788524177.430407 write 0x016ef8 thru 0x01ee9e (0x7fa7 bytes) 000909 1788524177.451980 mapwrite 0x008aa5 thru 0x00d338 (0x4894 bytes) 000910 1788524177.483774 trunc from 0x02ebd8 to 0x03b956 (0xcd7f bytes) 000911 1788524177.492673 fallocate 0x021f8c thru 0x02d131 (0xb1a6 bytes) 000912 1788524177.501830 write 0x026ab4 thru 0x02b963 (0x4eb0 bytes) 000914 1788524177.522050 punch from 0x0170a7 to 0x022e58 (0xbdb2 bytes) 000916 1788524177.527405 mapread 0x020e69 thru 0x022b61 (0x1cf9 bytes) 000917 1788524177.537046 trunc from 0x032d47 to 0x03b956 (0x8c10 bytes) 000919 1788524177.541880 read 0x00628e thru 0x006bca (0x93d bytes) 000921 1788524177.543306 trunc from 0x0333dd to 0x036985 (0x35a9 bytes) 000922 1788524177.547343 read 0x028106 thru 0x02af36 (0x2e31 bytes) 000923 1788524177.547981 mapread 0x017d83 thru 0x020c5d (0x8edb bytes) 000924 1788524177.558366 trunc from 0x013b32 to 0x017427 (0x38f6 bytes) 000925 1788524177.568246 MIRROR_RESYNC 12 mirrors 000926 1788524180.752528 punch from 0x01ab24 to 0x029f16 (0xf3f3 bytes) 000927 1788524180.797300 punch from 0x024e61 to 0x0317ef (0xc98f bytes) 000928 1788524180.803375 trunc from 0x03b957 to 0x005b9a (0x35dbc bytes) 000929 1788524180.822924 mapread 0x0003cc thru 0x0021fd (0x1e32 bytes) 000931 1788524180.837255 punch from 0x003ddb to 0x005b9a (0x1dc0 bytes) 000932 1788524180.842136 trunc from 0x001def to 0x005b9a (0x3dac bytes) 000933 1788524180.846970 trunc from 0x005b9b to 0x0107de (0xac44 bytes) 000934 1788524180.866492 write 0x00e276 thru 0x012c32 (0x49bd bytes) 000935 1788524180.871957 fallocate 0x029a17 thru 0x02d491 (0x3a7b bytes) 000936 1788524180.879104 mapread 0x005063 thru 0x010ad2 (0xba70 bytes) 000937 1788524180.888640 write 0x01f85d thru 0x02d4be (0xdc62 bytes) 000938 1788524180.898220 trunc from 0x02d4bf to 0x02490f (0x8baf bytes) 000939 1788524180.911870 trunc from 0x00be66 to 0x00ca04 (0xb9f bytes) 000940 1788524180.917113 read 0x021d7e thru 0x02490f (0x2b92 bytes) 000941 1788524180.918139 write 0x03d6fc thru 0x03ffff (0x2904 bytes) 000942 1788524180.919918 punch from 0x020d37 to 0x026985 (0x5c4f bytes) 000943 1788524180.954049 trunc from 0x040000 to 0x032547 (0xdab8 bytes) 000944 1788524180.993256 trunc from 0x032548 to 0x00e488 (0x240bf bytes) 000945 1788524181.022041 trunc from 0x00e489 to 0x02ef77 (0x20aef bytes) 000947 1788524181.065954 punch from 0x007354 to 0x016a10 (0xf6bd bytes) 000948 1788524181.078540 trunc from 0x023ee0 to 0x02dca5 (0x9dc6 bytes) 000949 1788524181.098127 read 0x022876 thru 0x026b7c (0x4307 bytes) 000950 1788524181.114489 mapwrite 0x036d5c thru 0x037a07 (0xcac bytes) 000951 1788524181.201519 read 0x01a5dc thru 0x027917 (0xd33c bytes) 000952 1788524181.204469 fallocate 0x012306 thru 0x01b0f1 (0x8dec bytes) 000953 1788524181.217405 read 0x020b61 thru 0x028aeb (0x7f8b bytes) 000954 1788524181.223928 MIRROR_SPLIT mirror 4 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37a08): Mirror 5: 0xa6bc1bb5 (stale) Mirror 6: 0xa6bc1bb5 (stale) Mirror 7: 0xa6bc1bb5 (stale) Mirror 11: 0xa6bc1bb5 (stale) Mirror 12: 0xa6bc1bb5 (stale) Mirror 13: 0xa6bc1bb5 (stale) Mirror 14: 0xa6bc1bb5 (stale) Mirror 15: 0xa6bc1bb5 (stale) Mirror 17: 0xa6bc1bb5 (stale) Mirror 18: 0xa6bc1bb5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a08): Mirror 5: 0xa6bc1bb5 (stale) Mirror 6: 0xa6bc1bb5 (stale) Mirror 7: 0xa6bc1bb5 (stale) Mirror 11: 0xa6bc1bb5 (stale) Mirror 12: 0xa6bc1bb5 (stale) Mirror 13: 0xa6bc1bb5 (stale) Mirror 14: 0xa6bc1bb5 (stale) Mirror 15: 0xa6bc1bb5 (stale) Mirror 17: 0xa6bc1bb5 (stale) Mirror 18: 0xa6bc1bb5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a08: skipped 000955 1788524182.127348 write 0x008aaa thru 0x00f811 (0x6d68 bytes) 000956 1788524182.143326 trunc from 0x037a08 to 0x03a921 (0x2f1a bytes) 000957 1788524182.160131 write 0x02d817 thru 0x031b05 (0x42ef bytes) 000958 1788524182.170226 write 0x008cb8 thru 0x01694e (0xdc97 bytes) 000959 1788524182.177066 mapwrite 0x026903 thru 0x03394c (0xd04a bytes) 000960 1788524182.227775 write 0x039835 thru 0x03fd23 (0x64ef bytes) 000961 1788524182.229003 mapwrite 0x02192b thru 0x02d71f (0xbdf5 bytes) 000963 1788524182.250260 punch from 0x01522e to 0x021e28 (0xcbfb bytes) 000964 1788524182.258369 trunc from 0x00beb7 to 0x00c190 (0x2da bytes) 000965 1788524182.274353 write 0x02c3e0 thru 0x0383b7 (0xbfd8 bytes) 000966 1788524182.282564 mapwrite 0x00472d thru 0x0073f5 (0x2cc9 bytes) 000967 1788524182.353615 trunc from 0x00b1a5 to 0x00e13a (0x2f96 bytes) 000968 1788524182.359204 trunc from 0x038695 to 0x03c1d8 (0x3b44 bytes) 000970 1788524182.399250 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fd24): Mirror 5: 0x71df8057 (stale) Mirror 7: 0x71df8057 (stale) Mirror 11: 0x71df8057 (stale) Mirror 12: 0x71df8057 (stale) Mirror 13: 0x71df8057 (stale) Mirror 14: 0x71df8057 (stale) Mirror 15: 0x71df8057 (stale) Mirror 17: 0x71df8057 (stale) Mirror 18: 0x71df8057 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd24): Mirror 5: 0x71df8057 (stale) Mirror 7: 0x71df8057 (stale) Mirror 11: 0x71df8057 (stale) Mirror 12: 0x71df8057 (stale) Mirror 13: 0x71df8057 (stale) Mirror 14: 0x71df8057 (stale) Mirror 15: 0x71df8057 (stale) Mirror 17: 0x71df8057 (stale) Mirror 18: 0x71df8057 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd24: skipped 000971 1788524182.997196 punch from 0x019f26 to 0x028ddc (0xeeb7 bytes) 000972 1788524183.002983 trunc from 0x00723c to 0x00e325 (0x70ea bytes) 000973 1788524183.007027 mapwrite 0x02e851 thru 0x0322fd (0x3aad bytes) 000974 1788524183.042091 fallocate 0x0282ed thru 0x031845 (0x9559 bytes) 000975 1788524183.047300 write 0x006aad thru 0x0125ae (0xbb02 bytes) 000976 1788524183.071624 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fd24): Mirror 7: 0x71df8057 (stale) Mirror 11: 0x71df8057 (stale) Mirror 12: 0x71df8057 (stale) Mirror 13: 0x71df8057 (stale) Mirror 14: 0x71df8057 (stale) Mirror 15: 0x71df8057 (stale) Mirror 17: 0x71df8057 (stale) Mirror 18: 0x71df8057 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd24): Mirror 7: 0x71df8057 (stale) Mirror 11: 0x71df8057 (stale) Mirror 12: 0x71df8057 (stale) Mirror 13: 0x71df8057 (stale) Mirror 14: 0x71df8057 (stale) Mirror 15: 0x71df8057 (stale) Mirror 17: 0x71df8057 (stale) Mirror 18: 0x71df8057 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd24: skipped 000977 1788524183.710185 trunc from 0x03fd24 to 0x034758 (0xb5cb bytes) 000978 1788524183.732401 mapread 0x013097 thru 0x014592 (0x14fc bytes) 000979 1788524183.740355 mapread 0x02bc0b thru 0x032e0b (0x7201 bytes) 000980 1788524183.749181 mapread 0x01f281 thru 0x0227ba (0x353a bytes) 000981 1788524183.756158 trunc from 0x034759 to 0x01c468 (0x182f0 bytes) 000982 1788524183.780906 trunc from 0x01c469 to 0x007e09 (0x1465f bytes) 000983 1788524183.800198 fallocate 0x029662 thru 0x034d53 (0xb6f2 bytes) 000984 1788524183.808529 mapread 0x00334b thru 0x007e09 (0x4abf bytes) 000985 1788524183.824130 read 0x006077 thru 0x007287 (0x1211 bytes) 000986 1788524183.824772 mapwrite 0x03bcb5 thru 0x03d6b3 (0x19ff bytes) 000987 1788524183.870066 trunc from 0x03d6b4 to 0x009d39 (0x3397a bytes) 000988 1788524183.885211 mapread 0x00834d thru 0x009d39 (0x19ed bytes) 000990 1788524183.891625 write 0x0198d4 thru 0x0246d8 (0xae05 bytes) 000991 1788524183.897192 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x246d9): Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x246d9): Mirror 7: 0x6756f8b6 (stale) Mirror 11: 0x6756f8b6 (stale) Mirror 12: 0x6756f8b6 (stale) Mirror 13: 0x6756f8b6 (stale) Mirror 14: 0x6756f8b6 (stale) Mirror 15: 0x6756f8b6 (stale) Mirror 17: 0x6756f8b6 (stale) Mirror 18: 0x6756f8b6 (stale) CRC-32 checksum value for chunk [0, 0x246d9): Mirror 1: 0x51a32e69 Mirror 19: 0x51a32e69 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x246d9: skipped 000992 1788524184.852435 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x246d9): Mirror 11: 0x6756f8b6 (stale) Mirror 12: 0x6756f8b6 (stale) Mirror 13: 0x6756f8b6 (stale) Mirror 14: 0x6756f8b6 (stale) Mirror 15: 0x6756f8b6 (stale) Mirror 17: 0x6756f8b6 (stale) Mirror 18: 0x6756f8b6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x246d9): Mirror 11: 0x6756f8b6 (stale) Mirror 12: 0x6756f8b6 (stale) Mirror 13: 0x6756f8b6 (stale) Mirror 14: 0x6756f8b6 (stale) Mirror 15: 0x6756f8b6 (stale) Mirror 17: 0x6756f8b6 (stale) Mirror 18: 0x6756f8b6 (stale) CRC-32 checksum value for chunk [0, 0x246d9): Mirror 1: 0x51a32e69 Mirror 19: 0x51a32e69 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x246d9: skipped 000994 1788524185.583012 trunc from 0x0149fa to 0x01a87f (0x5e86 bytes) 000995 1788524185.639466 trunc from 0x002555 to 0x010b39 (0xe5e5 bytes) 000996 1788524185.644360 fallocate 0x0207f2 thru 0x026d06 (0x6515 bytes) 000997 1788524185.648307 read 0x006848 thru 0x012973 (0xc12c bytes) 000998 1788524185.655630 read 0x020b6e thru 0x0246d8 (0x3b6b bytes) 000999 1788524185.677557 mapwrite 0x01587f thru 0x02094d (0xb0cf bytes) 001000 1788524185.719602 trunc from 0x0028b4 to 0x00eb03 (0xc250 bytes) All operations completed A-OK! PASS 70a (99s) == sanity-flr test 71: check mirror extend parent fid ==== 08:16:30 (1788524190) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1f8:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (9s) == sanity-flr test 72: check append on FLR file ========== 08:16:39 (1788524199) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.165358 s, 12.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133893 s, 7.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x117:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 08:16:47 (1788524207) Starting client: oleg654-client.virtnet: -o user_xattr,flock 192.168.206.154@tcp:/lustre /mnt/lustre2 Starting client: oleg654-client.virtnet: -o user_xattr,flock 192.168.206.154@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 7434944 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 1529728 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7239072 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8054624 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9338592 .. Extending file size to 8380192 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8711040 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8815424 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8882240 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 61500 61501 61504 61505 61507 192.168.206.154@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg654-client.virtnet /mnt/lustre2 (opts:) 192.168.206.154@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg654-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 08:18:07 (1788524287) Starting client: oleg654-client.virtnet: -o user_xattr,flock 192.168.206.154@tcp:/lustre /mnt/lustre2 Starting client: oleg654-client.virtnet: -o user_xattr,flock 192.168.206.154@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7316000 .. Extending file size to 8573728 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9060576 .. Extending file size to 7419072 .. Extending file size to 7865024 .. Extending file size to 8028576 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8249216 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9224576 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 63209 63211 63212 63214 63215 failed 192.168.206.154@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg654-client.virtnet /mnt/lustre2 (opts:) 192.168.206.154@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg654-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 08:19:25 (1788524365) Starting client: oleg654-client.virtnet: -o user_xattr,flock 192.168.206.154@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2276 1269028 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2260 1269044 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25764 3581256 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17900 3589120 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43664 7170376 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2276 1269028 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2260 1269044 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25764 3581256 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17900 3589120 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43664 7170376 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.107266 s, 19.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.252824 s, 8.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xafceddae (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.154@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg654-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 08:19:43 (1788524383) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104602 s, 20.0 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 08:19:49 (1788524389) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x60:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 08:19:56 (1788524396) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0926309 s, 22.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (6s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 08:20:02 (1788524402) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0680661 s, 30.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 08:20:09 (1788524409) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.167843 s, 12.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } PASS 204c (31s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 08:20:40 (1788524440) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg654-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806674 s, 13.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x270:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x271:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x270:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x270:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x271:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x270:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x271:0x0] } PASS 204d (46s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 08:21:26 (1788524486) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg654-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.061274 s, 34.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (47s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 08:22:14 (1788524534) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg654-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg654-server: Pool lustre.test_204f created Adding targets to pool oleg654-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg654-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.116362 s, 18.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg654-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg654-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg654-server: Pool lustre.test_204f destroyed PASS 204f (64s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 08:23:18 (1788524598) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 08:23:26 (1788524606) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 08:23:32 (1788524612) oleg654-server: Pool lustre.test_206 created Adding targets to pool oleg654-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg654-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg654-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg654-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg654-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 08:23:59 (1788524639) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938084 s, 11.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393384 s, 26.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x88b6483b Mirror 3: 0x88b6483b PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 08:24:10 (1788524650) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 08:24:15 (1788524655) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 08:24:19 (1788524659) Starting client: oleg654-client.virtnet: -o user_xattr,flock 192.168.206.154@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0401364 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1dd0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1710000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1710000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1710000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x720000 Read this (4 bytes): BBBB 192.168.206.154@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg654-client.virtnet /mnt/lustre2 (opts:) PASS 209a (33s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 08:24:51 (1788524691) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0222951 s, 367 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 08:24:59 (1788524699) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531328 s, 19.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11f:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611267 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 08:25:01.000000000 -0400 Modify: 2026-09-04 08:25:01.000000000 -0400 Change: 2026-09-04 08:25:02.000000000 -0400 Birth: 2026-09-04 08:25:01.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 08:25:07 (1788524707) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492263 s, 21.3 MB/s oleg654-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg654-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg654-server: O/280000401/d0/140: File not found by ext2_lookup PASS 210b (23s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 08:25:29 (1788524729) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.223534 s, 183 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0187862 s, 218 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 4 08:25 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x88:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 4 08:25 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 08:25:36 (1788524736) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (6s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 08:25:42 (1788524742) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (7s) == sanity-flr test complete, duration 1668 sec =========== 08:25:49 (1788524749) === sanity-flr: start cleanup 08:25:51 (1788524751) === === sanity-flr: finish cleanup 08:25:58 (1788524758) ===