-----============= acceptance-small: sanity-flr ============----- Sun Sep 6 04:43:50 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg253-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 04:44:09 (1788684249) === oleg253-client.virtnet: executing check_config_client /mnt/lustre oleg253-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg253-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff98fc84b35000.idle_timeout=debug osc.lustre-OST0001-osc-ffff98fc84b35000.idle_timeout=debug disable quota as required oleg253-server: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 04:44:47 (1788684287) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7630] == sanity-flr test 0a: lfs mirror create with -N option == 04:44:49 (1788684289) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 04:45:07 (1788684307) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 04:45:10 (1788684310) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 04:45:13 (1788684313) PASS 0d (18s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 04:45:31 (1788684331) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 04:45:35 (1788684335) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 04:45:39 (1788684339) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff98fc84b35000.stats=clear osc.lustre-OST0001-osc-ffff98fc84b35000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 04:45:47 (1788684347) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 04:45:59 (1788684359) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 04:46:10 (1788684370) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 04:46:14 (1788684374) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 04:46:23 (1788684383) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 04:46:32 (1788684392) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 04:46:38 (1788684398) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 04:46:41 (1788684401) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 04:46:50 (1788684410) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 04:46:59 (1788684419) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 04:47:06 (1788684426) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 2.31512 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108342 s, 9.7 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 04:47:17 (1788684437) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.44217 s, 35.6 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 04:47:25 (1788684445) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 04:47:36 (1788684456) Stopping /mnt/lustre-ost1 (opts:) on oleg253-server oleg253-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98fc84b35000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98fc84b35000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg253-server: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg253-server: oleg253-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg253-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg253-server oleg253-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98fc84b35000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98fc84b35000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg253-server: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg253-server: oleg253-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg253-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (71s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 04:48:47 (1788684527) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.688078 s, 16.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg253-server oleg253-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98fc84b35000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98fc84b35000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg253-server: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg253-server: oleg253-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg253-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (45s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 04:49:32 (1788684572) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg253-server oleg253-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98fc84b35000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98fc84b35000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg253-server: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg253-server: oleg253-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg253-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg253-server oleg253-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98fc84b35000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98fc84b35000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg253-server: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg253-server: oleg253-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg253-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (75s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 04:50:47 (1788684647) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.99161 s, 13.1 MB/s -rw-r--r-- 1 root root 52428800 Sep 6 04:50 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.113s user 0m0.001s sys 0m0.158s Stopping /mnt/lustre-ost1 (opts:) on oleg253-server oleg253-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98fc84b35000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98fc84b35000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.711s user 0m0.001s sys 0m0.192s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg253-server: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg253-server: oleg253-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg253-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg253-server oleg253-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98fc84b35000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98fc84b35000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.307s user 0m0.001s sys 0m0.164s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg253-server: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg253-server: oleg253-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg253-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg253-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98fc84b35000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98fc84b35000.ost_server_uuid in FULL state after 0 sec PASS 33b (94s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 04:52:22 (1788684742) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 04:52:25 (1788684745) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 04:52:29 (1788684749) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 04:52:32 (1788684752) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 04:52:38 (1788684758) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.534501 s, 23.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 04:52:50 (1788684770) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025974 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330311 s, 31.7 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 04:52:57 (1788684777) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 04:53:04 (1788684784) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00104842 s, 977 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00141478 s, 724 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00106771 s, 959 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00185716 s, 551 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 04:53:14 (1788684794) create test files with size 15760319 .. 250e6bf235ab818a6afcaad4a85c0597 - 6ec45ad243074849ea172f3a04c27bc6 - f41418f518502d84f5a751c2e6ea4883 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg253-client.virtnet /mnt/lustre (opts:) Starting client: oleg253-client.virtnet: -o user_xattr,flock 192.168.202.153@tcp:/lustre /mnt/lustre PASS 37 (21s) == sanity-flr test 38: resync ============================ 04:53:35 (1788684815) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 04:53:50 (1788684830) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 04:53:58 (1788684838) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104275 s, 10.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xca:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xc8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038715 s, 27.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xc9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xca:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 04:54:07 (1788684847) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.272118 s, 7.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.506547 s, 8.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.324222 s, 6.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 04:54:26 (1788684866) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 04:54:29 (1788684869) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 04:54:32 (1788684872) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147391 s, 71.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (5s) == sanity-flr test 44a: lfs mirror split check =========== 04:54:37 (1788684877) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 04:54:41 (1788684881) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd1:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd3:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd3:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 04:54:50 (1788684890) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.328565 s, 31.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg253-client.virtnet /mnt/lustre (opts:) Starting client: oleg253-client.virtnet: -o user_xattr,flock 192.168.202.153@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 04:55:01 (1788684901) llite.lustre-ffff98fc86192800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.221367 s, 47.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg253-client.virtnet /mnt/lustre (opts:) Starting client: oleg253-client.virtnet: -o user_xattr,flock 192.168.202.153@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff98fc852d5800.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 04:55:16 (1788684916) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (39s) == sanity-flr test 46: Verify setstripe --copy option ==== 04:55:55 (1788684955) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (38s) == sanity-flr test 47: Verify mirror obj alloc =========== 04:56:33 (1788684993) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 04:56:37 (1788684997) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.262692 s, 12.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.158127 s, 19.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x750fdc17 (nosync,stale) Mirror 2: 0x750fdc17 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 2d7c1bd465a74815a047bfc000088630 - mirror 1 checksum: 751d95a6d12fe5e1e4c5cb7621b05a91 - mirror 2 checksum: 2d7c1bd465a74815a047bfc000088630 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-06T04:56:39-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-06T04:56:39-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe9:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 2d7c1bd465a74815a047bfc000088630 - mirror 1 checksum: 751d95a6d12fe5e1e4c5cb7621b05a91 - mirror 2 checksum: 751d95a6d12fe5e1e4c5cb7621b05a91 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-06T04:56:39-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-06T04:56:39-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe9:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 04:56:49 (1788685009) Failing mds1 on oleg253-server Stopping /mnt/lustre-mds1 (opts:) on oleg253-server 04:56:57 (1788685017) shut down facet: mds1 facet_host: oleg253-server facet_failover_host: oleg253-server Failover mds1 to oleg253-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg253-server: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-MDT0000 04:57:20 (1788685040) targets are mounted 04:57:20 (1788685040) facet_failover done oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324436 s, 32.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xea:0x0] } PASS 50A (48s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 04:57:37 (1788685057) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 04:57:40 (1788685060) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 04:57:45 (1788685065) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00624783 s, 2.6 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 04:57:52 (1788685072) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 04:57:56 (1788685076) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00432882 s, 946 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0985027 s, 41.6 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 04:58:04 (1788685084) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 04:58:25 (1788685105) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 04:58:54 (1788685134) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 04:59:18 (1788685158) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 04:59:43 (1788685183) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 04:59:52 (1788685192) mirror create pid 54331 mirror split pid 54333 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54331 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54333 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 05:00:59 (1788685259) Seed set to 5261 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero length punch hole 000005 1788685261.622547 MIRROR_EXTEND to 2 mirrors skipping zero size read skipping zero size read skipping zero length punch hole 000010 1788685262.344092 MIRROR_RESYNC 2 mirrors skipping zero size read skipping zero size read truncating to largest ever: 0x14bf0 000013 1788685262.554598 trunc from 00000000 to 0x014bef (0x14bf0 bytes) 000016 1788685262.670827 punch from 0x00f99e to 0x014bef (0x5252 bytes) 000017 1788685262.706721 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) 000018 1788685262.944305 punch from 0x011035 to 0x01340f (0x23db bytes) 000019 1788685262.951089 mapread 0x012461 thru 0x014bef (0x278f bytes) 000022 1788685263.044238 MIRROR_EXTEND to 2 mirrors 000023 1788685264.176368 read 0x002e7e thru 0x004780 (0x1903 bytes) 000024 1788685264.181602 mapread 0x004c8e thru 0x007433 (0x27a6 bytes) truncating to largest ever: 0x39411 000025 1788685264.197986 trunc from 0x014bf0 to 0x039410 (0x24821 bytes) 000027 1788685264.284066 mapread 0x028718 thru 0x032932 (0xa21b bytes) 000028 1788685264.309143 trunc from 0x039411 to 0x02c308 (0xd108 bytes) 000030 1788685264.336521 trunc from 0x02c309 to 0x01b3d3 (0x10f35 bytes) 000032 1788685264.353881 mapread 0x01748c thru 0x01b3d3 (0x3f48 bytes) 000033 1788685264.361689 write 0x031c5a thru 0x03bccd (0xa074 bytes) 000034 1788685264.363280 write 0x016238 thru 0x020793 (0xa55c bytes) 000035 1788685264.381386 MIRROR_RESYNC 2 mirrors 000036 1788685265.094385 MIRROR_RESYNC 2 mirrors 000037 1788685265.511612 trunc from 0x03bcce to 0x0126fd (0x295d0 bytes) truncating to largest ever: 0x3a77b 000038 1788685265.645781 trunc from 0x0126fe to 0x03a77a (0x2807d bytes) 000039 1788685265.672525 mapread 0x011cb9 thru 0x013a2b (0x1d73 bytes) 000040 1788685265.693228 read 0x0172dd thru 0x025788 (0xe4ac bytes) 000042 1788685265.706955 write 0x024987 thru 0x033d61 (0xf3db bytes) 000045 1788685265.732314 punch from 0x003ae1 to 0x00ca25 (0x8f45 bytes) 000048 1788685265.749357 punch from 0x00303f to 0x0105c4 (0xd586 bytes) 000050 1788685265.756213 trunc from 0x03a77b to 0x029d9a (0x109e0 bytes) 000051 1788685265.782235 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) 000052 1788685265.936239 write 0x000785 thru 0x002d59 (0x25d5 bytes) 000053 1788685265.946538 mapwrite 0x0246da thru 0x028b22 (0x4449 bytes) 000054 1788685266.022611 read 0x024996 thru 0x029d9a (0x5405 bytes) 000055 1788685266.023650 punch from 0x0066b5 to 0x0072d9 (0xc25 bytes) 000056 1788685266.030683 read 0x00b0ab thru 0x014805 (0x975b bytes) 000057 1788685266.038935 read 0x027cc1 thru 0x029036 (0x1376 bytes) 000060 1788685266.042622 mapread 0x0295b5 thru 0x029d9a (0x7e6 bytes) 000061 1788685266.043785 trunc from 0x029d9b to 0x03a549 (0x107af bytes) 000063 1788685266.062906 read 0x0211a5 thru 0x0224f4 (0x1350 bytes) 000065 1788685266.065592 mapwrite 0x0395e4 thru 0x03ffff (0x6a1c bytes) 000066 1788685266.160609 trunc from 0x040000 to 0x0021f7 (0x3de08 bytes) 000068 1788685266.180542 mapwrite 0x0025d2 thru 0x010000 (0xda2f bytes) 000069 1788685266.391700 mapwrite 0x00acd2 thru 0x014a73 (0x9da2 bytes) 000070 1788685266.584999 trunc from 0x014a74 to 0x024499 (0xfa26 bytes) 000071 1788685266.618752 write 0x025636 thru 0x034088 (0xea53 bytes) 000073 1788685266.626298 trunc from 0x034089 to 0x0365f5 (0x256d bytes) 000076 1788685266.650360 trunc from 0x0365f6 to 0x019c75 (0x1c980 bytes) 000078 1788685266.675097 write 0x013ac9 thru 0x0158ca (0x1e02 bytes) 000079 1788685266.694945 MIRROR_EXTEND to 2 mirrors 000080 1788685267.516557 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) 000081 1788685267.635301 mapwrite 0x009f6b thru 0x00b7c7 (0x185d bytes) 000082 1788685267.696501 read 0x016134 thru 0x019c75 (0x3b42 bytes) 000083 1788685267.702490 punch from 0x00dbeb to 0x010238 (0x264e bytes) 000086 1788685267.708499 mapread 0x0099b1 thru 0x013fe5 (0xa635 bytes) 000088 1788685267.717709 punch from 0x008284 to 0x0113e4 (0x9161 bytes) 000090 1788685267.724624 write 0x00adb1 thru 0x0176dd (0xc92d bytes) 000097 1788685267.733804 punch from 0x00583f to 0x00ee20 (0x95e2 bytes) 000098 1788685267.744747 mapwrite 0x01f36c thru 0x027447 (0x80dc bytes) 000099 1788685267.817696 MIRROR_EXTEND to 2 mirrors 000100 1788685268.490544 MIRROR_EXTEND to 3 mirrors 000101 1788685269.359944 MIRROR_EXTEND to 4 mirrors 000102 1788685270.254607 mapread 0x024719 thru 0x027447 (0x2d2f bytes) 000103 1788685270.308151 mapread 0x005849 thru 0x00c140 (0x68f8 bytes) 000104 1788685270.364090 mapread 0x004a26 thru 0x0123a8 (0xd983 bytes) 000105 1788685270.393308 punch from 0x0011b0 to 0x00edd7 (0xdc28 bytes) 000106 1788685270.476679 mapread 0x00a8f2 thru 0x00b14f (0x85e bytes) 000109 1788685270.491592 mapwrite 0x03ac5a thru 0x03f4c2 (0x4869 bytes) 000110 1788685270.622664 mapwrite 0x0086fc thru 0x009333 (0xc38 bytes) 000112 1788685270.695151 punch from 0x0001c3 to 0x0002e2 (0x120 bytes) 000113 1788685270.724146 write 0x015a8a thru 0x021fa4 (0xc51b bytes) 000114 1788685270.742396 mapwrite 0x0394cf thru 0x03ffff (0x6b31 bytes) 000115 1788685270.856385 mapwrite 0x003b50 thru 0x006269 (0x271a bytes) 000116 1788685270.946552 MIRROR_RESYNC 4 mirrors 000117 1788685272.405748 mapwrite 0x010d58 thru 0x018f11 (0x81ba bytes) 000118 1788685272.688703 trunc from 0x040000 to 0x02c01f (0x13fe0 bytes) 000119 1788685272.726324 trunc from 0x02c020 to 0x0085ad (0x23a72 bytes) 000120 1788685272.746955 mapwrite 0x036fe4 thru 0x03ffff (0x901c bytes) 000123 1788685272.891431 write 0x02dbb8 thru 0x03d05a (0xf4a3 bytes) 000124 1788685272.904937 write 0x019261 thru 0x01db46 (0x48e6 bytes) 000125 1788685272.922722 mapwrite 0x02d891 thru 0x038186 (0xa8f6 bytes) 000127 1788685273.019184 punch from 0x000e8c to 0x00b258 (0xa3cd bytes) 000128 1788685273.033938 mapwrite 0x032a20 thru 0x03ffff (0xd5e0 bytes) 000130 1788685273.113641 write 0x00f849 thru 0x015004 (0x57bc bytes) 000132 1788685273.126159 trunc from 0x040000 to 0x003d6f (0x3c290 bytes) 000133 1788685273.144474 trunc from 0x003d70 to 0x02b2d4 (0x27565 bytes) 000134 1788685273.166291 read 0x01ddcc thru 0x0244bd (0x66f2 bytes) 000135 1788685273.177171 trunc from 0x02b2d5 to 0x00fad1 (0x1b803 bytes) 000136 1788685273.192471 mapread 0x00ef16 thru 0x00fad1 (0xbbc bytes) 000138 1788685273.203102 trunc from 0x00fad2 to 0x01f977 (0xfea6 bytes) 000139 1788685273.217071 mapwrite 0x0172e0 thru 0x022dfc (0xbb1d bytes) 000141 1788685273.308094 write 0x02a661 thru 0x03078a (0x612a bytes) 000142 1788685273.309532 mapwrite 0x006f3e thru 0x007f12 (0xfd5 bytes) 000146 1788685273.405205 trunc from 0x03078b to 0x00d2e5 (0x234a5 bytes) 000148 1788685273.425382 mapread 0x00687f thru 0x00a981 (0x4103 bytes) 000150 1788685273.427032 trunc from 0x00d2e6 to 0x023ea2 (0x16bbd bytes) 000151 1788685273.452544 MIRROR_RESYNC 4 mirrors 000152 1788685274.429501 mapwrite 0x00fb9e thru 0x011225 (0x1688 bytes) 000155 1788685274.820766 write 0x010ef1 thru 0x015c0a (0x4d1a bytes) 000156 1788685274.822791 mapread 0x013f3e thru 0x01eaa0 (0xab63 bytes) 000159 1788685274.854370 read 0x01189e thru 0x016ed5 (0x5638 bytes) 000161 1788685274.857532 trunc from 0x023ea3 to 0x010ec6 (0x12fdc bytes) 000162 1788685274.924513 read 0x000ca3 thru 0x002a86 (0x1de4 bytes) 000163 1788685274.943111 mapread 0x00cd95 thru 0x010ec6 (0x4132 bytes) 000164 1788685274.944223 punch from 0x0019eb to 0x009c4d (0x8263 bytes) 000165 1788685274.968495 mapread 0x002a5b thru 0x00b249 (0x87ef bytes) 000166 1788685275.022901 read 0x00855e thru 0x00ad30 (0x27d3 bytes) 000167 1788685275.023672 punch from 0x00c328 to 0x00f263 (0x2f3c bytes) 000168 1788685275.040137 mapread 0x001922 thru 0x00a36f (0x8a4e bytes) 000169 1788685275.050865 trunc from 0x010ec7 to 0x01fae9 (0xec23 bytes) 000171 1788685275.078117 MIRROR_RESYNC 4 mirrors 000174 1788685275.994767 mapread 0x0008a6 thru 0x005d8d (0x54e8 bytes) 000176 1788685276.022316 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) 000177 1788685276.332339 write 0x024c8f thru 0x029979 (0x4ceb bytes) 000179 1788685276.391314 mapwrite 0x02d8fc thru 0x037609 (0x9d0e bytes) 000180 1788685276.505614 trunc from 0x03760a to 0x03568e (0x1f7b bytes) 000181 1788685276.531940 trunc from 0x03568f to 0x00f990 (0x25cfe bytes) 000182 1788685276.561080 punch from 0x00f891 to 0x00f990 (0x100 bytes) 000183 1788685276.571315 read 0x00a0e6 thru 0x00ab10 (0xa2b bytes) 000184 1788685276.577891 read 0x0072be thru 0x00f990 (0x86d3 bytes) 000185 1788685276.589861 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xf991): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf991): Mirror 2: 0x5f2ff94e (stale) Mirror 4: 0x5f2ff94e (stale) CRC-32 checksum value for chunk [0, 0xf991): Mirror 1: 0x5f2ff94e Mirror 5: 0x5f2ff94e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf991: skipped 000186 1788685277.346612 mapwrite 0x00a22f thru 0x019b31 (0xf903 bytes) 000187 1788685277.554121 MIRROR_RESYNC 4 mirrors 000190 1788685278.382755 mapwrite 0x02c9f0 thru 0x03043e (0x3a4f bytes) 000191 1788685278.638110 read 0x00a66a thru 0x018acc (0xe463 bytes) 000192 1788685278.650390 write 0x0139e5 thru 0x0225ac (0xebc8 bytes) 000193 1788685278.670351 trunc from 0x03043f to 0x016f31 (0x1950d bytes) 000195 1788685278.698751 mapread 0x016d15 thru 0x016f31 (0x21d bytes) 000199 1788685278.707474 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x16f32): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16f32): Mirror 2: 0x12e3b80f (stale) Mirror 4: 0x12e3b80f (stale) Mirror 5: 0x12e3b80f (stale) CRC-32 checksum value for chunk [0, 0x16f32): Mirror 1: 0x81b2481c Mirror 6: 0x81b2481c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16f32: skipped 000201 1788685280.015268 punch from 0x00a74d to 0x016f31 (0xc7e5 bytes) 000203 1788685280.139310 mapread 0x00ca89 thru 0x016f31 (0xa4a9 bytes) 000204 1788685280.181990 punch from 0x009f2e to 0x00ff3c (0x600f bytes) 000205 1788685280.191086 mapwrite 0x03089b thru 0x03deba (0xd620 bytes) 000206 1788685280.339871 punch from 0x016ada to 0x022c84 (0xc1ab bytes) 000207 1788685280.348495 mapwrite 0x0112c5 thru 0x01e381 (0xd0bd bytes) 000208 1788685280.472544 MIRROR_RESYNC 5 mirrors 000210 1788685282.428577 mapwrite 0x01c3a1 thru 0x02c34f (0xffaf bytes) 000212 1788685282.693743 trunc from 0x03debb to 0x037601 (0x68b9 bytes) 000213 1788685282.763249 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x37602): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37602): Mirror 2: 0xdde7175d (stale) Mirror 4: 0xdde7175d (stale) Mirror 5: 0xdde7175d (stale) Mirror 6: 0xdde7175d (stale) CRC-32 checksum value for chunk [0, 0x37602): Mirror 1: 0xf1d2ddeb Mirror 7: 0xf1d2ddeb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37602: skipped 000214 1788685284.119340 punch from 0x01162e to 0x0210dc (0xfaaf bytes) 000215 1788685284.266868 write 0x01d2ee thru 0x024842 (0x7555 bytes) 000216 1788685284.308298 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x37602): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37602): Mirror 2: 0xdde7175d (stale) Mirror 4: 0xdde7175d (stale) Mirror 5: 0xdde7175d (stale) Mirror 6: 0xdde7175d (stale) Mirror 7: 0xf1d2ddeb (stale) CRC-32 checksum value for chunk [0, 0x37602): Mirror 1: 0xacbaa083 Mirror 8: 0xacbaa083 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37602: skipped 000222 1788685285.500712 read 0x031e3d thru 0x037601 (0x57c5 bytes) 000223 1788685285.523996 write 0x00fe5c thru 0x01e18f (0xe334 bytes) 000224 1788685285.665080 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x37602): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37602): Mirror 2: 0xdde7175d (stale) Mirror 4: 0xdde7175d (stale) Mirror 5: 0xdde7175d (stale) Mirror 6: 0xdde7175d (stale) Mirror 7: 0xf1d2ddeb (stale) Mirror 8: 0xacbaa083 (stale) CRC-32 checksum value for chunk [0, 0x37602): Mirror 1: 0xb1185ae3 Mirror 9: 0xb1185ae3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37602: skipped 000225 1788685286.633683 mapwrite 0x002ced thru 0x00c32a (0x963e bytes) 000226 1788685286.827832 mapwrite 0x03f88e thru 0x03ffff (0x772 bytes) 000227 1788685286.913330 read 0x012fed thru 0x01618c (0x31a0 bytes) 000228 1788685286.921519 mapwrite 0x0085bb thru 0x01855e (0xffa4 bytes) 000229 1788685286.984013 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xde666e83 (stale) Mirror 4: 0xde666e83 (stale) Mirror 6: 0xde666e83 (stale) Mirror 7: 0xee3d401e (stale) Mirror 8: 0x6fea99c8 (stale) Mirror 9: 0x886b0da0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xde666e83 (stale) Mirror 4: 0xde666e83 (stale) Mirror 6: 0xde666e83 (stale) Mirror 7: 0xee3d401e (stale) Mirror 8: 0x6fea99c8 (stale) Mirror 9: 0x886b0da0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000230 1788685287.387749 mapread 0x02a26b thru 0x033ef7 (0x9c8d bytes) 000231 1788685287.405643 read 0x01ffaa thru 0x024552 (0x45a9 bytes) 000232 1788685287.414807 mapwrite 0x015346 thru 0x016656 (0x1311 bytes) 000234 1788685287.498810 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xde666e83 (stale) Mirror 6: 0xde666e83 (stale) Mirror 7: 0xee3d401e (stale) Mirror 8: 0x6fea99c8 (stale) Mirror 9: 0x886b0da0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xde666e83 (stale) Mirror 6: 0xde666e83 (stale) Mirror 7: 0xee3d401e (stale) Mirror 8: 0x6fea99c8 (stale) Mirror 9: 0x886b0da0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000236 1788685287.890305 punch from 0x02e153 to 0x035710 (0x75be bytes) 000237 1788685287.896578 mapwrite 0x036361 thru 0x03db28 (0x77c8 bytes) 000238 1788685287.960171 punch from 0x012564 to 0x016ae8 (0x4585 bytes) 000239 1788685287.967306 punch from 0x017855 to 0x01deab (0x6657 bytes) 000241 1788685287.972597 read 0x00a92a thru 0x00cbea (0x22c1 bytes) 000242 1788685287.978686 read 0x0020e1 thru 0x00d631 (0xb551 bytes) 000243 1788685287.991679 trunc from 0x040000 to 0x00a622 (0x359dd bytes) 000244 1788685288.006492 read 0x00052e thru 0x007c2e (0x7701 bytes) 000246 1788685288.012267 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xa623): Mirror 4: 0xd5bf7932 (stale) Mirror 6: 0xd5bf7932 (stale) Mirror 7: 0xd5bf7932 (stale) Mirror 9: 0xd5bf7932 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa623): Mirror 4: 0xd5bf7932 (stale) Mirror 6: 0xd5bf7932 (stale) Mirror 7: 0xd5bf7932 (stale) Mirror 9: 0xd5bf7932 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa623: skipped 000248 1788685288.498225 trunc from 0x00a623 to 0x034350 (0x29d2e bytes) 000251 1788685288.533756 punch from 0x011b1e to 0x019eb0 (0x8393 bytes) 000252 1788685288.540015 read 0x00a57e thru 0x015444 (0xaec7 bytes) 000253 1788685288.551456 mapread 0x0056d5 thru 0x00e559 (0x8e85 bytes) 000255 1788685288.577285 mapread 0x020ee0 thru 0x024e8f (0x3fb0 bytes) 000256 1788685288.587639 punch from 0x0280e3 to 0x02fde5 (0x7d03 bytes) 000257 1788685288.592862 mapwrite 0x000bdb thru 0x009559 (0x897f bytes) 000259 1788685288.662791 write 0x032e3b thru 0x0382ce (0x5494 bytes) 000260 1788685288.672158 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x382cf): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x382cf): Mirror 4: 0xcbfa37d7 (stale) Mirror 6: 0xcbfa37d7 (stale) Mirror 7: 0x75e18a80 (stale) Mirror 9: 0xb499e319 (stale) CRC-32 checksum value for chunk [0, 0x382cf): Mirror 1: 0xcb8619b9 Mirror 10: 0xcb8619b9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x382cf: skipped truncating to largest ever: 0x3de54 000262 1788685289.659309 trunc from 0x0382cf to 0x03de53 (0x5b85 bytes) 000263 1788685289.750596 mapwrite 0x01165f thru 0x013664 (0x2006 bytes) 000264 1788685289.809345 punch from 0x013d43 to 0x01c6ac (0x896a bytes) 000265 1788685289.815459 mapwrite 0x03de44 thru 0x03ffff (0x21bc bytes) 000266 1788685289.883837 mapwrite 0x03b0e1 thru 0x03c00b (0xf2b bytes) 000267 1788685289.927372 mapwrite 0x030302 thru 0x03fed1 (0xfbd0 bytes) 000270 1788685290.016513 write 0x01baea thru 0x02acce (0xf1e5 bytes) 000272 1788685290.035339 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xde666e83 (stale) Mirror 6: 0xde666e83 (stale) Mirror 7: 0xee3d401e (stale) Mirror 9: 0x886b0da0 (stale) Mirror 10: 0x333805ce (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbf6acaf Mirror 11: 0xbf6acaf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000274 1788685290.951444 write 0x00b769 thru 0x0123b2 (0x6c4a bytes) 000276 1788685291.040078 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xde666e83 (stale) Mirror 7: 0xee3d401e (stale) Mirror 9: 0x886b0da0 (stale) Mirror 10: 0x333805ce (stale) Mirror 11: 0xbf6acaf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xde666e83 (stale) Mirror 7: 0xee3d401e (stale) Mirror 9: 0x886b0da0 (stale) Mirror 10: 0x333805ce (stale) Mirror 11: 0xbf6acaf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000277 1788685291.457898 read 0x006e64 thru 0x00be86 (0x5023 bytes) 000279 1788685291.467340 mapread 0x028461 thru 0x03794d (0xf4ed bytes) 000280 1788685291.492463 write 0x02145a thru 0x02622a (0x4dd1 bytes) 000281 1788685291.500779 mapwrite 0x012c32 thru 0x01d8e2 (0xacb1 bytes) 000282 1788685291.591652 mapread 0x02460e thru 0x033612 (0xf005 bytes) 000283 1788685291.592646 punch from 0x03bf39 to 0x03e8db (0x29a3 bytes) 000285 1788685291.635983 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xde666e83 (stale) Mirror 7: 0xee3d401e (stale) Mirror 9: 0x886b0da0 (stale) Mirror 10: 0x333805ce (stale) Mirror 11: 0xbf6acaf (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbd6dd908 Mirror 12: 0xbd6dd908 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000286 1788685292.732813 punch from 0x033f4c to 0x03ffff (0xc0b4 bytes) 000290 1788685292.816530 trunc from 0x040000 to 0x007701 (0x388fe bytes) 000291 1788685292.839233 mapread 0x005868 thru 0x007701 (0x1e9a bytes) 000292 1788685292.858878 read 0x00564b thru 0x0076f2 (0x20a8 bytes) 000293 1788685292.859470 write 0x021ee9 thru 0x028ad0 (0x6be8 bytes) 000294 1788685292.860652 punch from 0x00a65e to 0x00ccc2 (0x2665 bytes) 000295 1788685292.871733 mapwrite 0x0158bc thru 0x024e2f (0xf574 bytes) 000296 1788685292.939160 write 0x01085d thru 0x016744 (0x5ee8 bytes) 000297 1788685292.941198 write 0x026ac7 thru 0x0317c7 (0xad01 bytes) 000298 1788685292.942937 mapread 0x02f002 thru 0x0317c7 (0x27c6 bytes) 000299 1788685292.946212 mapwrite 0x004b6b thru 0x00bee4 (0x737a bytes) 000301 1788685293.048019 write 0x01f303 thru 0x02ac39 (0xb937 bytes) 000302 1788685293.053800 MIRROR_RESYNC 7 mirrors 000303 1788685294.205804 punch from 0x01ffc6 to 0x022dd2 (0x2e0d bytes) 000304 1788685294.272574 mapwrite 0x00ea13 thru 0x012c0a (0x41f8 bytes) 000305 1788685294.338096 trunc from 0x0317c8 to 0x01568d (0x1c13a bytes) 000306 1788685294.363466 mapwrite 0x01758d thru 0x027035 (0xfaa9 bytes) 000307 1788685294.444172 mapread 0x015ceb thru 0x01a552 (0x4868 bytes) 000310 1788685294.447581 punch from 0x0036a8 to 0x008bc1 (0x551a bytes) 000311 1788685294.453056 read 0x02009f thru 0x022e34 (0x2d96 bytes) 000312 1788685294.453702 trunc from 0x027036 to 0x006e6e (0x201c7 bytes) 000314 1788685294.476369 write 0x00c90f thru 0x0179fb (0xb0ed bytes) 000315 1788685294.478446 punch from 0x017165 to 0x0179fb (0x897 bytes) 000316 1788685294.500790 mapread 0x000791 thru 0x006fbf (0x682f bytes) 000318 1788685294.513308 mapwrite 0x00a63b thru 0x0188d1 (0xe297 bytes) 000319 1788685294.608896 mapread 0x00b193 thru 0x00f75e (0x45cc bytes) 000320 1788685294.612550 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x188d2): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x188d2): Mirror 6: 0xa53bee3f (stale) Mirror 7: 0xa53bee3f (stale) Mirror 9: 0xa53bee3f (stale) Mirror 10: 0xa53bee3f (stale) Mirror 11: 0xa53bee3f (stale) Mirror 12: 0xa53bee3f (stale) CRC-32 checksum value for chunk [0, 0x188d2): Mirror 1: 0x665fca21 Mirror 13: 0x665fca21 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x188d2: skipped 000322 1788685295.986349 read 0x00ef7c thru 0x0188d1 (0x9956 bytes) 000323 1788685296.008760 write 0x02e68f thru 0x030713 (0x2085 bytes) 000324 1788685296.096367 read 0x01ae1b thru 0x01d2ee (0x24d4 bytes) 000325 1788685296.117065 punch from 0x0159d5 to 0x01633b (0x967 bytes) 000326 1788685296.126383 mapwrite 0x02a13e thru 0x035b3d (0xba00 bytes) 000329 1788685296.216098 write 0x017231 thru 0x0191d9 (0x1fa9 bytes) 000331 1788685296.218307 punch from 0x023753 to 0x028cb5 (0x5563 bytes) 000333 1788685296.231572 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x35b3e): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35b3e): Mirror 6: 0xcdeb4d41 (stale) Mirror 7: 0xcdeb4d41 (stale) Mirror 9: 0xcdeb4d41 (stale) Mirror 10: 0xcdeb4d41 (stale) Mirror 11: 0xcdeb4d41 (stale) Mirror 12: 0xcdeb4d41 (stale) Mirror 13: 0x22f35d0c (stale) CRC-32 checksum value for chunk [0, 0x35b3e): Mirror 1: 0x92e29ec Mirror 14: 0x92e29ec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35b3e: skipped 000335 1788685297.080769 punch from 0x00f738 to 0x01093e (0x1207 bytes) 000337 1788685297.169054 MIRROR_RESYNC 9 mirrors 000338 1788685298.554007 mapread 0x01efe0 thru 0x026cae (0x7ccf bytes) 000339 1788685298.582582 mapread 0x0087db thru 0x00d2a4 (0x4aca bytes) 000340 1788685298.602827 punch from 0x025a39 to 0x02e2e8 (0x88b0 bytes) 000342 1788685298.704277 write 0x01c671 thru 0x028576 (0xbf06 bytes) 000344 1788685298.722059 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x35b3e): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35b3e): Mirror 6: 0xa051929 (stale) Mirror 7: 0xa051929 (stale) Mirror 9: 0xa051929 (stale) Mirror 10: 0xa051929 (stale) Mirror 11: 0xa051929 (stale) Mirror 12: 0xa051929 (stale) Mirror 13: 0xa051929 (stale) Mirror 14: 0xa051929 (stale) CRC-32 checksum value for chunk [0, 0x35b3e): Mirror 1: 0xa4062fc2 Mirror 15: 0xa4062fc2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35b3e: skipped 000347 1788685300.232973 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x35b3e): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35b3e): Mirror 6: 0xa051929 (stale) Mirror 7: 0xa051929 (stale) Mirror 9: 0xa051929 (stale) Mirror 10: 0xa051929 (stale) Mirror 11: 0xa051929 (stale) Mirror 12: 0xa051929 (stale) Mirror 13: 0xa051929 (stale) Mirror 14: 0xa051929 (stale) CRC-32 checksum value for chunk [0, 0x35b3e): Mirror 1: 0xa4062fc2 Mirror 15: 0xa4062fc2 Mirror 16: 0xa4062fc2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35b3e: skipped 000348 1788685301.584531 mapread 0x002f64 thru 0x006b99 (0x3c36 bytes) 000349 1788685301.618542 punch from 0x0305c1 to 0x035b3d (0x557d bytes) 000350 1788685301.731864 read 0x028fa4 thru 0x035b3d (0xcb9a bytes) 000351 1788685301.755066 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x35b3e): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35b3e): Mirror 6: 0xa051929 (stale) Mirror 7: 0xa051929 (stale) Mirror 9: 0xa051929 (stale) Mirror 10: 0xa051929 (stale) Mirror 11: 0xa051929 (stale) Mirror 12: 0xa051929 (stale) Mirror 13: 0xa051929 (stale) Mirror 14: 0xa051929 (stale) Mirror 15: 0xa4062fc2 (stale) Mirror 16: 0xa4062fc2 (stale) CRC-32 checksum value for chunk [0, 0x35b3e): Mirror 1: 0x11d1b318 Mirror 17: 0x11d1b318 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35b3e: skipped 000352 1788685303.263182 read 0x016b27 thru 0x01a810 (0x3cea bytes) 000353 1788685303.297122 punch from 0x0349f5 to 0x035b3d (0x1149 bytes) 000354 1788685303.445358 write 0x00659e thru 0x00c54e (0x5fb1 bytes) 000356 1788685303.471615 trunc from 0x035b3e to 0x01654d (0x1f5f0 bytes) 000357 1788685303.515870 punch from 0x0027ac to 0x0095cd (0x6e22 bytes) 000358 1788685303.533763 read 0x00df2a thru 0x01654d (0x8624 bytes) 000360 1788685303.572994 trunc from 0x01654e to 0x02a3e1 (0x13e94 bytes) 000363 1788685303.609784 MIRROR_SPLIT mirror 4 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a3e2): Mirror 6: 0xed5a72f9 (stale) Mirror 7: 0xed5a72f9 (stale) Mirror 9: 0xed5a72f9 (stale) Mirror 11: 0xed5a72f9 (stale) Mirror 12: 0xed5a72f9 (stale) Mirror 13: 0xed5a72f9 (stale) Mirror 14: 0xed5a72f9 (stale) Mirror 15: 0x5cf746d5 (stale) Mirror 16: 0x5cf746d5 (stale) Mirror 17: 0x5cf746d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a3e2): Mirror 6: 0xed5a72f9 (stale) Mirror 7: 0xed5a72f9 (stale) Mirror 9: 0xed5a72f9 (stale) Mirror 11: 0xed5a72f9 (stale) Mirror 12: 0xed5a72f9 (stale) Mirror 13: 0xed5a72f9 (stale) Mirror 14: 0xed5a72f9 (stale) Mirror 15: 0x5cf746d5 (stale) Mirror 16: 0x5cf746d5 (stale) Mirror 17: 0x5cf746d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a3e2: skipped 000364 1788685304.207076 read 0x01326a thru 0x01a3ae (0x7145 bytes) 000366 1788685304.228105 mapwrite 0x024b9f thru 0x033aa3 (0xef05 bytes) 000368 1788685304.416331 write 0x00e2f2 thru 0x0133e0 (0x50ef bytes) truncating to largest ever: 0x3f898 000370 1788685304.426263 trunc from 0x033aa4 to 0x03f897 (0xbdf4 bytes) 000371 1788685304.452168 mapread 0x0031d2 thru 0x0061b2 (0x2fe1 bytes) 000372 1788685304.476775 mapread 0x0013ae thru 0x0023de (0x1031 bytes) 000373 1788685304.485528 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3f898): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f898): Mirror 6: 0x9ce98f88 (stale) Mirror 7: 0x9ce98f88 (stale) Mirror 9: 0x9ce98f88 (stale) Mirror 11: 0x9ce98f88 (stale) Mirror 12: 0x9ce98f88 (stale) Mirror 13: 0x9ce98f88 (stale) Mirror 14: 0x9ce98f88 (stale) Mirror 15: 0x955a3eac (stale) Mirror 16: 0x955a3eac (stale) Mirror 17: 0xf5e40ab4 (stale) CRC-32 checksum value for chunk [0, 0x3f898): Mirror 1: 0x57e99ad0 Mirror 18: 0x57e99ad0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f898: skipped 000374 1788685306.118478 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3f898): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f898): Mirror 6: 0x9ce98f88 (stale) Mirror 7: 0x9ce98f88 (stale) Mirror 9: 0x9ce98f88 (stale) Mirror 11: 0x9ce98f88 (stale) Mirror 12: 0x9ce98f88 (stale) Mirror 13: 0x9ce98f88 (stale) Mirror 14: 0x9ce98f88 (stale) Mirror 15: 0x955a3eac (stale) Mirror 16: 0x955a3eac (stale) Mirror 17: 0xf5e40ab4 (stale) CRC-32 checksum value for chunk [0, 0x3f898): Mirror 1: 0x57e99ad0 Mirror 18: 0x57e99ad0 Mirror 19: 0x57e99ad0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f898: skipped 000376 1788685307.361749 punch from 0x03633c to 0x03f897 (0x955c bytes) 000378 1788685307.454396 mapwrite 0x03e483 thru 0x03ffff (0x1b7d bytes) 000380 1788685307.515686 mapwrite 0x03522e thru 0x03fbf3 (0xa9c6 bytes) 000384 1788685307.586665 write 0x02a5cb thru 0x02b483 (0xeb9 bytes) 000385 1788685307.597626 read 0x038a9a thru 0x03f4c8 (0x6a2f bytes) 000388 1788685307.605363 mapwrite 0x008d9b thru 0x015da4 (0xd00a bytes) 000389 1788685307.707544 mapread 0x003e16 thru 0x00bb0d (0x7cf8 bytes) 000390 1788685307.708617 write 0x03662d thru 0x038d64 (0x2738 bytes) 000391 1788685307.712715 punch from 0x03cd9e to 0x03ffff (0x3262 bytes) 000392 1788685307.727188 punch from 0x023cfa to 0x02d573 (0x987a bytes) 000393 1788685307.736993 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xc6f7b08e (stale) Mirror 9: 0xc6f7b08e (stale) Mirror 11: 0xc6f7b08e (stale) Mirror 12: 0xc6f7b08e (stale) Mirror 13: 0xc6f7b08e (stale) Mirror 14: 0xc6f7b08e (stale) Mirror 15: 0xc44dab54 (stale) Mirror 16: 0xc44dab54 (stale) Mirror 17: 0x73b9f2ff (stale) Mirror 18: 0x293420c2 (stale) Mirror 19: 0x293420c2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xc6f7b08e (stale) Mirror 9: 0xc6f7b08e (stale) Mirror 11: 0xc6f7b08e (stale) Mirror 12: 0xc6f7b08e (stale) Mirror 13: 0xc6f7b08e (stale) Mirror 14: 0xc6f7b08e (stale) Mirror 15: 0xc44dab54 (stale) Mirror 16: 0xc44dab54 (stale) Mirror 17: 0x73b9f2ff (stale) Mirror 18: 0x293420c2 (stale) Mirror 19: 0x293420c2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000394 1788685308.406538 punch from 0x009726 to 0x00cc0b (0x34e6 bytes) 000395 1788685308.414379 trunc from 0x040000 to 0x01b05a (0x24fa5 bytes) 000396 1788685308.439467 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x1b05b): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b05b): Mirror 7: 0x161280bd (stale) Mirror 9: 0x161280bd (stale) Mirror 11: 0x161280bd (stale) Mirror 12: 0x161280bd (stale) Mirror 13: 0x161280bd (stale) Mirror 14: 0x161280bd (stale) Mirror 15: 0x161280bd (stale) Mirror 16: 0x161280bd (stale) Mirror 17: 0x161280bd (stale) Mirror 18: 0x80c885d1 (stale) Mirror 19: 0x80c885d1 (stale) CRC-32 checksum value for chunk [0, 0x1b05b): Mirror 1: 0x4bc15ed2 Mirror 20: 0x4bc15ed2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b05b: skipped 000398 1788685309.638471 punch from 0x008024 to 0x00f12e (0x710b bytes) 000399 1788685309.736439 read 0x00197d thru 0x00eba1 (0xd225 bytes) 000400 1788685309.747828 MIRROR_SPLIT mirror 7 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b05b): Mirror 7: 0x161280bd (stale) Mirror 9: 0x161280bd (stale) Mirror 11: 0x161280bd (stale) Mirror 12: 0x161280bd (stale) Mirror 13: 0x161280bd (stale) Mirror 14: 0x161280bd (stale) Mirror 16: 0x161280bd (stale) Mirror 17: 0x161280bd (stale) Mirror 18: 0x80c885d1 (stale) Mirror 19: 0x80c885d1 (stale) Mirror 20: 0x4bc15ed2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b05b): Mirror 7: 0x161280bd (stale) Mirror 9: 0x161280bd (stale) Mirror 11: 0x161280bd (stale) Mirror 12: 0x161280bd (stale) Mirror 13: 0x161280bd (stale) Mirror 14: 0x161280bd (stale) Mirror 16: 0x161280bd (stale) Mirror 17: 0x161280bd (stale) Mirror 18: 0x80c885d1 (stale) Mirror 19: 0x80c885d1 (stale) Mirror 20: 0x4bc15ed2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b05b: skipped 000402 1788685310.300628 write 0x032295 thru 0x03317b (0xee7 bytes) 000403 1788685310.301664 write 0x000a26 thru 0x006f88 (0x6563 bytes) 000405 1788685310.314748 punch from 0x023a99 to 0x02b972 (0x7eda bytes) 000406 1788685310.334092 punch from 0x024884 to 0x02eff1 (0xa76e bytes) 000407 1788685310.340578 punch from 0x024aef to 0x026071 (0x1583 bytes) 000409 1788685310.347819 read 0x014727 thru 0x01d922 (0x91fc bytes) 000410 1788685310.366368 read 0x02dbcb thru 0x02ec43 (0x1079 bytes) 000413 1788685310.381822 mapwrite 0x026d51 thru 0x032c72 (0xbf22 bytes) 000414 1788685310.501935 write 0x00f3a5 thru 0x016aa7 (0x7703 bytes) 000416 1788685310.504089 punch from 0x00e69c to 0x0111b9 (0x2b1e bytes) 000417 1788685310.528785 trunc from 0x03317c to 0x01e043 (0x15138 bytes) 000418 1788685310.551954 read 0x012e7a thru 0x01596a (0x2af1 bytes) 000419 1788685310.552607 read 0x01a76b thru 0x01e043 (0x38d9 bytes) 000420 1788685310.555004 trunc from 0x01e044 to 0x01cd33 (0x1310 bytes) 000421 1788685310.566930 mapread 0x00d79f thru 0x01cd33 (0xf595 bytes) 000422 1788685310.572707 write 0x02060b thru 0x02b29e (0xac94 bytes) 000423 1788685310.574781 trunc from 0x02b29f to 0x022645 (0x8c59 bytes) 000425 1788685310.596494 read 0x0212bf thru 0x0212f4 (0x36 bytes) 000426 1788685310.596893 read 0x00a824 thru 0x00c1c7 (0x19a4 bytes) 000427 1788685310.597631 punch from 0x021668 to 0x022645 (0xfde bytes) 000428 1788685310.608734 trunc from 0x022646 to 0x01299c (0xfca9 bytes) 000429 1788685310.640614 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x1299d): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1299d): Mirror 7: 0x5309f689 (stale) Mirror 9: 0x5309f689 (stale) Mirror 11: 0x5309f689 (stale) Mirror 12: 0x5309f689 (stale) Mirror 13: 0x5309f689 (stale) Mirror 14: 0x5309f689 (stale) Mirror 16: 0x5309f689 (stale) Mirror 17: 0x5309f689 (stale) Mirror 18: 0x116eaa0a (stale) Mirror 19: 0x116eaa0a (stale) Mirror 20: 0x29f8bcf2 (stale) CRC-32 checksum value for chunk [0, 0x1299d): Mirror 1: 0xe667d562 Mirror 21: 0xe667d562 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1299d: skipped 000430 1788685311.996705 punch from 0x0026e2 to 0x00c223 (0x9b42 bytes) 000432 1788685312.124480 mapwrite 0x00e0bb thru 0x01ccd4 (0xec1a bytes) 000433 1788685312.251622 MIRROR_RESYNC 13 mirrors 000434 1788685314.158805 mapwrite 0x02d9f1 thru 0x02ed09 (0x1319 bytes) 000435 1788685314.360710 MIRROR_RESYNC 13 mirrors 000437 1788685316.924697 punch from 0x0194b9 to 0x01a4c1 (0x1009 bytes) 000438 1788685317.031687 read 0x0020e0 thru 0x004099 (0x1fba bytes) 000439 1788685317.042584 mapread 0x01558e thru 0x016e88 (0x18fb bytes) 000440 1788685317.056251 trunc from 0x02ed0a to 0x003dc4 (0x2af45 bytes) 000441 1788685317.099027 MIRROR_SPLIT mirror 9 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dc5): Mirror 7: 0x10d47b2b (stale) Mirror 9: 0x10d47b2b (stale) Mirror 11: 0x10d47b2b (stale) Mirror 12: 0x10d47b2b (stale) Mirror 13: 0x10d47b2b (stale) Mirror 14: 0x10d47b2b (stale) Mirror 16: 0x10d47b2b (stale) Mirror 17: 0x10d47b2b (stale) Mirror 19: 0x10d47b2b (stale) Mirror 20: 0x10d47b2b (stale) Mirror 21: 0x10d47b2b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dc5): Mirror 7: 0x10d47b2b (stale) Mirror 9: 0x10d47b2b (stale) Mirror 11: 0x10d47b2b (stale) Mirror 12: 0x10d47b2b (stale) Mirror 13: 0x10d47b2b (stale) Mirror 14: 0x10d47b2b (stale) Mirror 16: 0x10d47b2b (stale) Mirror 17: 0x10d47b2b (stale) Mirror 19: 0x10d47b2b (stale) Mirror 20: 0x10d47b2b (stale) Mirror 21: 0x10d47b2b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dc5: skipped 000442 1788685317.578498 trunc from 0x003dc5 to 0x01ba6f (0x17cab bytes) 000443 1788685317.602707 read 0x0168a8 thru 0x01ba6f (0x51c8 bytes) 000445 1788685317.612236 mapread 0x00b8d4 thru 0x01aa10 (0xf13d bytes) 000446 1788685317.634533 mapread 0x019c93 thru 0x01ba6f (0x1ddd bytes) 000448 1788685317.635793 write 0x027890 thru 0x030e02 (0x9573 bytes) 000449 1788685317.641683 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x30e03): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30e03): Mirror 7: 0xfcd93779 (stale) Mirror 9: 0xfcd93779 (stale) Mirror 11: 0xfcd93779 (stale) Mirror 12: 0xfcd93779 (stale) Mirror 13: 0xfcd93779 (stale) Mirror 14: 0xfcd93779 (stale) Mirror 16: 0xfcd93779 (stale) Mirror 17: 0xfcd93779 (stale) Mirror 19: 0xfcd93779 (stale) Mirror 20: 0xfcd93779 (stale) Mirror 21: 0xfcd93779 (stale) CRC-32 checksum value for chunk [0, 0x30e03): Mirror 1: 0xfcfdbca4 Mirror 22: 0xfcfdbca4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30e03: skipped 000450 1788685318.960539 write 0x0085e5 thru 0x00f3be (0x6dda bytes) 000452 1788685319.050634 mapread 0x02101a thru 0x02726f (0x6256 bytes) 000453 1788685319.069659 trunc from 0x030e03 to 0x00e220 (0x22be2 bytes) 000454 1788685319.087799 write 0x03eba6 thru 0x03ffff (0x145a bytes) 000458 1788685319.091521 trunc from 0x040000 to 0x018340 (0x27cbf bytes) 000459 1788685319.104141 mapread 0x00219a thru 0x008e2c (0x6c93 bytes) 000460 1788685319.111050 mapwrite 0x0061e9 thru 0x00da59 (0x7871 bytes) 000461 1788685319.187058 MIRROR_RESYNC 13 mirrors 000462 1788685322.051630 trunc from 0x018341 to 0x011bed (0x6753 bytes) 000464 1788685322.306791 punch from 0x004206 to 0x004e3d (0xc38 bytes) 000468 1788685322.333812 punch from 0x00debd to 0x011bed (0x3d31 bytes) 000470 1788685322.349531 MIRROR_SPLIT mirror 4 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11bee): Mirror 7: 0x770e5472 (stale) Mirror 9: 0x770e5472 (stale) Mirror 11: 0x770e5472 (stale) Mirror 13: 0x770e5472 (stale) Mirror 14: 0x770e5472 (stale) Mirror 16: 0x770e5472 (stale) Mirror 17: 0x770e5472 (stale) Mirror 19: 0x770e5472 (stale) Mirror 20: 0x770e5472 (stale) Mirror 21: 0x770e5472 (stale) Mirror 22: 0x770e5472 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11bee): Mirror 7: 0x770e5472 (stale) Mirror 9: 0x770e5472 (stale) Mirror 11: 0x770e5472 (stale) Mirror 13: 0x770e5472 (stale) Mirror 14: 0x770e5472 (stale) Mirror 16: 0x770e5472 (stale) Mirror 17: 0x770e5472 (stale) Mirror 19: 0x770e5472 (stale) Mirror 20: 0x770e5472 (stale) Mirror 21: 0x770e5472 (stale) Mirror 22: 0x770e5472 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11bee: skipped 000471 1788685323.224089 mapread 0x002692 thru 0x0107c6 (0xe135 bytes) 000472 1788685323.255361 trunc from 0x011bee to 0x01553d (0x3950 bytes) 000473 1788685323.277774 mapwrite 0x03ebe9 thru 0x03ffff (0x1417 bytes) 000474 1788685323.351608 mapread 0x0312fa thru 0x036a99 (0x57a0 bytes) 000475 1788685323.358792 mapwrite 0x00b34d thru 0x015335 (0x9fe9 bytes) 000476 1788685323.400192 read 0x02ff61 thru 0x0304de (0x57e bytes) 000477 1788685323.400961 mapread 0x013521 thru 0x01ab2b (0x760b bytes) 000478 1788685323.415480 write 0x01adde thru 0x01e83f (0x3a62 bytes) 000480 1788685323.417010 punch from 0x030c03 to 0x03e0da (0xd4d8 bytes) 000481 1788685323.429612 trunc from 0x040000 to 0x0212f6 (0x1ed09 bytes) 000482 1788685323.451431 mapwrite 0x0163fc thru 0x023055 (0xcc5a bytes) 000483 1788685323.577800 punch from 0x011282 to 0x01d8ec (0xc66b bytes) 000484 1788685323.588636 punch from 0x022c54 to 0x023055 (0x402 bytes) 000486 1788685323.614202 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23056): Mirror 7: 0xf8d7c652 (stale) Mirror 9: 0xf8d7c652 (stale) Mirror 11: 0xf8d7c652 (stale) Mirror 13: 0xf8d7c652 (stale) Mirror 14: 0xf8d7c652 (stale) Mirror 16: 0xf8d7c652 (stale) Mirror 19: 0xf8d7c652 (stale) Mirror 20: 0xf8d7c652 (stale) Mirror 21: 0xf8d7c652 (stale) Mirror 22: 0xf8d7c652 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23056): Mirror 7: 0xf8d7c652 (stale) Mirror 9: 0xf8d7c652 (stale) Mirror 11: 0xf8d7c652 (stale) Mirror 13: 0xf8d7c652 (stale) Mirror 14: 0xf8d7c652 (stale) Mirror 16: 0xf8d7c652 (stale) Mirror 19: 0xf8d7c652 (stale) Mirror 20: 0xf8d7c652 (stale) Mirror 21: 0xf8d7c652 (stale) Mirror 22: 0xf8d7c652 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23056: skipped 000488 1788685324.490817 write 0x012f5a thru 0x02138e (0xe435 bytes) 000489 1788685324.498641 mapread 0x018c42 thru 0x01eef6 (0x62b5 bytes) 000490 1788685324.500417 read 0x008e4a thru 0x012308 (0x94bf bytes) 000491 1788685324.510771 write 0x002502 thru 0x00ab43 (0x8642 bytes) 000492 1788685324.530841 trunc from 0x023056 to 0x0111fb (0x11e5a bytes) 000494 1788685324.559789 MIRROR_RESYNC 11 mirrors 000495 1788685326.690429 mapread 0x00ac8c thru 0x00debc (0x3231 bytes) 000496 1788685326.710479 mapwrite 0x015cf1 thru 0x017824 (0x1b34 bytes) 000497 1788685326.946590 trunc from 0x017825 to 0x0064cd (0x11357 bytes) 000498 1788685326.990426 punch from 0x004479 to 0x0064cd (0x2055 bytes) 000500 1788685326.996487 mapwrite 0x0148f3 thru 0x015974 (0x1082 bytes) 000502 1788685327.247593 read 0x009ac3 thru 0x011f06 (0x8444 bytes) 000503 1788685327.258354 punch from 0x00ca03 to 0x015974 (0x8f72 bytes) 000505 1788685327.276939 mapwrite 0x01201a thru 0x01594c (0x3933 bytes) 000507 1788685327.397086 mapread 0x00f8db thru 0x014e1b (0x5541 bytes) 000508 1788685327.414461 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15975): Mirror 7: 0xb5765b93 (stale) Mirror 9: 0xb5765b93 (stale) Mirror 11: 0xb5765b93 (stale) Mirror 14: 0xb5765b93 (stale) Mirror 16: 0xb5765b93 (stale) Mirror 19: 0xb5765b93 (stale) Mirror 20: 0xb5765b93 (stale) Mirror 21: 0xb5765b93 (stale) Mirror 22: 0xb5765b93 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15975): Mirror 7: 0xb5765b93 (stale) Mirror 9: 0xb5765b93 (stale) Mirror 11: 0xb5765b93 (stale) Mirror 14: 0xb5765b93 (stale) Mirror 16: 0xb5765b93 (stale) Mirror 19: 0xb5765b93 (stale) Mirror 20: 0xb5765b93 (stale) Mirror 21: 0xb5765b93 (stale) Mirror 22: 0xb5765b93 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15975: skipped 000509 1788685328.321791 write 0x0335c6 thru 0x03d419 (0x9e54 bytes) 000512 1788685328.325317 write 0x039015 thru 0x03ffff (0x6feb bytes) 000514 1788685328.332550 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xdfd6ad2b (stale) Mirror 9: 0xdfd6ad2b (stale) Mirror 11: 0xdfd6ad2b (stale) Mirror 14: 0xdfd6ad2b (stale) Mirror 16: 0xdfd6ad2b (stale) Mirror 19: 0xdfd6ad2b (stale) Mirror 20: 0xdfd6ad2b (stale) Mirror 21: 0xdfd6ad2b (stale) Mirror 22: 0xdfd6ad2b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf2d31cd8 Mirror 23: 0xf2d31cd8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000515 1788685330.708598 trunc from 0x040000 to 0x002b39 (0x3d4c6 bytes) 000518 1788685330.954280 read 0x0014bc thru 0x002b39 (0x167e bytes) 000520 1788685330.964293 punch from 0x0023aa to 0x002b39 (0x790 bytes) 000522 1788685330.980569 trunc from 0x002b3a to 0x03ce19 (0x3a2e0 bytes) 000523 1788685331.051277 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3ce1a): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ce1a): Mirror 7: 0xd9fee6ec (stale) Mirror 9: 0xd9fee6ec (stale) Mirror 11: 0xd9fee6ec (stale) Mirror 14: 0xd9fee6ec (stale) Mirror 16: 0xd9fee6ec (stale) Mirror 19: 0xd9fee6ec (stale) Mirror 20: 0xd9fee6ec (stale) Mirror 21: 0xd9fee6ec (stale) Mirror 22: 0xd9fee6ec (stale) Mirror 23: 0xf0891f30 (stale) CRC-32 checksum value for chunk [0, 0x3ce1a): Mirror 1: 0xe2fff1a4 Mirror 24: 0xe2fff1a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ce1a: skipped 000526 1788685333.366374 mapread 0x025330 thru 0x030f67 (0xbc38 bytes) 000528 1788685333.427187 read 0x018c6a thru 0x022fe8 (0xa37f bytes) 000529 1788685333.445065 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ce1a): Mirror 7: 0xd9fee6ec (stale) Mirror 9: 0xd9fee6ec (stale) Mirror 11: 0xd9fee6ec (stale) Mirror 14: 0xd9fee6ec (stale) Mirror 16: 0xd9fee6ec (stale) Mirror 19: 0xd9fee6ec (stale) Mirror 21: 0xd9fee6ec (stale) Mirror 22: 0xd9fee6ec (stale) Mirror 23: 0xf0891f30 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ce1a): Mirror 7: 0xd9fee6ec (stale) Mirror 9: 0xd9fee6ec (stale) Mirror 11: 0xd9fee6ec (stale) Mirror 14: 0xd9fee6ec (stale) Mirror 16: 0xd9fee6ec (stale) Mirror 19: 0xd9fee6ec (stale) Mirror 21: 0xd9fee6ec (stale) Mirror 22: 0xd9fee6ec (stale) Mirror 23: 0xf0891f30 (stale) CRC-32 checksum value for chunk [0, 0x3ce1a): Mirror 1: 0xe2fff1a4 Mirror 24: 0xe2fff1a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ce1a: skipped 000530 1788685335.216955 read 0x00244e thru 0x011f60 (0xfb13 bytes) 000533 1788685335.332396 mapwrite 0x03bf10 thru 0x03ffff (0x40f0 bytes) 000534 1788685335.735469 mapwrite 0x0129f9 thru 0x0192b3 (0x68bb bytes) 000535 1788685335.977465 read 0x019930 thru 0x01edd3 (0x54a4 bytes) 000536 1788685336.027536 trunc from 0x040000 to 0x00f6ab (0x30954 bytes) 000537 1788685336.084662 mapread 0x004b54 thru 0x009dae (0x525b bytes) 000538 1788685336.085699 trunc from 0x00f6ac to 0x00c75a (0x2f51 bytes) 000539 1788685336.168224 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc75b): Mirror 7: 0x5cc41195 (stale) Mirror 9: 0x5cc41195 (stale) Mirror 11: 0x5cc41195 (stale) Mirror 14: 0x5cc41195 (stale) Mirror 16: 0x5cc41195 (stale) Mirror 19: 0x5cc41195 (stale) Mirror 22: 0x5cc41195 (stale) Mirror 23: 0xca0c8e9e (stale) Mirror 24: 0x3b041eb6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc75b): Mirror 7: 0x5cc41195 (stale) Mirror 9: 0x5cc41195 (stale) Mirror 11: 0x5cc41195 (stale) Mirror 14: 0x5cc41195 (stale) Mirror 16: 0x5cc41195 (stale) Mirror 19: 0x5cc41195 (stale) Mirror 22: 0x5cc41195 (stale) Mirror 23: 0xca0c8e9e (stale) Mirror 24: 0x3b041eb6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc75b: skipped 000540 1788685337.243786 read 0x00ae49 thru 0x00c75a (0x1912 bytes) 000541 1788685337.250367 punch from 0x005926 to 0x00c75a (0x6e35 bytes) 000542 1788685337.259026 read 0x0039f9 thru 0x00b589 (0x7b91 bytes) 000543 1788685337.267385 trunc from 0x00c75b to 0x002fed (0x976d bytes) 000544 1788685337.284517 punch from 0x001028 to 0x002fed (0x1fc6 bytes) 000545 1788685337.289664 punch from 0x0021ed to 0x002b00 (0x914 bytes) 000546 1788685337.294574 read 0x00067e thru 0x002fed (0x2970 bytes) 000548 1788685337.302347 mapwrite 0x020c26 thru 0x02afc6 (0xa3a1 bytes) 000549 1788685337.409340 trunc from 0x02afc7 to 0x02e574 (0x35ae bytes) 000551 1788685337.433026 punch from 0x019381 to 0x019616 (0x296 bytes) 000553 1788685337.477486 MIRROR_RESYNC 10 mirrors 000554 1788685341.339277 MIRROR_RESYNC 10 mirrors 000555 1788685342.406078 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) 000558 1788685343.037370 write 0x002558 thru 0x006bec (0x4695 bytes) 000561 1788685343.123665 mapread 0x017469 thru 0x0229e2 (0xb57a bytes) 000562 1788685343.146916 trunc from 0x02e575 to 0x03e08f (0xfb1b bytes) 000563 1788685343.165070 read 0x036446 thru 0x03e08f (0x7c4a bytes) 000564 1788685343.174593 write 0x033974 thru 0x0365d0 (0x2c5d bytes) 000565 1788685343.178578 trunc from 0x03e090 to 0x034e17 (0x9278 bytes) 000568 1788685343.190595 read 0x01b5d3 thru 0x01df50 (0x297e bytes) 000569 1788685343.191436 read 0x01b4bf thru 0x0273da (0xbf1c bytes) 000572 1788685343.193682 read 0x02614c thru 0x026299 (0x14e bytes) 000573 1788685343.194383 mapwrite 0x02b490 thru 0x039b04 (0xe675 bytes) 000574 1788685343.254687 mapwrite 0x023b5d thru 0x02c462 (0x8906 bytes) 000576 1788685343.282615 punch from 0x02bb0c to 0x037dd7 (0xc2cc bytes) 000578 1788685343.286837 trunc from 0x039b05 to 0x02c98f (0xd175 bytes) 000579 1788685343.298102 write 0x005265 thru 0x011944 (0xc6e0 bytes) 000580 1788685343.299254 mapread 0x00816a thru 0x016ca3 (0xeb3a bytes) 000581 1788685343.304279 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c990): Mirror 9: 0x9d9f89df (stale) Mirror 11: 0x9d9f89df (stale) Mirror 14: 0x9d9f89df (stale) Mirror 16: 0x9d9f89df (stale) Mirror 19: 0x9d9f89df (stale) Mirror 23: 0x9d9f89df (stale) Mirror 24: 0x9d9f89df (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c990): Mirror 9: 0x9d9f89df (stale) Mirror 11: 0x9d9f89df (stale) Mirror 14: 0x9d9f89df (stale) Mirror 16: 0x9d9f89df (stale) Mirror 19: 0x9d9f89df (stale) Mirror 23: 0x9d9f89df (stale) Mirror 24: 0x9d9f89df (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c990: skipped 000585 1788685343.665966 mapread 0x001db3 thru 0x0063ff (0x464d bytes) 000587 1788685343.695076 trunc from 0x02c990 to 0x036fa8 (0xa619 bytes) 000589 1788685343.714637 read 0x025ea0 thru 0x0291bd (0x331e bytes) 000592 1788685343.723509 mapwrite 0x00d75d thru 0x00f17f (0x1a23 bytes) 000593 1788685343.760280 punch from 0x000783 to 0x00d672 (0xcef0 bytes) 000594 1788685343.770379 MIRROR_RESYNC 8 mirrors 000595 1788685346.359592 read 0x035c16 thru 0x036fa8 (0x1393 bytes) 000596 1788685346.371963 write 0x00a664 thru 0x01312f (0x8acc bytes) 000598 1788685346.536354 read 0x02783a thru 0x02ac0d (0x33d4 bytes) 000599 1788685346.549334 trunc from 0x036fa9 to 0x031820 (0x5788 bytes) 000600 1788685346.569252 trunc from 0x031821 to 0x001d42 (0x2fade bytes) 000601 1788685346.618696 MIRROR_RESYNC 8 mirrors 000602 1788685347.465750 read 0x001757 thru 0x001d42 (0x5ec bytes) 000604 1788685347.470390 punch from 0x000f32 to 0x001d42 (0xe11 bytes) 000606 1788685347.532575 MIRROR_RESYNC 8 mirrors 000607 1788685348.609360 read 0x001b6d thru 0x001d42 (0x1d6 bytes) 000610 1788685348.628858 read 0x000651 thru 0x001d42 (0x16f2 bytes) 000614 1788685348.642223 write 0x0201fe thru 0x02c56b (0xc36e bytes) 000616 1788685348.731365 write 0x001370 thru 0x00294f (0x15e0 bytes) 000618 1788685348.741078 punch from 0x02c262 to 0x02c56b (0x30a bytes) 000619 1788685348.755575 write 0x00794e thru 0x0090b5 (0x1768 bytes) 000620 1788685348.757175 mapread 0x025394 thru 0x02bb4f (0x67bc bytes) 000622 1788685348.758713 write 0x022b29 thru 0x0231e0 (0x6b8 bytes) 000623 1788685348.760535 write 0x0134aa thru 0x01dbe0 (0xa737 bytes) 000624 1788685348.762342 trunc from 0x02c56c to 0x00ace3 (0x21888 bytes) 000625 1788685348.782070 mapread 0x009aff thru 0x00ace3 (0x11e5 bytes) 000626 1788685348.782918 read 0x007a3a thru 0x00ac72 (0x3239 bytes) 000629 1788685348.785285 write 0x02a02b thru 0x0310b7 (0x708d bytes) 000630 1788685348.787621 write 0x024140 thru 0x02a52c (0x63ed bytes) 000631 1788685348.792811 mapread 0x014118 thru 0x01fa9f (0xb988 bytes) 000632 1788685348.793683 punch from 0x00e79d to 0x0127b7 (0x401b bytes) 000633 1788685348.804875 trunc from 0x0310b8 to 0x0144ad (0x1cc0a bytes) 000634 1788685348.817215 punch from 0x00604a to 0x0144ad (0xe464 bytes) 000635 1788685348.827321 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x144ae): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x144ae): Mirror 9: 0xbe2e94f2 (stale) Mirror 11: 0xbe2e94f2 (stale) Mirror 14: 0xbe2e94f2 (stale) Mirror 16: 0xbe2e94f2 (stale) Mirror 19: 0xbe2e94f2 (stale) Mirror 23: 0xbe2e94f2 (stale) Mirror 24: 0xbe2e94f2 (stale) CRC-32 checksum value for chunk [0, 0x144ae): Mirror 1: 0xdc1450eb Mirror 25: 0xdc1450eb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x144ae: skipped 000636 1788685350.109255 MIRROR_RESYNC 9 mirrors 000637 1788685352.214161 read 0x010c26 thru 0x0144ad (0x3888 bytes) 000641 1788685352.238616 MIRROR_EXTEND to 10 mirrors 000643 1788685353.292377 punch from 0x00a62c to 0x0144ad (0x9e82 bytes) 000647 1788685353.391506 write 0x007489 thru 0x00ffd4 (0x8b4c bytes) 000648 1788685353.402065 read 0x00eb5b thru 0x012d0f (0x41b5 bytes) 000649 1788685353.407968 mapwrite 0x01cd00 thru 0x01e41a (0x171b bytes) 000654 1788685353.582893 read 0x017e0b thru 0x019b3c (0x1d32 bytes) 000655 1788685353.584086 read 0x013438 thru 0x01ca2b (0x95f4 bytes) 000656 1788685353.612595 mapwrite 0x01b2d3 thru 0x026cea (0xba18 bytes) 000657 1788685353.779378 mapwrite 0x011f16 thru 0x017e6a (0x5f55 bytes) 000658 1788685353.824955 write 0x03b6da thru 0x03ffff (0x4926 bytes) 000659 1788685353.826568 write 0x0245e6 thru 0x02a1fc (0x5c17 bytes) 000660 1788685353.831087 punch from 0x0069b9 to 0x0159ce (0xf016 bytes) 000663 1788685353.847638 mapwrite 0x010e96 thru 0x01279d (0x1908 bytes) 000664 1788685353.939382 punch from 0x0247a6 to 0x02bc34 (0x748f bytes) 000665 1788685353.944970 read 0x025a95 thru 0x02feb5 (0xa421 bytes) 000666 1788685353.973468 write 0x034be8 thru 0x039ab4 (0x4ecd bytes) 000667 1788685353.974854 mapwrite 0x016ed8 thru 0x01a4ca (0x35f3 bytes) 000668 1788685354.029503 trunc from 0x040000 to 0x016ae4 (0x2951b bytes) 000670 1788685354.052356 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x16ae5): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16ae5): Mirror 9: 0xf43a394c (stale) Mirror 11: 0xf43a394c (stale) Mirror 14: 0xf43a394c (stale) Mirror 16: 0xf43a394c (stale) Mirror 19: 0xf43a394c (stale) Mirror 23: 0xf43a394c (stale) Mirror 24: 0xf43a394c (stale) Mirror 25: 0xf43a394c (stale) Mirror 26: 0xf43a394c (stale) CRC-32 checksum value for chunk [0, 0x16ae5): Mirror 1: 0xb856f6a1 Mirror 27: 0xb856f6a1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16ae5: skipped 000672 1788685355.058120 trunc from 0x016ae5 to 0x0227d5 (0xbcf1 bytes) 000673 1788685355.160811 mapwrite 0x039e46 thru 0x03c46c (0x2627 bytes) 000674 1788685355.396660 read 0x013877 thru 0x016af6 (0x3280 bytes) 000675 1788685355.406253 trunc from 0x03c46d to 0x032f9b (0x94d1 bytes) 000676 1788685355.422307 write 0x00c102 thru 0x018470 (0xc36f bytes) 000677 1788685355.435253 trunc from 0x032f9c to 0x03d7c6 (0xa82b bytes) 000678 1788685355.516298 trunc from 0x03d7c7 to 0x01e6f2 (0x1f0d4 bytes) 000679 1788685355.576521 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e6f3): Mirror 9: 0xb586d5c2 (stale) Mirror 11: 0xb586d5c2 (stale) Mirror 14: 0xb586d5c2 (stale) Mirror 16: 0xb586d5c2 (stale) Mirror 19: 0xb586d5c2 (stale) Mirror 23: 0xb586d5c2 (stale) Mirror 24: 0xb586d5c2 (stale) Mirror 25: 0xb586d5c2 (stale) Mirror 26: 0xb586d5c2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e6f3): Mirror 9: 0xb586d5c2 (stale) Mirror 11: 0xb586d5c2 (stale) Mirror 14: 0xb586d5c2 (stale) Mirror 16: 0xb586d5c2 (stale) Mirror 19: 0xb586d5c2 (stale) Mirror 23: 0xb586d5c2 (stale) Mirror 24: 0xb586d5c2 (stale) Mirror 25: 0xb586d5c2 (stale) Mirror 26: 0xb586d5c2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e6f3: skipped 000680 1788685356.243729 trunc from 0x01e6f3 to 0x00c2a0 (0x12452 bytes) 000681 1788685356.262382 read 0x00a3b0 thru 0x00c2a0 (0x1ef1 bytes) 000682 1788685356.275436 mapwrite 0x037a7a thru 0x03ffff (0x8586 bytes) 000683 1788685356.444162 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x96d1e4a1 (stale) Mirror 11: 0x96d1e4a1 (stale) Mirror 14: 0x96d1e4a1 (stale) Mirror 16: 0x96d1e4a1 (stale) Mirror 19: 0x96d1e4a1 (stale) Mirror 23: 0x96d1e4a1 (stale) Mirror 24: 0x96d1e4a1 (stale) Mirror 25: 0x96d1e4a1 (stale) Mirror 26: 0x96d1e4a1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x80673e45 Mirror 27: 0x80673e45 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000684 1788685357.693497 read 0x01b8c2 thru 0x026b2e (0xb26d bytes) 000685 1788685357.716599 MIRROR_RESYNC 11 mirrors 000687 1788685359.780921 punch from 0x00b065 to 0x00df14 (0x2eb0 bytes) 000688 1788685359.865476 read 0x028ed0 thru 0x033baa (0xacdb bytes) 000689 1788685359.875618 mapwrite 0x01971a thru 0x01a019 (0x900 bytes) 000690 1788685359.939777 mapwrite 0x00b2a5 thru 0x019da5 (0xeb01 bytes) 000691 1788685360.026929 mapwrite 0x00912d thru 0x00b683 (0x2557 bytes) 000693 1788685360.070021 trunc from 0x040000 to 0x010897 (0x2f768 bytes) 000694 1788685360.089024 punch from 0x00fcbb to 0x010897 (0xbdd bytes) 000695 1788685360.095372 mapwrite 0x00be7f thru 0x00fc11 (0x3d93 bytes) 000697 1788685360.154673 mapwrite 0x00af3f thru 0x0101a6 (0x5268 bytes) 000698 1788685360.213790 mapwrite 0x02ee3a thru 0x032acb (0x3c92 bytes) 000701 1788685360.287025 write 0x015c1c thru 0x016362 (0x747 bytes) 000703 1788685360.297904 mapread 0x000414 thru 0x00676d (0x635a bytes) 000704 1788685360.305834 read 0x005ca6 thru 0x009f69 (0x42c4 bytes) 000705 1788685360.306461 write 0x03e437 thru 0x03ffff (0x1bc9 bytes) 000708 1788685360.311371 punch from 0x006042 to 0x00a28e (0x424d bytes) 000710 1788685360.324685 mapread 0x015770 thru 0x01c951 (0x71e2 bytes) 000711 1788685360.334392 read 0x016713 thru 0x01a709 (0x3ff7 bytes) 000712 1788685360.335000 trunc from 0x040000 to 0x037027 (0x8fd8 bytes) 000713 1788685360.356345 trunc from 0x037028 to 0x014f35 (0x220f2 bytes) 000715 1788685360.379044 mapread 0x00ded9 thru 0x014f35 (0x705d bytes) 000716 1788685360.380021 punch from 0x000fc3 to 0x00e8ea (0xd928 bytes) 000717 1788685360.386522 mapread 0x005e80 thru 0x00c948 (0x6ac9 bytes) 000719 1788685360.407042 write 0x016e44 thru 0x0205ce (0x978b bytes) 000720 1788685360.409494 trunc from 0x0205cf to 0x006017 (0x1a5b7 bytes) 000722 1788685360.454822 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6018): Mirror 11: 0x243e4710 (stale) Mirror 14: 0x243e4710 (stale) Mirror 16: 0x243e4710 (stale) Mirror 19: 0x243e4710 (stale) Mirror 23: 0x243e4710 (stale) Mirror 24: 0x243e4710 (stale) Mirror 25: 0x243e4710 (stale) Mirror 26: 0x243e4710 (stale) Mirror 27: 0x243e4710 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6018): Mirror 11: 0x243e4710 (stale) Mirror 14: 0x243e4710 (stale) Mirror 16: 0x243e4710 (stale) Mirror 19: 0x243e4710 (stale) Mirror 23: 0x243e4710 (stale) Mirror 24: 0x243e4710 (stale) Mirror 25: 0x243e4710 (stale) Mirror 26: 0x243e4710 (stale) Mirror 27: 0x243e4710 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6018: skipped 000723 1788685361.255230 mapread 0x0029b0 thru 0x006017 (0x3668 bytes) 000725 1788685361.310490 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x6018): Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6018): Mirror 11: 0x243e4710 (stale) Mirror 14: 0x243e4710 (stale) Mirror 16: 0x243e4710 (stale) Mirror 19: 0x243e4710 (stale) Mirror 23: 0x243e4710 (stale) Mirror 24: 0x243e4710 (stale) Mirror 25: 0x243e4710 (stale) Mirror 26: 0x243e4710 (stale) Mirror 27: 0x243e4710 (stale) CRC-32 checksum value for chunk [0, 0x6018): Mirror 1: 0xa329aaca Mirror 28: 0xa329aaca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6018: skipped 000727 1788685362.355044 trunc from 0x006018 to 0x02cce1 (0x26cca bytes) 000730 1788685362.468173 mapread 0x021f6b thru 0x027102 (0x5198 bytes) 000732 1788685362.494715 trunc from 0x02cce2 to 0x013f8c (0x18d55 bytes) 000733 1788685362.519684 mapread 0x011b03 thru 0x013f8c (0x248a bytes) 000734 1788685362.532947 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x13f8d): Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13f8d): Mirror 11: 0xcbf99e9a (stale) Mirror 14: 0xcbf99e9a (stale) Mirror 16: 0xcbf99e9a (stale) Mirror 19: 0xcbf99e9a (stale) Mirror 23: 0xcbf99e9a (stale) Mirror 24: 0xcbf99e9a (stale) Mirror 25: 0xcbf99e9a (stale) Mirror 26: 0xcbf99e9a (stale) Mirror 27: 0xcbf99e9a (stale) Mirror 28: 0xaf1426c5 (stale) CRC-32 checksum value for chunk [0, 0x13f8d): Mirror 1: 0xaf1426c5 Mirror 29: 0xaf1426c5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13f8d: skipped 000735 1788685363.607192 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x13f8d): Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13f8d): Mirror 11: 0xcbf99e9a (stale) Mirror 14: 0xcbf99e9a (stale) Mirror 16: 0xcbf99e9a (stale) Mirror 19: 0xcbf99e9a (stale) Mirror 23: 0xcbf99e9a (stale) Mirror 24: 0xcbf99e9a (stale) Mirror 25: 0xcbf99e9a (stale) Mirror 26: 0xcbf99e9a (stale) Mirror 27: 0xcbf99e9a (stale) Mirror 28: 0xaf1426c5 (stale) CRC-32 checksum value for chunk [0, 0x13f8d): Mirror 1: 0xaf1426c5 Mirror 29: 0xaf1426c5 Mirror 30: 0xaf1426c5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13f8d: skipped 000736 1788685364.753102 MIRROR_RESYNC 13 mirrors 000738 1788685366.836079 mapread 0x007123 thru 0x007ab4 (0x992 bytes) 000739 1788685366.841877 trunc from 0x013f8d to 0x03e919 (0x2a98d bytes) 000740 1788685366.950397 write 0x03a3ba thru 0x03ffff (0x5c46 bytes) 000741 1788685366.955372 write 0x0067f4 thru 0x00d165 (0x6972 bytes) 000742 1788685366.966650 mapread 0x017e62 thru 0x02359c (0xb73b bytes) 000743 1788685367.009314 mapread 0x017795 thru 0x022c7b (0xb4e7 bytes) 000744 1788685367.011296 trunc from 0x040000 to 0x03cce2 (0x331d bytes) 000745 1788685367.041301 write 0x01d34b thru 0x026943 (0x95f9 bytes) 000746 1788685367.044079 write 0x038ac7 thru 0x03ffff (0x7539 bytes) 000747 1788685367.052669 read 0x01690f thru 0x01fc5d (0x934f bytes) 000748 1788685367.054221 mapread 0x037519 thru 0x03ffff (0x8ae7 bytes) 000749 1788685367.063219 punch from 0x026c50 to 0x036961 (0xfd12 bytes) 000752 1788685367.082138 punch from 0x039cdf to 0x03ffff (0x6321 bytes) 000753 1788685367.087672 punch from 0x02ee99 to 0x02fed4 (0x103c bytes) 000754 1788685367.093165 read 0x02214a thru 0x029c46 (0x7afd bytes) 000759 1788685367.113325 write 0x03670e thru 0x03f6c5 (0x8fb8 bytes) 000760 1788685367.115870 write 0x03e23b thru 0x03e9a8 (0x76e bytes) 000762 1788685367.123101 mapwrite 0x03f675 thru 0x03ffff (0x98b bytes) 000763 1788685367.212381 mapread 0x0127f5 thru 0x015934 (0x3140 bytes) 000766 1788685367.216264 read 0x02657f thru 0x031288 (0xad0a bytes) 000767 1788685367.222309 MIRROR_RESYNC 13 mirrors 000768 1788685369.421880 mapwrite 0x03fc81 thru 0x03ffff (0x37f bytes) 000771 1788685369.663597 mapwrite 0x02ce2a thru 0x0321dc (0x53b3 bytes) 000774 1788685369.792477 mapwrite 0x03a0fb thru 0x03ffff (0x5f05 bytes) 000775 1788685369.846189 write 0x03d6a0 thru 0x03ffff (0x2960 bytes) 000777 1788685369.853778 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xa78b16c5 (stale) Mirror 14: 0xa78b16c5 (stale) Mirror 16: 0xa78b16c5 (stale) Mirror 19: 0xa78b16c5 (stale) Mirror 23: 0xa78b16c5 (stale) Mirror 24: 0xa78b16c5 (stale) Mirror 25: 0xa78b16c5 (stale) Mirror 26: 0xa78b16c5 (stale) Mirror 27: 0xa78b16c5 (stale) Mirror 28: 0xa78b16c5 (stale) Mirror 29: 0xa78b16c5 (stale) Mirror 30: 0xa78b16c5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5e0aa250 Mirror 31: 0x5e0aa250 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000778 1788685371.109977 write 0x0185a4 thru 0x0237a1 (0xb1fe bytes) 000779 1788685371.310078 mapread 0x01bbc6 thru 0x02b561 (0xf99c bytes) 000780 1788685371.353006 punch from 0x01c115 to 0x0230cc (0x6fb8 bytes) 000781 1788685371.379298 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xa78b16c5 (stale) Mirror 14: 0xa78b16c5 (stale) Mirror 16: 0xa78b16c5 (stale) Mirror 19: 0xa78b16c5 (stale) Mirror 23: 0xa78b16c5 (stale) Mirror 24: 0xa78b16c5 (stale) Mirror 25: 0xa78b16c5 (stale) Mirror 26: 0xa78b16c5 (stale) Mirror 27: 0xa78b16c5 (stale) Mirror 28: 0xa78b16c5 (stale) Mirror 29: 0xa78b16c5 (stale) Mirror 30: 0xa78b16c5 (stale) Mirror 31: 0x5e0aa250 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc7e27bf2 Mirror 32: 0xc7e27bf2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000782 1788685372.699824 write 0x02a95d thru 0x038b2c (0xe1d0 bytes) 000783 1788685372.878924 mapread 0x025fe8 thru 0x02c455 (0x646e bytes) 000784 1788685372.894887 mapread 0x00d42c thru 0x0184fa (0xb0cf bytes) 000785 1788685372.909526 punch from 0x00797d to 0x010c69 (0x92ed bytes) 000786 1788685372.921684 mapread 0x00e771 thru 0x01642e (0x7cbe bytes) 000787 1788685372.933292 mapwrite 0x0133db thru 0x01b232 (0x7e58 bytes) 000788 1788685372.979609 read 0x032ed1 thru 0x03ffff (0xd12f bytes) 000789 1788685372.989388 read 0x011dde thru 0x017bab (0x5dce bytes) 000790 1788685372.990422 mapread 0x013df9 thru 0x018873 (0x4a7b bytes) 000791 1788685372.992241 read 0x02f004 thru 0x034ca8 (0x5ca5 bytes) 000792 1788685372.993071 punch from 0x03b21c to 0x03ffff (0x4de4 bytes) 000793 1788685373.001587 MIRROR_SPLIT mirror 3 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xa78b16c5 (stale) Mirror 14: 0xa78b16c5 (stale) Mirror 19: 0xa78b16c5 (stale) Mirror 23: 0xa78b16c5 (stale) Mirror 24: 0xa78b16c5 (stale) Mirror 25: 0xa78b16c5 (stale) Mirror 26: 0xa78b16c5 (stale) Mirror 27: 0xa78b16c5 (stale) Mirror 28: 0xa78b16c5 (stale) Mirror 29: 0xa78b16c5 (stale) Mirror 30: 0xa78b16c5 (stale) Mirror 31: 0x5e0aa250 (stale) Mirror 32: 0xc7e27bf2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xa78b16c5 (stale) Mirror 14: 0xa78b16c5 (stale) Mirror 19: 0xa78b16c5 (stale) Mirror 23: 0xa78b16c5 (stale) Mirror 24: 0xa78b16c5 (stale) Mirror 25: 0xa78b16c5 (stale) Mirror 26: 0xa78b16c5 (stale) Mirror 27: 0xa78b16c5 (stale) Mirror 28: 0xa78b16c5 (stale) Mirror 29: 0xa78b16c5 (stale) Mirror 30: 0xa78b16c5 (stale) Mirror 31: 0x5e0aa250 (stale) Mirror 32: 0xc7e27bf2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000794 1788685373.838180 mapwrite 0x01149a thru 0x01f4f0 (0xe057 bytes) 000795 1788685373.941123 trunc from 0x040000 to 0x008024 (0x37fdb bytes) 000798 1788685373.991762 read 0x00564f thru 0x008024 (0x29d6 bytes) 000800 1788685374.000678 read 0x004180 thru 0x006dc4 (0x2c45 bytes) 000801 1788685374.019110 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x8025): Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8025): Mirror 11: 0xda69ee8a (stale) Mirror 14: 0xda69ee8a (stale) Mirror 19: 0xda69ee8a (stale) Mirror 23: 0xda69ee8a (stale) Mirror 24: 0xda69ee8a (stale) Mirror 25: 0xda69ee8a (stale) Mirror 26: 0xda69ee8a (stale) Mirror 27: 0xda69ee8a (stale) Mirror 28: 0xda69ee8a (stale) Mirror 29: 0xda69ee8a (stale) Mirror 30: 0xda69ee8a (stale) Mirror 31: 0xda69ee8a (stale) Mirror 32: 0xda69ee8a (stale) CRC-32 checksum value for chunk [0, 0x8025): Mirror 1: 0x842bb75b Mirror 33: 0x842bb75b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8025: skipped 000802 1788685375.677396 trunc from 0x008025 to 0x02b910 (0x238ec bytes) 000803 1788685375.777395 trunc from 0x02b911 to 0x00726f (0x246a1 bytes) 000804 1788685375.801496 trunc from 0x007270 to 0x002dbd (0x44b2 bytes) 000806 1788685375.816873 trunc from 0x002dbe to 0x0203ba (0x1d5fd bytes) 000807 1788685375.832073 mapwrite 0x019919 thru 0x023cf2 (0xa3da bytes) 000809 1788685375.925856 mapwrite 0x03653c thru 0x037c81 (0x1746 bytes) 000810 1788685375.989436 mapread 0x017c8f thru 0x018dc4 (0x1136 bytes) 000811 1788685375.996017 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x37c82): Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37c82): Mirror 11: 0x10991441 (stale) Mirror 14: 0x10991441 (stale) Mirror 19: 0x10991441 (stale) Mirror 23: 0x10991441 (stale) Mirror 24: 0x10991441 (stale) Mirror 25: 0x10991441 (stale) Mirror 26: 0x10991441 (stale) Mirror 27: 0x10991441 (stale) Mirror 28: 0x10991441 (stale) Mirror 29: 0x10991441 (stale) Mirror 30: 0x10991441 (stale) Mirror 31: 0x8f93d920 (stale) Mirror 32: 0x1208af23 (stale) Mirror 33: 0xca91e560 (stale) CRC-32 checksum value for chunk [0, 0x37c82): Mirror 1: 0xa5e5d407 Mirror 34: 0xa5e5d407 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37c82: skipped 000813 1788685377.143716 mapread 0x0190d6 thru 0x01c818 (0x3743 bytes) 000814 1788685377.203876 punch from 0x029b9b to 0x031c1d (0x8083 bytes) 000815 1788685377.316607 punch from 0x01fb36 to 0x02be83 (0xc34e bytes) 000817 1788685377.327665 mapread 0x00f8de thru 0x01026f (0x992 bytes) 000818 1788685377.342765 MIRROR_SPLIT mirror 1 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37c82): Mirror 14: 0x10991441 (stale) Mirror 19: 0x10991441 (stale) Mirror 23: 0x10991441 (stale) Mirror 24: 0x10991441 (stale) Mirror 25: 0x10991441 (stale) Mirror 26: 0x10991441 (stale) Mirror 27: 0x10991441 (stale) Mirror 28: 0x10991441 (stale) Mirror 29: 0x10991441 (stale) Mirror 30: 0x10991441 (stale) Mirror 31: 0x8f93d920 (stale) Mirror 32: 0x1208af23 (stale) Mirror 33: 0xca91e560 (stale) Mirror 34: 0xa5e5d407 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37c82): Mirror 14: 0x10991441 (stale) Mirror 19: 0x10991441 (stale) Mirror 23: 0x10991441 (stale) Mirror 24: 0x10991441 (stale) Mirror 25: 0x10991441 (stale) Mirror 26: 0x10991441 (stale) Mirror 27: 0x10991441 (stale) Mirror 28: 0x10991441 (stale) Mirror 29: 0x10991441 (stale) Mirror 30: 0x10991441 (stale) Mirror 31: 0x8f93d920 (stale) Mirror 32: 0x1208af23 (stale) Mirror 33: 0xca91e560 (stale) Mirror 34: 0xa5e5d407 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37c82: skipped 000819 1788685378.154594 mapread 0x02ceb0 thru 0x035dd1 (0x8f22 bytes) 000820 1788685378.172429 mapread 0x036468 thru 0x037c81 (0x181a bytes) 000821 1788685378.178613 read 0x02b2fd thru 0x030b82 (0x5886 bytes) 000824 1788685378.180532 punch from 0x00c7d4 to 0x01465a (0x7e87 bytes) 000825 1788685378.191374 mapread 0x02e8cc thru 0x033ddc (0x5511 bytes) 000830 1788685378.198829 mapread 0x01fa43 thru 0x02859a (0x8b58 bytes) 000831 1788685378.212163 punch from 0x0228fa to 0x031478 (0xeb7f bytes) 000833 1788685378.218396 mapwrite 0x00e2cd thru 0x00ff6c (0x1ca0 bytes) 000835 1788685378.300778 MIRROR_RESYNC 15 mirrors 000836 1788685380.538778 read 0x02d2e0 thru 0x037c81 (0xa9a2 bytes) 000837 1788685380.549458 read 0x029488 thru 0x037c81 (0xe7fa bytes) 000840 1788685380.574121 mapread 0x02f455 thru 0x03111f (0x1ccb bytes) 000842 1788685380.574936 punch from 0x002c7a to 0x0062c5 (0x364c bytes) 000843 1788685380.644360 mapread 0x02d7be thru 0x036b9f (0x93e2 bytes) 000844 1788685380.697299 trunc from 0x037c82 to 0x02fb28 (0x8159 bytes) 000846 1788685380.721959 read 0x009d5d thru 0x015da8 (0xc04c bytes) 000848 1788685380.737194 MIRROR_RESYNC 15 mirrors 000849 1788685383.027569 read 0x0243e8 thru 0x02e51b (0xa134 bytes) 000851 1788685383.038056 mapread 0x01b882 thru 0x02308b (0x780a bytes) 000852 1788685383.061367 punch from 0x0033a4 to 0x010477 (0xd0d4 bytes) 000853 1788685383.193365 mapwrite 0x01d2be thru 0x02222d (0x4f70 bytes) 000855 1788685383.283171 punch from 0x01fce6 to 0x02298e (0x2ca9 bytes) 000857 1788685383.293109 MIRROR_SPLIT mirror 10 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fb29): Mirror 14: 0xf5da7081 (stale) Mirror 19: 0xf5da7081 (stale) Mirror 23: 0xf5da7081 (stale) Mirror 24: 0xf5da7081 (stale) Mirror 25: 0xf5da7081 (stale) Mirror 26: 0xf5da7081 (stale) Mirror 27: 0xf5da7081 (stale) Mirror 28: 0xf5da7081 (stale) Mirror 29: 0xf5da7081 (stale) Mirror 31: 0xf5da7081 (stale) Mirror 32: 0xf5da7081 (stale) Mirror 33: 0xf5da7081 (stale) Mirror 34: 0xf5da7081 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fb29): Mirror 14: 0xf5da7081 (stale) Mirror 19: 0xf5da7081 (stale) Mirror 23: 0xf5da7081 (stale) Mirror 24: 0xf5da7081 (stale) Mirror 25: 0xf5da7081 (stale) Mirror 26: 0xf5da7081 (stale) Mirror 27: 0xf5da7081 (stale) Mirror 28: 0xf5da7081 (stale) Mirror 29: 0xf5da7081 (stale) Mirror 31: 0xf5da7081 (stale) Mirror 32: 0xf5da7081 (stale) Mirror 33: 0xf5da7081 (stale) Mirror 34: 0xf5da7081 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fb29: skipped 000858 1788685384.102128 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x2fb29): Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fb29): Mirror 14: 0xf5da7081 (stale) Mirror 19: 0xf5da7081 (stale) Mirror 23: 0xf5da7081 (stale) Mirror 24: 0xf5da7081 (stale) Mirror 25: 0xf5da7081 (stale) Mirror 26: 0xf5da7081 (stale) Mirror 27: 0xf5da7081 (stale) Mirror 28: 0xf5da7081 (stale) Mirror 29: 0xf5da7081 (stale) Mirror 31: 0xf5da7081 (stale) Mirror 32: 0xf5da7081 (stale) Mirror 33: 0xf5da7081 (stale) Mirror 34: 0xf5da7081 (stale) CRC-32 checksum value for chunk [0, 0x2fb29): Mirror 1: 0xc7e2cd93 Mirror 35: 0xc7e2cd93 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fb29: skipped 000859 1788685385.260532 read 0x00cf50 thru 0x01342c (0x64dd bytes) 000860 1788685385.288133 MIRROR_SPLIT mirror 13 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fb29): Mirror 14: 0xf5da7081 (stale) Mirror 19: 0xf5da7081 (stale) Mirror 23: 0xf5da7081 (stale) Mirror 24: 0xf5da7081 (stale) Mirror 25: 0xf5da7081 (stale) Mirror 26: 0xf5da7081 (stale) Mirror 27: 0xf5da7081 (stale) Mirror 28: 0xf5da7081 (stale) Mirror 29: 0xf5da7081 (stale) Mirror 31: 0xf5da7081 (stale) Mirror 32: 0xf5da7081 (stale) Mirror 33: 0xf5da7081 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fb29): Mirror 14: 0xf5da7081 (stale) Mirror 19: 0xf5da7081 (stale) Mirror 23: 0xf5da7081 (stale) Mirror 24: 0xf5da7081 (stale) Mirror 25: 0xf5da7081 (stale) Mirror 26: 0xf5da7081 (stale) Mirror 27: 0xf5da7081 (stale) Mirror 28: 0xf5da7081 (stale) Mirror 29: 0xf5da7081 (stale) Mirror 31: 0xf5da7081 (stale) Mirror 32: 0xf5da7081 (stale) Mirror 33: 0xf5da7081 (stale) CRC-32 checksum value for chunk [0, 0x2fb29): Mirror 1: 0xc7e2cd93 Mirror 35: 0xc7e2cd93 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fb29: skipped 000861 1788685386.456865 MIRROR_RESYNC 14 mirrors 000863 1788685388.695016 trunc from 0x02fb29 to 0x00ec09 (0x20f1f bytes) 000864 1788685388.898009 mapwrite 0x03fc03 thru 0x03ffff (0x3fd bytes) 000865 1788685389.081100 read 0x00cac7 thru 0x00f4a2 (0x29dc bytes) 000866 1788685389.092013 mapread 0x00e803 thru 0x015599 (0x6d97 bytes) 000868 1788685389.105588 mapread 0x0377c4 thru 0x03ffff (0x883c bytes) 000869 1788685389.117300 read 0x03fdba thru 0x03ffff (0x246 bytes) 000870 1788685389.118015 punch from 0x03c742 to 0x03ffff (0x38be bytes) 000871 1788685389.126503 mapread 0x02dfae thru 0x03a34d (0xc3a0 bytes) 000873 1788685389.134131 read 0x019b10 thru 0x019dae (0x29f bytes) 000874 1788685389.134882 read 0x033d87 thru 0x038524 (0x479e bytes) 000875 1788685389.135242 trunc from 0x040000 to 0x03374f (0xc8b0 bytes) 000876 1788685389.149969 read 0x001cb8 thru 0x002c91 (0xfda bytes) 000877 1788685389.151012 punch from 0x008d3c to 0x014d77 (0xc03c bytes) 000878 1788685389.155512 mapwrite 0x03dc5f thru 0x03ffff (0x23a1 bytes) 000879 1788685389.250277 punch from 0x001e3a to 0x00a99e (0x8b65 bytes) 000880 1788685389.260622 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 35: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xb61b00be (stale) Mirror 19: 0xb61b00be (stale) Mirror 23: 0xb61b00be (stale) Mirror 24: 0xb61b00be (stale) Mirror 25: 0xb61b00be (stale) Mirror 26: 0xb61b00be (stale) Mirror 27: 0xb61b00be (stale) Mirror 28: 0xb61b00be (stale) Mirror 29: 0xb61b00be (stale) Mirror 31: 0xb61b00be (stale) Mirror 32: 0xb61b00be (stale) Mirror 33: 0xb61b00be (stale) Mirror 35: 0xb61b00be (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9638c453 Mirror 36: 0x9638c453 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000882 1788685391.218054 read 0x01ed70 thru 0x023df2 (0x5083 bytes) 000884 1788685391.258917 mapread 0x010901 thru 0x0203c5 (0xfac5 bytes) 000885 1788685391.359354 read 0x019662 thru 0x0289d6 (0xf375 bytes) 000886 1788685391.438178 write 0x01917c thru 0x01ed38 (0x5bbd bytes) 000887 1788685391.679345 mapwrite 0x039aab thru 0x03ffff (0x6555 bytes) 000889 1788685391.794925 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xb61b00be (stale) Mirror 19: 0xb61b00be (stale) Mirror 23: 0xb61b00be (stale) Mirror 24: 0xb61b00be (stale) Mirror 25: 0xb61b00be (stale) Mirror 26: 0xb61b00be (stale) Mirror 27: 0xb61b00be (stale) Mirror 28: 0xb61b00be (stale) Mirror 29: 0xb61b00be (stale) Mirror 31: 0xb61b00be (stale) Mirror 32: 0xb61b00be (stale) Mirror 33: 0xb61b00be (stale) Mirror 35: 0xb61b00be (stale) Mirror 36: 0x9638c453 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf516a0a2 Mirror 37: 0xf516a0a2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000891 1788685393.302538 write 0x015a9f thru 0x0226e8 (0xcc4a bytes) 000892 1788685393.453767 read 0x00d97e thru 0x018324 (0xa9a7 bytes) 000893 1788685393.467496 trunc from 0x040000 to 0x0045e8 (0x3ba17 bytes) 000898 1788685393.515071 trunc from 0x0045e9 to 0x01c9bd (0x183d5 bytes) 000899 1788685393.539530 mapread 0x00e962 thru 0x0149c4 (0x6063 bytes) 000900 1788685393.589334 punch from 0x006936 to 0x015435 (0xeb00 bytes) 000902 1788685393.629773 MIRROR_SPLIT mirror 7 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c9be): Mirror 14: 0x81b2d18a (stale) Mirror 19: 0x81b2d18a (stale) Mirror 23: 0x81b2d18a (stale) Mirror 24: 0x81b2d18a (stale) Mirror 25: 0x81b2d18a (stale) Mirror 26: 0x81b2d18a (stale) Mirror 28: 0x81b2d18a (stale) Mirror 29: 0x81b2d18a (stale) Mirror 31: 0x81b2d18a (stale) Mirror 32: 0x81b2d18a (stale) Mirror 33: 0x81b2d18a (stale) Mirror 35: 0x81b2d18a (stale) Mirror 36: 0xfb964a13 (stale) Mirror 37: 0xbeadee29 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c9be): Mirror 14: 0x81b2d18a (stale) Mirror 19: 0x81b2d18a (stale) Mirror 23: 0x81b2d18a (stale) Mirror 24: 0x81b2d18a (stale) Mirror 25: 0x81b2d18a (stale) Mirror 26: 0x81b2d18a (stale) Mirror 28: 0x81b2d18a (stale) Mirror 29: 0x81b2d18a (stale) Mirror 31: 0x81b2d18a (stale) Mirror 32: 0x81b2d18a (stale) Mirror 33: 0x81b2d18a (stale) Mirror 35: 0x81b2d18a (stale) Mirror 36: 0xfb964a13 (stale) Mirror 37: 0xbeadee29 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c9be: skipped 000903 1788685394.427326 mapread 0x009d89 thru 0x014893 (0xab0b bytes) 000904 1788685394.462101 punch from 0x005d8f to 0x010e29 (0xb09b bytes) 000905 1788685394.468811 trunc from 0x01c9be to 0x023106 (0x6749 bytes) 000906 1788685394.488929 mapread 0x004fa5 thru 0x00c0e8 (0x7144 bytes) 000907 1788685394.505467 MIRROR_RESYNC 15 mirrors 000908 1788685397.079073 write 0x037369 thru 0x03ffff (0x8c97 bytes) 000912 1788685397.147339 read 0x0331b3 thru 0x0381c4 (0x5012 bytes) 000913 1788685397.152949 punch from 0x011537 to 0x01c40c (0xaed6 bytes) 000914 1788685397.168118 mapread 0x00daa6 thru 0x00ed87 (0x12e2 bytes) 000915 1788685397.177886 write 0x02098e thru 0x02d54f (0xcbc2 bytes) 000916 1788685397.187877 write 0x0146a1 thru 0x018f88 (0x48e8 bytes) 000917 1788685397.197846 mapread 0x032c3f thru 0x036de5 (0x41a7 bytes) 000918 1788685397.206469 punch from 0x023234 to 0x02d985 (0xa752 bytes) 000919 1788685397.228270 write 0x034001 thru 0x03ffff (0xbfff bytes) 000920 1788685397.230297 read 0x020e45 thru 0x021625 (0x7e1 bytes) 000926 1788685397.233876 write 0x012d8a thru 0x020857 (0xdace bytes) 000927 1788685397.252095 read 0x0372f2 thru 0x03ffff (0x8d0e bytes) 000928 1788685397.252715 mapwrite 0x0249ec thru 0x031075 (0xc68a bytes) 000929 1788685397.300738 punch from 0x00655f to 0x00a799 (0x423b bytes) 000930 1788685397.308690 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) Mirror 37: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xe20eea22 (stale) Mirror 19: 0xe20eea22 (stale) Mirror 23: 0xe20eea22 (stale) Mirror 24: 0xe20eea22 (stale) Mirror 25: 0xe20eea22 (stale) Mirror 26: 0xe20eea22 (stale) Mirror 28: 0xe20eea22 (stale) Mirror 29: 0xe20eea22 (stale) Mirror 31: 0xe20eea22 (stale) Mirror 32: 0xe20eea22 (stale) Mirror 33: 0xe20eea22 (stale) Mirror 35: 0xe20eea22 (stale) Mirror 36: 0xe20eea22 (stale) Mirror 37: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfab68e6 Mirror 38: 0xfab68e6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000935 1788685398.465890 mapread 0x030bf0 thru 0x0313f7 (0x808 bytes) 000937 1788685398.501368 write 0x01cb22 thru 0x021e88 (0x5367 bytes) 000938 1788685398.649834 trunc from 0x040000 to 0x03d213 (0x2dec bytes) 000940 1788685398.692091 write 0x03c99b thru 0x03ffff (0x3665 bytes) 000941 1788685398.698220 mapwrite 0x00f58a thru 0x01cb48 (0xd5bf bytes) 000944 1788685398.773614 mapwrite 0x003f79 thru 0x007ad9 (0x3b61 bytes) 000945 1788685398.835172 trunc from 0x040000 to 0x01503e (0x2afc1 bytes) 000946 1788685398.856444 write 0x017705 thru 0x0239d4 (0xc2d0 bytes) 000947 1788685398.859209 trunc from 0x0239d5 to 0x0317e3 (0xde0f bytes) 000949 1788685398.882275 punch from 0x010882 to 0x01be9b (0xb61a bytes) 000950 1788685398.900380 mapread 0x0170c7 thru 0x018052 (0xf8c bytes) 000951 1788685398.908896 trunc from 0x0317e4 to 0x00c488 (0x2535b bytes) 000954 1788685398.926006 punch from 0x0093d7 to 0x00c488 (0x30b2 bytes) 000957 1788685398.933219 mapread 0x0066cd thru 0x008406 (0x1d3a bytes) 000958 1788685398.934819 write 0x010d51 thru 0x017986 (0x6c36 bytes) 000959 1788685398.937125 trunc from 0x017987 to 0x022fcd (0xb647 bytes) 000960 1788685398.954080 write 0x002c17 thru 0x00f547 (0xc931 bytes) 000963 1788685398.962023 read 0x00d102 thru 0x019763 (0xc662 bytes) 000964 1788685398.967863 mapwrite 0x039bfb thru 0x03ffff (0x6405 bytes) 000965 1788685399.039324 mapwrite 0x03bd37 thru 0x03ffff (0x42c9 bytes) 000966 1788685399.087912 mapwrite 0x010244 thru 0x01d33a (0xd0f7 bytes) 000967 1788685399.136458 trunc from 0x040000 to 0x03112a (0xeed5 bytes) 000968 1788685399.151837 mapread 0x01d022 thru 0x02c5c6 (0xf5a5 bytes) 000969 1788685399.172524 MIRROR_SPLIT mirror 4 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3112b): Mirror 14: 0xa51f863c (stale) Mirror 19: 0xa51f863c (stale) Mirror 23: 0xa51f863c (stale) Mirror 25: 0xa51f863c (stale) Mirror 26: 0xa51f863c (stale) Mirror 28: 0xa51f863c (stale) Mirror 29: 0xa51f863c (stale) Mirror 31: 0xa51f863c (stale) Mirror 32: 0xa51f863c (stale) Mirror 33: 0xa51f863c (stale) Mirror 35: 0xa51f863c (stale) Mirror 36: 0xa51f863c (stale) Mirror 37: 0xa51f863c (stale) Mirror 38: 0x228c96dd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3112b): Mirror 14: 0xa51f863c (stale) Mirror 19: 0xa51f863c (stale) Mirror 23: 0xa51f863c (stale) Mirror 25: 0xa51f863c (stale) Mirror 26: 0xa51f863c (stale) Mirror 28: 0xa51f863c (stale) Mirror 29: 0xa51f863c (stale) Mirror 31: 0xa51f863c (stale) Mirror 32: 0xa51f863c (stale) Mirror 33: 0xa51f863c (stale) Mirror 35: 0xa51f863c (stale) Mirror 36: 0xa51f863c (stale) Mirror 37: 0xa51f863c (stale) Mirror 38: 0x228c96dd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3112b: skipped 000970 1788685399.837249 write 0x0332aa thru 0x03b1d4 (0x7f2b bytes) 000971 1788685399.842473 read 0x029cfc thru 0x030951 (0x6c56 bytes) 000972 1788685399.856796 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x3b1d5): Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) Mirror 37: (stale - read error) Mirror 38: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b1d5): Mirror 14: 0x1fe0c0b3 (stale) Mirror 19: 0x1fe0c0b3 (stale) Mirror 23: 0x1fe0c0b3 (stale) Mirror 25: 0x1fe0c0b3 (stale) Mirror 26: 0x1fe0c0b3 (stale) Mirror 28: 0x1fe0c0b3 (stale) Mirror 29: 0x1fe0c0b3 (stale) Mirror 31: 0x1fe0c0b3 (stale) Mirror 32: 0x1fe0c0b3 (stale) Mirror 33: 0x1fe0c0b3 (stale) Mirror 35: 0x1fe0c0b3 (stale) Mirror 36: 0x1fe0c0b3 (stale) Mirror 37: 0x1fe0c0b3 (stale) Mirror 38: 0x5df0968a (stale) CRC-32 checksum value for chunk [0, 0x3b1d5): Mirror 1: 0xc06bce95 Mirror 39: 0xc06bce95 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b1d5: skipped 000973 1788685401.584223 read 0x016e41 thru 0x01e860 (0x7a20 bytes) 000974 1788685401.615228 trunc from 0x03b1d5 to 0x0162b3 (0x24f21 bytes) 000975 1788685401.786507 trunc from 0x0162b4 to 0x033492 (0x1d1df bytes) 000977 1788685401.806978 mapwrite 0x023a1d thru 0x025015 (0x15f9 bytes) 000981 1788685401.880868 read 0x01401d thru 0x016767 (0x274b bytes) 000982 1788685401.895295 mapwrite 0x01dfdd thru 0x02cf81 (0xefa5 bytes) 000984 1788685401.973954 punch from 0x01ccd7 to 0x029d1f (0xd049 bytes) 000985 1788685401.978905 punch from 0x012ef8 to 0x01f0c9 (0xc1d2 bytes) 000987 1788685402.008904 MIRROR_SPLIT mirror 13 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33493): Mirror 14: 0xb3e56d8f (stale) Mirror 19: 0xb3e56d8f (stale) Mirror 23: 0xb3e56d8f (stale) Mirror 25: 0xb3e56d8f (stale) Mirror 26: 0xb3e56d8f (stale) Mirror 28: 0xb3e56d8f (stale) Mirror 29: 0xb3e56d8f (stale) Mirror 31: 0xb3e56d8f (stale) Mirror 32: 0xb3e56d8f (stale) Mirror 33: 0xb3e56d8f (stale) Mirror 35: 0xb3e56d8f (stale) Mirror 36: 0xb3e56d8f (stale) Mirror 38: 0x6c1b913f (stale) Mirror 39: 0x7da6c70b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33493): Mirror 14: 0xb3e56d8f (stale) Mirror 19: 0xb3e56d8f (stale) Mirror 23: 0xb3e56d8f (stale) Mirror 25: 0xb3e56d8f (stale) Mirror 26: 0xb3e56d8f (stale) Mirror 28: 0xb3e56d8f (stale) Mirror 29: 0xb3e56d8f (stale) Mirror 31: 0xb3e56d8f (stale) Mirror 32: 0xb3e56d8f (stale) Mirror 33: 0xb3e56d8f (stale) Mirror 35: 0xb3e56d8f (stale) Mirror 36: 0xb3e56d8f (stale) Mirror 38: 0x6c1b913f (stale) Mirror 39: 0x7da6c70b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33493: skipped 000988 1788685402.789273 trunc from 0x033493 to 0x034475 (0xfe3 bytes) 000989 1788685402.845825 mapread 0x00aec4 thru 0x012db2 (0x7eef bytes) 000992 1788685402.895171 mapwrite 0x016f81 thru 0x023e5d (0xcedd bytes) 000994 1788685403.032951 punch from 0x00f2e5 to 0x017b22 (0x883e bytes) 000997 1788685403.045048 read 0x009e05 thru 0x012d5d (0x8f59 bytes) 000998 1788685403.056170 punch from 0x01fb44 to 0x024531 (0x49ee bytes) 000999 1788685403.063013 mapwrite 0x00b104 thru 0x00df28 (0x2e25 bytes) 001000 1788685403.127275 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x34476): Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) Mirror 38: (stale - read error) Mirror 39: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34476): Mirror 14: 0x1514697a (stale) Mirror 19: 0x1514697a (stale) Mirror 23: 0x1514697a (stale) Mirror 25: 0x1514697a (stale) Mirror 26: 0x1514697a (stale) Mirror 28: 0x1514697a (stale) Mirror 29: 0x1514697a (stale) Mirror 31: 0x1514697a (stale) Mirror 32: 0x1514697a (stale) Mirror 33: 0x1514697a (stale) Mirror 35: 0x1514697a (stale) Mirror 36: 0x1514697a (stale) Mirror 38: 0xc1ba90a5 (stale) Mirror 39: 0x9b02e666 (stale) CRC-32 checksum value for chunk [0, 0x34476): Mirror 1: 0x869aa7c9 Mirror 40: 0x869aa7c9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34476: skipped All operations completed A-OK! PASS 70a (150s) == sanity-flr test 71: check mirror extend parent fid ==== 05:03:30 (1788685410) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 05:03:33 (1788685413) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.289373 s, 7.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119064 s, 8.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f3:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 05:03:42 (1788685422) Starting client: oleg253-client.virtnet: -o user_xattr,flock 192.168.202.153@tcp:/lustre /mnt/lustre2 Starting client: oleg253-client.virtnet: -o user_xattr,flock 192.168.202.153@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7573248 .. Extending file size to 8420544 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8279104 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8571328 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8688096 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8895232 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8846784 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 57526 57527 57529 57530 57533 192.168.202.153@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg253-client.virtnet /mnt/lustre2 (opts:) 192.168.202.153@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg253-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 05:05:05 (1788685505) Starting client: oleg253-client.virtnet: -o user_xattr,flock 192.168.202.153@tcp:/lustre /mnt/lustre2 Starting client: oleg253-client.virtnet: -o user_xattr,flock 192.168.202.153@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7250496 .. Extending file size to 4839904 .. Extending file size to 6998752 .. Extending file size to 7767776 .. Extending file size to 7794112 .. Extending file size to 8199168 .. Extending file size to 8282944 .. Extending file size to 9189120 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8626944 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8918144 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 58994 58996 58997 58998 59001 192.168.202.153@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg253-client.virtnet /mnt/lustre2 (opts:) 192.168.202.153@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg253-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (85s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 05:06:31 (1788685591) Starting client: oleg253-client.virtnet: -o user_xattr,flock 192.168.202.153@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.107795 s, 19.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.308364 s, 6.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x569ccd85 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.153@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg253-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 05:06:53 (1788685613) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.178811 s, 11.7 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 05:07:02 (1788685622) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x211:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x305:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 05:07:09 (1788685629) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.22632 s, 9.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x306:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x309:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 05:07:17 (1788685637) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.26638 s, 7.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x307:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x308:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x307:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x308:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30c:0x0] } PASS 204b (11s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 05:07:28 (1788685648) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.521197 s, 4.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30e:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 05:08:04 (1788685684) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg253-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841392 s, 12.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x310:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x310:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } PASS 204d (58s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 05:09:02 (1788685742) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg253-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.146881 s, 14.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x311:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x311:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (54s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 05:09:57 (1788685797) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg253-server mds-ost sync done. Creating new pool test_204f oleg253-server: Pool lustre.test_204f created Adding targets to pool oleg253-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg253-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.209406 s, 10.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x312:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x312:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg253-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg253-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg253-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 05:11:17 (1788685877) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 05:11:26 (1788685886) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 05:11:35 (1788685895) oleg253-server: Pool lustre.test_206 created Adding targets to pool oleg253-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg253-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg253-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg253-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg253-server: Pool lustre.test_206 destroyed PASS 206 (23s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 05:11:58 (1788685918) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812358 s, 12.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358594 s, 29.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x96bc55a Mirror 3: 0x96bc55a PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 05:12:09 (1788685929) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 05:12:13 (1788685933) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 05:12:17 (1788685937) Starting client: oleg253-client.virtnet: -o user_xattr,flock 192.168.202.153@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.074538 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x13a0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xcd0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xcd0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xcd0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xf70000 Read this (4 bytes): BBBB 192.168.202.153@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg253-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 05:12:54 (1788685974) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0335108 s, 244 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 05:13:01 (1788685981) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059948 s, 17.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x236 lmm_fid: [0x200000404:0x236:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x236 lmm_fid: [0x200000404:0x236:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x31d:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834486 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 05:13:03.000000000 -0400 Modify: 2026-09-06 05:13:03.000000000 -0400 Change: 2026-09-06 05:13:04.000000000 -0400 Birth: 2026-09-06 05:13:03.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 05:13:10 (1788685990) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 05:13:14 (1788685994) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.61651 s, 66.4 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0972127 s, 42.1 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 6 05:13 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x238:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 6 05:13 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 05:13:24 (1788686004) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 05:13:30 (1788686010) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (13s) == sanity-flr test complete, duration 1791 sec =========== 05:13:43 (1788686023) === sanity-flr: start cleanup 05:13:46 (1788686026) === === sanity-flr: finish cleanup 05:13:53 (1788686033) ===