-----============= acceptance-small: sanity-flr ============----- Wed Jul 15 09:59:17 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg207-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 09:59:23 (1784123963) === oleg207-client.virtnet: executing check_config_client /mnt/lustre oleg207-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg207-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8bafa08b1000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8bafa08b1000.idle_timeout=debug disable quota as required oleg207-server: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 09:59:36 (1784123976) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7588] == sanity-flr test 0a: lfs mirror create with -N option == 09:59:37 (1784123977) PASS 0a (7s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:59:44 (1784123984) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:59:45 (1784123985) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:59:46 (1784123986) PASS 0d (9s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:59:55 (1784123995) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:59:56 (1784123996) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 09:59:57 (1784123997) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8bafa08b1000.stats=clear osc.lustre-OST0001-osc-ffff8bafa08b1000.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:00:00 (1784124000) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:00:05 (1784124005) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:00:07 (1784124007) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 10:00:09 (1784124009) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 10:00:11 (1784124011) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 10:00:15 (1784124015) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:00:18 (1784124018) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:00:20 (1784124020) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:00:23 (1784124023) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 10:00:27 (1784124027) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:00:30 (1784124030) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.34362 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414339 s, 25.3 MB/s PASS 21 (4s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:00:34 (1784124034) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 0.136924 s, 107 MB/s PASS 22 (4s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 10:00:38 (1784124038) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:00:42 (1784124042) Stopping /mnt/lustre-ost1 (opts:) on oleg207-server oleg207-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8bafa08b1000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8bafa08b1000.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg207-server: oleg207-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg207-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg207-server oleg207-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8bafa08b1000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8bafa08b1000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg207-server: oleg207-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg207-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (23s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:01:05 (1784124065) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.170875 s, 55.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg207-server oleg207-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8bafa08b1000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8bafa08b1000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg207-server: oleg207-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg207-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (15s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:01:20 (1784124080) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg207-server oleg207-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8bafa08b1000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8bafa08b1000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg207-server: oleg207-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg207-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg207-server oleg207-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8bafa08b1000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8bafa08b1000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg207-server: oleg207-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg207-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (26s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:01:46 (1784124106) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.21439 s, 43.2 MB/s -rw-r--r-- 1 root root 52428800 Jul 15 10:01 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.261s user 0m0.000s sys 0m0.147s Stopping /mnt/lustre-ost1 (opts:) on oleg207-server oleg207-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8bafa08b1000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8bafa08b1000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.359s user 0m0.000s sys 0m0.156s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg207-server: oleg207-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg207-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg207-server oleg207-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8bafa08b1000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8bafa08b1000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.277s user 0m0.001s sys 0m0.145s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg207-server: oleg207-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg207-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg207-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8bafa08b1000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8bafa08b1000.ost_server_uuid in FULL state after 0 sec PASS 33b (33s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:02:19 (1784124139) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:02:20 (1784124140) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:02:21 (1784124141) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 10:02:22 (1784124142) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 10:02:25 (1784124145) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.179604 s, 70.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:02:29 (1784124149) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125552 s, 83.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012436 s, 84.3 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:02:32 (1784124152) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:02:35 (1784124155) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000643425 s, 1.6 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00070639 s, 1.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000642908 s, 1.6 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00069342 s, 1.5 MB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 10:02:39 (1784124159) create test files with size 15736055 .. a1e1cc5bb27c336d2fc63e6d5891c2df - 8c337256c703daee212fc070ef864493 - e3a430661c77f394c59ed149bdbc822c - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg207-client.virtnet /mnt/lustre (opts:) Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre PASS 37 (7s) == sanity-flr test 38: resync ============================ 10:02:46 (1784124166) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (5s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:02:51 (1784124171) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:02:53 (1784124173) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374322 s, 28.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xba:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd4:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144882 s, 72.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd5:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 10:02:56 (1784124176) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0329373 s, 63.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.104683 s, 40.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0497354 s, 42.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (5s) == sanity-flr test 42: lfs mirror verify ================= 10:03:01 (1784124181) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 10:03:02 (1784124182) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:03:03 (1784124183) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123851 s, 84.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 10:03:06 (1784124186) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:03:07 (1784124187) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdd:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc3:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc3:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:03:10 (1784124190) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0864659 s, 121 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg207-client.virtnet /mnt/lustre (opts:) Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (20s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:03:30 (1784124210) llite.lustre-ffff8baf90e58800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0913039 s, 115 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 Stopping client oleg207-client.virtnet /mnt/lustre (opts:) Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 llite.lustre-ffff8baf82cfb800.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:03:34 (1784124214) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (13s) == sanity-flr test 46: Verify setstripe --copy option ==== 10:03:47 (1784124227) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (12s) == sanity-flr test 47: Verify mirror obj alloc =========== 10:03:59 (1784124239) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 10:04:01 (1784124241) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0884065 s, 35.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.066832 s, 47.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xb04ca25f (nosync,stale) Mirror 2: 0xb04ca25f (nosync,stale) ** verify mirror 2 doesn't change original checksum: da6173b6a0d40b9274c2c95486ea0c98 - mirror 1 checksum: cfe66a21775c1edd93e9eb99094ab953 - mirror 2 checksum: da6173b6a0d40b9274c2c95486ea0c98 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-15T10:04:02-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-15T10:04:02-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: da6173b6a0d40b9274c2c95486ea0c98 - mirror 1 checksum: cfe66a21775c1edd93e9eb99094ab953 - mirror 2 checksum: cfe66a21775c1edd93e9eb99094ab953 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-15T10:04:02-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-15T10:04:02-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } PASS 48 (3s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 10:04:05 (1784124245) Failing mds1 on oleg207-server Stopping /mnt/lustre-mds1 (opts:) on oleg207-server 10:04:07 (1784124247) shut down facet: mds1 facet_host: oleg207-server facet_failover_host: oleg207-server Failover mds1 to oleg207-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg207-server: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-MDT0000 10:04:22 (1784124262) targets are mounted 10:04:22 (1784124262) facet_failover done oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156926 s, 66.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } PASS 50A (23s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 10:04:28 (1784124268) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 10:04:30 (1784124270) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 10:04:31 (1784124271) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00343109 s, 4.8 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 10:04:34 (1784124274) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 10:04:35 (1784124275) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 10:04:36 (1784124276) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (6s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 10:04:42 (1784124282) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 10:05:02 (1784124302) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 10:05:20 (1784124320) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 10:05:38 (1784124338) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 10:05:40 (1784124340) mirror create pid 53853 mirror split pid 53855 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 53853 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 53855 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 10:06:43 (1784124403) Seed set to 4403 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000003 1784124403.840719 mapwrite 0x03b992 thru 0x03f329 (0x3998 bytes) 000004 1784124403.875471 mapread 0x026123 thru 0x02b8ff (0x57dd bytes) 000006 1784124403.880223 mapwrite 0x029c48 thru 0x02b647 (0x1a00 bytes) truncating to largest ever: 0xc025 000007 1784124403.893410 trunc from 0x03f32a to 0x00c024 (0x33305 bytes) 000008 1784124403.900206 trunc from 0x00c025 to 0x005526 (0x6afe bytes) 000011 1784124403.906674 mapwrite 0x02cf6c thru 0x02d7b7 (0x84c bytes) truncating to largest ever: 0x32410 000014 1784124403.930189 trunc from 0x02d7b8 to 0x03240f (0x4c58 bytes) 000015 1784124403.936384 read 0x01b149 thru 0x02a573 (0xf42b bytes) truncating to largest ever: 0x366c5 000017 1784124403.941373 trunc from 0x032410 to 0x0366c4 (0x42b5 bytes) 000018 1784124403.950528 write 0x03c34e thru 0x03ffff (0x3cb2 bytes) 000019 1784124403.951520 read 0x02240e thru 0x02fabf (0xd6b2 bytes) 000020 1784124403.956276 mapwrite 0x0143b2 thru 0x01b711 (0x7360 bytes) 000021 1784124403.977899 mapwrite 0x03bf07 thru 0x03ffff (0x40f9 bytes) 000023 1784124403.993394 read 0x00906f thru 0x01182c (0x87be bytes) 000026 1784124403.995347 read 0x03e31c thru 0x03ffff (0x1ce4 bytes) 000029 1784124403.998586 MIRROR_EXTEND to 2 mirrors 000030 1784124404.188517 mapwrite 0x02ac77 thru 0x03723c (0xc5c6 bytes) 000031 1784124404.249159 read 0x004cc2 thru 0x00e13c (0x947b bytes) 000032 1784124404.253371 mapwrite 0x00218a thru 0x0071d5 (0x504c bytes) 000034 1784124404.276768 mapread 0x01e4eb thru 0x01e8dd (0x3f3 bytes) 000035 1784124404.280780 trunc from 0x040000 to 0x015586 (0x2aa79 bytes) 000036 1784124404.287647 mapwrite 0x00c961 thru 0x0171c6 (0xa866 bytes) 000037 1784124404.312355 read 0x017121 thru 0x0171c6 (0xa6 bytes) 000038 1784124404.312849 trunc from 0x0171c7 to 0x0115ba (0x5c0c bytes) 000039 1784124404.317362 mapread 0x002a10 thru 0x0115ba (0xebab bytes) 000042 1784124404.319364 mapread 0x00fc45 thru 0x010674 (0xa30 bytes) 000043 1784124404.320056 trunc from 0x0115bb to 0x0168d5 (0x531b bytes) 000044 1784124404.325276 trunc from 0x0168d6 to 0x019afd (0x3228 bytes) 000045 1784124404.330231 read 0x008eb9 thru 0x010f54 (0x809c bytes) truncating to largest ever: 0x39de7 000046 1784124404.330779 trunc from 0x019afe to 0x039de6 (0x202e9 bytes) 000048 1784124404.335220 read 0x0054fe thru 0x007c99 (0x279c bytes) 000051 1784124404.336472 trunc from 0x039de7 to 0x0106f6 (0x296f0 bytes) 000052 1784124404.340754 write 0x016bfb thru 0x01bb84 (0x4f8a bytes) 000056 1784124404.342961 mapwrite 0x00d43d thru 0x0189f8 (0xb5bc bytes) 000057 1784124404.360770 MIRROR_RESYNC 2 mirrors 000058 1784124404.536139 mapread 0x00c3d9 thru 0x0160be (0x9ce6 bytes) 000059 1784124404.543204 MIRROR_RESYNC 2 mirrors 000060 1784124404.657991 mapwrite 0x028281 thru 0x02d554 (0x52d4 bytes) 000064 1784124404.702410 write 0x0084d4 thru 0x011364 (0x8e91 bytes) 000066 1784124404.707765 mapread 0x0235b5 thru 0x0268ec (0x3338 bytes) 000067 1784124404.710658 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2d555): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d555): Mirror 2: 0x97f39c50 (stale) CRC-32 checksum value for chunk [0, 0x2d555): Mirror 1: 0x2e3d3b9e Mirror 3: 0x2e3d3b9e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d555: skipped 000069 1784124404.981261 read 0x003aee thru 0x0121bc (0xe6cf bytes) 000070 1784124404.990845 read 0x005614 thru 0x0100e8 (0xaad5 bytes) 000071 1784124404.991266 write 0x0033db thru 0x00effc (0xbc22 bytes) 000072 1784124405.019728 read 0x022a2f thru 0x02401a (0x15ec bytes) 000073 1784124405.020188 mapwrite 0x02c325 thru 0x038aa7 (0xc783 bytes) 000075 1784124405.057758 trunc from 0x038aa8 to 0x02a577 (0xe530 bytes) 000078 1784124405.063840 mapread 0x006e32 thru 0x00757b (0x74a bytes) 000080 1784124405.067058 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2a578): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a578): Mirror 2: 0xe1ae3ced (stale) Mirror 3: 0x8e693ad3 (stale) CRC-32 checksum value for chunk [0, 0x2a578): Mirror 1: 0xc98461be Mirror 4: 0xc98461be lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a578: skipped 000086 1784124405.305621 read 0x00ef20 thru 0x016d5e (0x7e3f bytes) 000087 1784124405.309956 mapread 0x00192d thru 0x011143 (0xf817 bytes) 000088 1784124405.318693 write 0x02e028 thru 0x038390 (0xa369 bytes) 000089 1784124405.342708 read 0x02ca70 thru 0x02f470 (0x2a01 bytes) 000092 1784124405.350252 mapwrite 0x0341f3 thru 0x039db9 (0x5bc7 bytes) 000093 1784124405.386444 trunc from 0x039dba to 0x0353f7 (0x49c2 bytes) 000094 1784124405.391756 mapwrite 0x03ad59 thru 0x03ed7e (0x4026 bytes) 000095 1784124405.413173 read 0x005a1d thru 0x007de1 (0x23c5 bytes) 000097 1784124405.416409 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x591:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ed7f): Mirror 2: 0x45db9c6f (stale) Mirror 3: 0x89dc4931 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed7f): Mirror 2: 0x45db9c6f (stale) Mirror 3: 0x89dc4931 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed7f: skipped 000099 1784124405.522962 write 0x034e4b thru 0x03e313 (0x94c9 bytes) 000100 1784124405.526852 mapread 0x0080c1 thru 0x0088f3 (0x833 bytes) 000101 1784124405.528676 write 0x01839a thru 0x023f70 (0xbbd7 bytes) 000102 1784124405.532312 mapread 0x01707b thru 0x01a3b8 (0x333e bytes) 000105 1784124405.537399 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3ed7f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed7f): Mirror 2: 0x45db9c6f (stale) Mirror 3: 0x89dc4931 (stale) CRC-32 checksum value for chunk [0, 0x3ed7f): Mirror 1: 0xba988a45 Mirror 4: 0xba988a45 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed7f: skipped 000107 1784124405.790555 mapwrite 0x03b283 thru 0x03ffff (0x4d7d bytes) 000108 1784124405.834355 trunc from 0x040000 to 0x00c8d9 (0x33726 bytes) 000111 1784124405.841664 read 0x004243 thru 0x00c8d9 (0x8697 bytes) 000113 1784124405.845839 trunc from 0x00c8da to 0x00f77e (0x2ea5 bytes) 000114 1784124405.851381 mapwrite 0x030e9a thru 0x036833 (0x599a bytes) 000116 1784124405.883465 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x36834): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36834): Mirror 2: 0x217e65b5 (stale) Mirror 3: 0x853aa508 (stale) Mirror 4: 0xd79100bd (stale) CRC-32 checksum value for chunk [0, 0x36834): Mirror 1: 0xe88326d7 Mirror 5: 0xe88326d7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36834: skipped 000119 1784124406.132609 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x36834): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36834): Mirror 2: 0x217e65b5 (stale) Mirror 3: 0x853aa508 (stale) Mirror 4: 0xd79100bd (stale) CRC-32 checksum value for chunk [0, 0x36834): Mirror 1: 0xe88326d7 Mirror 5: 0xe88326d7 Mirror 6: 0xe88326d7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36834: skipped 000120 1784124406.391459 mapread 0x032bab thru 0x033eb5 (0x130b bytes) 000121 1784124406.395033 read 0x01c692 thru 0x02b6fc (0xf06b bytes) 000122 1784124406.398168 mapwrite 0x006a0b thru 0x013c64 (0xd25a bytes) 000125 1784124406.469203 write 0x01822b thru 0x0254ac (0xd282 bytes) 000129 1784124406.473139 write 0x014484 thru 0x019336 (0x4eb3 bytes) 000130 1784124406.473997 mapwrite 0x0143b0 thru 0x021340 (0xcf91 bytes) 000132 1784124406.491107 mapwrite 0x022f3e thru 0x028137 (0x51fa bytes) 000136 1784124406.511140 trunc from 0x036834 to 0x0302ba (0x6579 bytes) 000137 1784124406.522333 write 0x01772b thru 0x01aca9 (0x357f bytes) 000139 1784124406.523980 mapread 0x00260a thru 0x002d70 (0x767 bytes) 000140 1784124406.524882 write 0x005961 thru 0x007701 (0x1da1 bytes) 000147 1784124406.531052 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x302bb): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x302bb): Mirror 2: 0xd0c8d531 (stale) Mirror 3: 0x7292f0bf (stale) Mirror 4: 0x49281292 (stale) Mirror 5: 0x31784ad1 (stale) Mirror 6: 0x31784ad1 (stale) CRC-32 checksum value for chunk [0, 0x302bb): Mirror 1: 0x6611bde9 Mirror 7: 0x6611bde9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x302bb: skipped 000149 1784124406.791168 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x591:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x302bb): Mirror 2: 0xd0c8d531 (stale) Mirror 4: 0x49281292 (stale) Mirror 5: 0x31784ad1 (stale) Mirror 6: 0x31784ad1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x302bb): Mirror 2: 0xd0c8d531 (stale) Mirror 4: 0x49281292 (stale) Mirror 5: 0x31784ad1 (stale) Mirror 6: 0x31784ad1 (stale) CRC-32 checksum value for chunk [0, 0x302bb): Mirror 1: 0x6611bde9 Mirror 7: 0x6611bde9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x302bb: skipped 000150 1784124407.014142 write 0x008be1 thru 0x017ce5 (0xf105 bytes) 000152 1784124407.044719 write 0x032a87 thru 0x0366e1 (0x3c5b bytes) 000154 1784124407.045853 write 0x0010b0 thru 0x00ff91 (0xeee2 bytes) 000155 1784124407.048131 trunc from 0x0366e2 to 0x0130bd (0x23624 bytes) 000156 1784124407.060482 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x591:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x130be): Mirror 4: 0xa20a32dd (stale) Mirror 5: 0x497c3bd4 (stale) Mirror 6: 0x497c3bd4 (stale) Mirror 7: 0xc4399acd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x130be): Mirror 4: 0xa20a32dd (stale) Mirror 5: 0x497c3bd4 (stale) Mirror 6: 0x497c3bd4 (stale) Mirror 7: 0xc4399acd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x130be: skipped 000157 1784124407.172172 mapread 0x00666a thru 0x00b2f4 (0x4c8b bytes) 000161 1784124407.180816 write 0x01cd39 thru 0x0262eb (0x95b3 bytes) 000162 1784124407.182064 trunc from 0x0262ec to 0x00b602 (0x1ace9 bytes) 000164 1784124407.189447 write 0x0385f1 thru 0x03ffff (0x7a0f bytes) 000165 1784124407.193104 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x591:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xc984d59c (stale) Mirror 5: 0xb0def3ad (stale) Mirror 6: 0xb0def3ad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xc984d59c (stale) Mirror 5: 0xb0def3ad (stale) Mirror 6: 0xb0def3ad (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000166 1784124407.314631 read 0x02d74d thru 0x032d4f (0x5603 bytes) 000171 1784124407.320169 trunc from 0x040000 to 0x0154d9 (0x2ab26 bytes) 000172 1784124407.327137 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x591:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x154da): Mirror 5: 0xb89203f3 (stale) Mirror 6: 0xb89203f3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x154da): Mirror 5: 0xb89203f3 (stale) Mirror 6: 0xb89203f3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x154da: skipped 000174 1784124407.421418 mapwrite 0x020670 thru 0x02a991 (0xa322 bytes) 000176 1784124407.456034 write 0x024485 thru 0x02fdef (0xb96b bytes) 000177 1784124407.457410 trunc from 0x02fdf0 to 0x024d55 (0xb09a bytes) 000179 1784124407.463831 trunc from 0x024d56 to 0x002ac1 (0x22294 bytes) 000180 1784124407.469306 mapread 0x001d2e thru 0x002ac1 (0xd94 bytes) 000183 1784124407.475989 write 0x00ce64 thru 0x016300 (0x949d bytes) 000187 1784124407.478507 read 0x003088 thru 0x005003 (0x1f7c bytes) 000188 1784124407.481162 trunc from 0x016301 to 0x000d74 (0x1558c bytes) 000190 1784124407.487109 read 0x0005f9 thru 0x000d74 (0x77c bytes) 000192 1784124407.489778 mapread 0x000802 thru 0x000d74 (0x573 bytes) 000193 1784124407.490570 write 0x00fb1f thru 0x01ce6c (0xd34e bytes) 000199 1784124407.494189 mapread 0x00c3ef thru 0x014748 (0x835a bytes) 000200 1784124407.498107 mapread 0x01bb63 thru 0x01ce6c (0x130a bytes) 000202 1784124407.499566 write 0x009877 thru 0x00f99d (0x6127 bytes) 000203 1784124407.500644 write 0x0175bc thru 0x026655 (0xf09a bytes) 000204 1784124407.502079 mapwrite 0x002b21 thru 0x00957b (0x6a5b bytes) 000207 1784124407.518765 mapread 0x014afd thru 0x018402 (0x3906 bytes) 000208 1784124407.519842 mapwrite 0x00f4e0 thru 0x01008d (0xbae bytes) 000209 1784124407.535931 read 0x0135f7 thru 0x022f60 (0xf96a bytes) 000210 1784124407.538694 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x591:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26656): Mirror 6: 0x6777e8ab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26656): Mirror 6: 0x6777e8ab (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26656: skipped 000212 1784124407.626743 mapwrite 0x010aeb thru 0x014e66 (0x437c bytes) 000213 1784124407.646790 mapwrite 0x03cc9e thru 0x03ffff (0x3362 bytes) 000214 1784124407.668842 read 0x0052b3 thru 0x00a248 (0x4f96 bytes) 000217 1784124407.674531 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x591:0x0]: Invalid argument (22) 000218 1784124407.718657 read 0x02817a thru 0x02c082 (0x3f09 bytes) 000219 1784124407.722037 write 0x031394 thru 0x03f105 (0xdd72 bytes) truncating to largest ever: 0x3eed8 000220 1784124407.726261 trunc from 0x040000 to 0x03eed7 (0x1128 bytes) 000222 1784124407.734092 MIRROR_EXTEND to 2 mirrors 000225 1784124407.898044 mapwrite 0x031efe thru 0x03db6d (0xbc70 bytes) 000226 1784124407.964625 read 0x01f114 thru 0x022564 (0x3451 bytes) 000228 1784124407.967450 mapread 0x020f97 thru 0x02efee (0xe058 bytes) 000232 1784124407.972164 write 0x0139fe thru 0x01c575 (0x8b78 bytes) 000233 1784124407.974850 read 0x02518c thru 0x027c35 (0x2aaa bytes) 000235 1784124407.975918 write 0x008f02 thru 0x009a07 (0xb06 bytes) 000236 1784124407.979141 trunc from 0x03eed8 to 0x011518 (0x2d9bf bytes) 000237 1784124407.991405 write 0x01a98c thru 0x01c119 (0x178e bytes) 000238 1784124407.992394 read 0x015982 thru 0x01b9f2 (0x6071 bytes) 000240 1784124407.995472 read 0x000512 thru 0x009770 (0x925f bytes) 000241 1784124407.996058 write 0x01fd55 thru 0x023960 (0x3c0c bytes) 000242 1784124407.996834 read 0x00efef thru 0x013dfd (0x4e0f bytes) 000244 1784124407.997778 read 0x016752 thru 0x023960 (0xd20f bytes) 000247 1784124408.001058 write 0x02a9e8 thru 0x02d082 (0x269b bytes) 000249 1784124408.002651 write 0x0138b5 thru 0x0217fd (0xdf49 bytes) 000250 1784124408.003828 read 0x022cb1 thru 0x02d082 (0xa3d2 bytes) 000252 1784124408.007153 mapread 0x0086fe thru 0x00fea3 (0x77a6 bytes) 000253 1784124408.009937 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x591:0x0]: Invalid argument (22) 000254 1784124408.054724 mapwrite 0x001d5d thru 0x002922 (0xbc6 bytes) 000255 1784124408.073005 MIRROR_EXTEND to 2 mirrors 000256 1784124408.242037 MIRROR_RESYNC 2 mirrors 000257 1784124408.368594 MIRROR_EXTEND to 3 mirrors 000259 1784124408.534895 read 0x023d50 thru 0x02d082 (0x9333 bytes) 000262 1784124408.538785 mapread 0x008c31 thru 0x014057 (0xb427 bytes) 000263 1784124408.547779 mapwrite 0x02c4dd thru 0x030706 (0x422a bytes) 000264 1784124408.601531 mapwrite 0x0182d1 thru 0x022d33 (0xaa63 bytes) 000265 1784124408.626933 trunc from 0x030707 to 0x00cb0f (0x23bf7 bytes) 000266 1784124408.635577 mapwrite 0x03e7ed thru 0x03ffff (0x1813 bytes) 000268 1784124408.663344 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x591:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x949325c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x949325c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000269 1784124408.766087 write 0x021ad7 thru 0x022d03 (0x122d bytes) 000272 1784124408.771842 read 0x02e0e7 thru 0x03cdad (0xecc7 bytes) 000273 1784124408.778529 MIRROR_RESYNC 2 mirrors 000274 1784124408.991694 mapread 0x03ab0b thru 0x03ffff (0x54f5 bytes) 000276 1784124408.999387 trunc from 0x040000 to 0x0194d3 (0x26b2c bytes) 000278 1784124409.031475 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x591:0x0]: Invalid argument (22) 000281 1784124409.088736 mapwrite 0x03b955 thru 0x03d353 (0x19ff bytes) 000285 1784124409.128120 mapread 0x005052 thru 0x01069d (0xb64c bytes) 000287 1784124409.133846 mapwrite 0x000a65 thru 0x00e053 (0xd5ef bytes) 000289 1784124409.165295 mapwrite 0x01eb6d thru 0x024239 (0x56cd bytes) 000291 1784124409.187945 trunc from 0x03d354 to 0x00c8ef (0x30a64 bytes) 000294 1784124409.195701 trunc from 0x00c8f0 to 0x009670 (0x327f bytes) 000296 1784124409.201853 write 0x000cf4 thru 0x005ec1 (0x51ce bytes) 000297 1784124409.202859 write 0x00bc6d thru 0x018611 (0xc9a5 bytes) 000298 1784124409.204305 write 0x005b61 thru 0x014154 (0xe5f4 bytes) 000303 1784124409.208024 mapwrite 0x00b857 thru 0x00c111 (0x8bb bytes) 000305 1784124409.230980 read 0x01298d thru 0x0147ba (0x1e2e bytes) 000306 1784124409.231602 read 0x0034f7 thru 0x00d8d4 (0xa3de bytes) 000308 1784124409.234734 MIRROR_EXTEND to 2 mirrors 000311 1784124409.424611 mapwrite 0x00ee71 thru 0x00ef3b (0xcb bytes) 000312 1784124409.488846 mapread 0x00f108 thru 0x012ebd (0x3db6 bytes) 000313 1784124409.493469 mapread 0x0150ed thru 0x018611 (0x3525 bytes) 000314 1784124409.499127 write 0x033095 thru 0x039e4b (0x6db7 bytes) 000317 1784124409.501252 read 0x031175 thru 0x036822 (0x56ae bytes) 000318 1784124409.504439 mapwrite 0x01b092 thru 0x02510e (0xa07d bytes) 000319 1784124409.531893 read 0x00c525 thru 0x01c2d3 (0xfdaf bytes) 000327 1784124409.535385 write 0x02b5b2 thru 0x03857c (0xcfcb bytes) 000329 1784124409.536789 read 0x003057 thru 0x008614 (0x55be bytes) 000330 1784124409.541044 mapwrite 0x010dbf thru 0x017a00 (0x6c42 bytes) 000331 1784124409.557651 write 0x01be07 thru 0x01dfab (0x21a5 bytes) 000332 1784124409.558575 read 0x01c257 thru 0x024369 (0x8113 bytes) 000333 1784124409.561527 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x591:0x0]: Invalid argument (22) 000336 1784124409.648579 read 0x026b2d thru 0x034e3d (0xe311 bytes) 000338 1784124409.656160 trunc from 0x039e4c to 0x02e0ae (0xbd9d bytes) 000339 1784124409.671446 mapread 0x02dd15 thru 0x02e0ae (0x39a bytes) 000340 1784124409.672404 trunc from 0x02e0af to 0x01979c (0x14912 bytes) 000344 1784124409.680775 read 0x00ac00 thru 0x01979c (0xeb9d bytes) 000345 1784124409.685715 write 0x039b0e thru 0x039be9 (0xdc bytes) 000348 1784124409.687580 trunc from 0x039bea to 0x019546 (0x206a3 bytes) 000350 1784124409.694109 trunc from 0x019547 to 0x009468 (0x100de bytes) 000351 1784124409.700257 trunc from 0x009469 to 0x02f61e (0x261b6 bytes) 000353 1784124409.707209 read 0x00ffb5 thru 0x01b8ad (0xb8f9 bytes) 000354 1784124409.711625 read 0x003ecf thru 0x0050ae (0x11e0 bytes) 000355 1784124409.712213 mapread 0x02750c thru 0x02f52e (0x8023 bytes) 000358 1784124409.714181 mapread 0x0273cd thru 0x02f61e (0x8252 bytes) 000360 1784124409.715703 trunc from 0x02f61f to 0x02bcba (0x3964 bytes) 000361 1784124409.721751 trunc from 0x02bcbb to 0x038d58 (0xd09e bytes) 000363 1784124409.727514 mapread 0x01b98c thru 0x02338b (0x7a00 bytes) 000364 1784124409.729945 MIRROR_EXTEND to 2 mirrors 000365 1784124409.918198 trunc from 0x038d59 to 0x022a99 (0x162bf bytes) 000366 1784124409.947561 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x591:0x0]: Invalid argument (22) 000368 1784124410.004535 mapwrite 0x03e56c thru 0x03ffff (0x1a94 bytes) 000369 1784124410.033607 mapread 0x00cfba thru 0x0144ab (0x74f2 bytes) 000370 1784124410.038945 read 0x00a504 thru 0x0153f0 (0xaeed bytes) 000372 1784124410.041621 trunc from 0x040000 to 0x01d0d0 (0x22f2f bytes) 000374 1784124410.047749 mapwrite 0x0266df thru 0x034592 (0xdeb4 bytes) 000375 1784124410.076839 mapwrite 0x026149 thru 0x02e6ef (0x85a7 bytes) 000377 1784124410.094587 mapread 0x02373b thru 0x025b00 (0x23c6 bytes) 000378 1784124410.095289 read 0x02f7f6 thru 0x034592 (0x4d9d bytes) 000379 1784124410.096112 mapwrite 0x003dc3 thru 0x00faca (0xbd08 bytes) 000382 1784124410.122002 write 0x0160f9 thru 0x016a59 (0x961 bytes) 000383 1784124410.122754 write 0x00af42 thru 0x0157e8 (0xa8a7 bytes) 000389 1784124410.126457 mapread 0x033b3c thru 0x034592 (0xa57 bytes) 000390 1784124410.127396 mapread 0x00aac1 thru 0x018e88 (0xe3c8 bytes) 000393 1784124410.129508 write 0x012c6c thru 0x01ae49 (0x81de bytes) 000395 1784124410.130964 read 0x01b914 thru 0x026e6d (0xb55a bytes) 000401 1784124410.133700 mapread 0x009e90 thru 0x0149a4 (0xab15 bytes) 000403 1784124410.135396 write 0x039829 thru 0x03bca3 (0x247b bytes) 000404 1784124410.136434 write 0x0140ff thru 0x01e047 (0x9f49 bytes) 000405 1784124410.137673 read 0x015dcb thru 0x0234ac (0xd6e2 bytes) 000406 1784124410.138174 mapread 0x0301c5 thru 0x03bca3 (0xbadf bytes) 000408 1784124410.142947 trunc from 0x03bca4 to 0x03acb3 (0xff0 bytes) 000410 1784124410.155000 mapread 0x02549f thru 0x027a13 (0x2575 bytes) 000411 1784124410.155705 trunc from 0x03acb4 to 0x02d2c6 (0xd9ed bytes) 000413 1784124410.162988 read 0x029889 thru 0x02d2c6 (0x3a3e bytes) 000415 1784124410.163875 trunc from 0x02d2c7 to 0x01a811 (0x12ab5 bytes) 000416 1784124410.177373 MIRROR_EXTEND to 2 mirrors 000418 1784124410.383392 MIRROR_RESYNC 2 mirrors 000421 1784124410.499031 trunc from 0x01a812 to 0x007b1c (0x12cf5 bytes) 000422 1784124410.535355 mapwrite 0x0228b2 thru 0x029c84 (0x73d3 bytes) 000425 1784124410.594544 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x591:0x0]: Invalid argument (22) 000426 1784124410.659899 trunc from 0x029c85 to 0x039e16 (0x10192 bytes) 000427 1784124410.667579 trunc from 0x039e17 to 0x00d77f (0x2c697 bytes) 000429 1784124410.674757 mapwrite 0x013bdf thru 0x01b5f4 (0x7a16 bytes) 000431 1784124410.715598 write 0x02d137 thru 0x030d8c (0x3c56 bytes) 000432 1784124410.716775 mapwrite 0x0328bb thru 0x039256 (0x699c bytes) 000433 1784124410.743181 MIRROR_EXTEND to 2 mirrors 000435 1784124410.968941 mapread 0x0306bb thru 0x033091 (0x29d7 bytes) 000436 1784124410.977076 write 0x0112f9 thru 0x015e24 (0x4b2c bytes) 000437 1784124411.002574 trunc from 0x039257 to 0x02c094 (0xd1c2 bytes) 000442 1784124411.020100 read 0x0172d5 thru 0x01cf29 (0x5c55 bytes) 000443 1784124411.024153 read 0x012cb7 thru 0x016cc7 (0x4011 bytes) 000444 1784124411.028903 mapread 0x00d3bf thru 0x010b2f (0x3771 bytes) 000446 1784124411.030307 trunc from 0x02c095 to 0x015e1c (0x16278 bytes) 000447 1784124411.039843 mapwrite 0x01de57 thru 0x023fcf (0x6179 bytes) 000448 1784124411.088607 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x23fd0): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23fd0): Mirror 2: 0xe669adc (stale) CRC-32 checksum value for chunk [0, 0x23fd0): Mirror 1: 0x14a678d8 Mirror 3: 0x14a678d8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23fd0: skipped 000449 1784124411.345113 read 0x01329b thru 0x01631c (0x3082 bytes) 000450 1784124411.347539 read 0x016c48 thru 0x01e464 (0x781d bytes) 000451 1784124411.350552 trunc from 0x023fd0 to 0x0114f4 (0x12adb bytes) 000452 1784124411.378968 write 0x01c2be thru 0x021661 (0x53a4 bytes) 000454 1784124411.380528 mapwrite 0x023841 thru 0x02923d (0x59fd bytes) 000455 1784124411.401512 write 0x004a5d thru 0x00c160 (0x7704 bytes) 000456 1784124411.406590 MIRROR_RESYNC 3 mirrors 000462 1784124411.629793 mapwrite 0x006d54 thru 0x00703a (0x2e7 bytes) 000464 1784124411.675654 mapread 0x016c43 thru 0x0250b0 (0xe46e bytes) 000465 1784124411.682640 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2923e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2923e): Mirror 2: 0x7543ff05 (stale) Mirror 3: 0x7543ff05 (stale) CRC-32 checksum value for chunk [0, 0x2923e): Mirror 1: 0x6a42abbc Mirror 4: 0x6a42abbc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2923e: skipped 000468 1784124411.936105 read 0x001924 thru 0x00c11b (0xa7f8 bytes) 000471 1784124411.944057 read 0x0113b7 thru 0x012449 (0x1093 bytes) 000472 1784124411.949566 trunc from 0x02923e to 0x013d33 (0x1550a bytes) 000473 1784124411.981674 trunc from 0x013d34 to 0x03aeca (0x27197 bytes) 000475 1784124411.991511 trunc from 0x03aecb to 0x0001b1 (0x3ad19 bytes) 000477 1784124411.996756 write 0x024c68 thru 0x025f71 (0x130a bytes) 000479 1784124411.998078 mapwrite 0x00b50c thru 0x00cc01 (0x16f6 bytes) 000480 1784124412.016570 write 0x03ff7f thru 0x03ffff (0x81 bytes) 000481 1784124412.017607 trunc from 0x040000 to 0x03794b (0x86b4 bytes) 000484 1784124412.024917 trunc from 0x03794c to 0x0125c9 (0x25382 bytes) 000485 1784124412.030913 mapread 0x008584 thru 0x0125c9 (0xa046 bytes) 000486 1784124412.048804 read 0x010821 thru 0x0125c9 (0x1da9 bytes) 000487 1784124412.049395 write 0x02cea7 thru 0x0346a1 (0x77fb bytes) 000488 1784124412.050482 read 0x01861c thru 0x01d8d1 (0x52b6 bytes) 000489 1784124412.054475 mapread 0x02c6e8 thru 0x0346a1 (0x7fba bytes) 000492 1784124412.056759 read 0x01d2df thru 0x0282e9 (0xb00b bytes) 000495 1784124412.061486 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x346a2): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x346a2): Mirror 2: 0x834c3115 (stale) Mirror 3: 0x834c3115 (stale) Mirror 4: 0x3d71a75a (stale) CRC-32 checksum value for chunk [0, 0x346a2): Mirror 1: 0x51f1d24a Mirror 5: 0x51f1d24a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x346a2: skipped 000497 1784124412.381748 MIRROR_RESYNC 5 mirrors 000498 1784124412.627139 read 0x0333aa thru 0x0346a1 (0x12f8 bytes) 000499 1784124412.629023 mapread 0x00e80d thru 0x00f7e5 (0xfd9 bytes) 000502 1784124412.633672 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x591:0x0]: Invalid argument (22) 000503 1784124412.749829 trunc from 0x0346a2 to 0x01f994 (0x14d0d bytes) 000504 1784124412.775918 mapwrite 0x028056 thru 0x036d7a (0xed25 bytes) 000505 1784124412.815860 trunc from 0x036d7b to 0x027cd3 (0xf0a7 bytes) 000506 1784124412.822140 write 0x003b0e thru 0x00f34a (0xb83d bytes) 000509 1784124412.827793 mapread 0x017c5d thru 0x024fc5 (0xd369 bytes) 000510 1784124412.834769 trunc from 0x027cd4 to 0x006dbc (0x20f17 bytes) 000511 1784124412.842755 mapread 0x0012ca thru 0x006dbc (0x5af3 bytes) 000513 1784124412.846242 mapread 0x00268d thru 0x002928 (0x29c bytes) 000514 1784124412.851140 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x6dbd): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6dbd): Mirror 3: 0xc61a42e5 (stale) Mirror 4: 0xc61a42e5 (stale) Mirror 5: 0xc61a42e5 (stale) CRC-32 checksum value for chunk [0, 0x6dbd): Mirror 1: 0x63f56f0d Mirror 6: 0x63f56f0d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6dbd: skipped 000515 1784124413.129672 mapwrite 0x033366 thru 0x03cc00 (0x989b bytes) 000517 1784124413.191367 MIRROR_RESYNC 5 mirrors 000518 1784124413.486803 write 0x03110d thru 0x038686 (0x757a bytes) 000520 1784124413.511538 read 0x006e5f thru 0x0134fd (0xc69f bytes) 000521 1784124413.515337 read 0x030256 thru 0x038eff (0x8caa bytes) 000522 1784124413.519884 mapread 0x028468 thru 0x02943f (0xfd8 bytes) 000523 1784124413.522631 MIRROR_RESYNC 5 mirrors 000526 1784124413.885179 mapwrite 0x023a4a thru 0x028973 (0x4f2a bytes) 000527 1784124413.987296 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3cc01): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cc01): Mirror 3: 0x99279218 (stale) Mirror 4: 0x99279218 (stale) Mirror 5: 0x99279218 (stale) Mirror 6: 0x99279218 (stale) CRC-32 checksum value for chunk [0, 0x3cc01): Mirror 1: 0x27f2de57 Mirror 7: 0x27f2de57 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cc01: skipped 000530 1784124414.363201 mapread 0x02fdab thru 0x03cc00 (0xce56 bytes) 000532 1784124414.383478 MIRROR_RESYNC 6 mirrors 000533 1784124414.685353 mapread 0x0069a4 thru 0x0141e0 (0xd83d bytes) 000535 1784124414.694453 read 0x0114d3 thru 0x01d154 (0xbc82 bytes) 000536 1784124414.697154 trunc from 0x03cc01 to 0x02786b (0x15395 bytes) 000537 1784124414.725217 read 0x00a775 thru 0x00b7c7 (0x1053 bytes) 000538 1784124414.730758 mapread 0x025754 thru 0x02786b (0x2118 bytes) 000539 1784124414.731852 trunc from 0x02786c to 0x038ca0 (0x11435 bytes) 000541 1784124414.742279 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x38ca1): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38ca1): Mirror 3: 0x1242190a (stale) Mirror 4: 0x1242190a (stale) Mirror 5: 0x1242190a (stale) Mirror 6: 0x1242190a (stale) Mirror 7: 0x1242190a (stale) CRC-32 checksum value for chunk [0, 0x38ca1): Mirror 1: 0x55780eb9 Mirror 8: 0x55780eb9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38ca1: skipped 000542 1784124415.078739 trunc from 0x038ca1 to 0x02c8a0 (0xc400 bytes) 000543 1784124415.135787 mapwrite 0x03b879 thru 0x03f343 (0x3acb bytes) 000545 1784124415.185109 mapwrite 0x02a4e9 thru 0x02f0a1 (0x4bb9 bytes) 000546 1784124415.222224 MIRROR_RESYNC 7 mirrors 000547 1784124415.641298 read 0x00b762 thru 0x01a58a (0xee29 bytes) 000548 1784124415.648787 write 0x02bff3 thru 0x02e373 (0x2381 bytes) 000549 1784124415.689768 read 0x0374d3 thru 0x037d3e (0x86c bytes) 000554 1784124415.702481 trunc from 0x03f344 to 0x033440 (0xbf03 bytes) 000556 1784124415.738145 MIRROR_RESYNC 7 mirrors 000563 1784124416.216014 write 0x0396f2 thru 0x03ffff (0x690e bytes) 000566 1784124416.238435 trunc from 0x040000 to 0x00abee (0x35411 bytes) 000568 1784124416.249282 write 0x0227de thru 0x02477b (0x1f9e bytes) 000569 1784124416.252093 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2477c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2477c): Mirror 3: 0x58a38bd (stale) Mirror 4: 0x58a38bd (stale) Mirror 5: 0x58a38bd (stale) Mirror 6: 0x58a38bd (stale) Mirror 7: 0x58a38bd (stale) Mirror 8: 0x58a38bd (stale) CRC-32 checksum value for chunk [0, 0x2477c): Mirror 1: 0x7c02a493 Mirror 9: 0x7c02a493 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2477c: skipped 000572 1784124416.687647 mapwrite 0x00d165 thru 0x0119e5 (0x4881 bytes) 000574 1784124416.768750 trunc from 0x02477c to 0x028266 (0x3aeb bytes) 000575 1784124416.778026 write 0x02bc1e thru 0x03b5b3 (0xf996 bytes) 000577 1784124416.779776 mapread 0x013026 thru 0x013565 (0x540 bytes) 000578 1784124416.780568 read 0x0017bd thru 0x0091f0 (0x7a34 bytes) 000579 1784124416.785941 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3b5b4): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b5b4): Mirror 3: 0x988dc962 (stale) Mirror 4: 0x988dc962 (stale) Mirror 5: 0x988dc962 (stale) Mirror 6: 0x988dc962 (stale) Mirror 7: 0x988dc962 (stale) Mirror 8: 0x988dc962 (stale) Mirror 9: 0x4f313d04 (stale) CRC-32 checksum value for chunk [0, 0x3b5b4): Mirror 1: 0xf2fb59f6 Mirror 10: 0xf2fb59f6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b5b4: skipped 000581 1784124417.117362 trunc from 0x03b5b4 to 0x01a957 (0x20c5c bytes) 000583 1784124417.155944 write 0x02e12a thru 0x0386fd (0xa5d4 bytes) 000584 1784124417.157043 write 0x03e5d4 thru 0x03ffff (0x1a2c bytes) 000588 1784124417.161937 MIRROR_RESYNC 9 mirrors 000592 1784124417.612816 trunc from 0x040000 to 0x012043 (0x2dfbc bytes) 000593 1784124417.633996 write 0x025f8f thru 0x027353 (0x13c5 bytes) 000597 1784124417.636472 mapwrite 0x00b9a0 thru 0x00d78d (0x1dee bytes) 000600 1784124417.661467 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x27354): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27354): Mirror 3: 0xa97b33d4 (stale) Mirror 4: 0xa97b33d4 (stale) Mirror 5: 0xa97b33d4 (stale) Mirror 6: 0xa97b33d4 (stale) Mirror 7: 0xa97b33d4 (stale) Mirror 8: 0xa97b33d4 (stale) Mirror 9: 0xa97b33d4 (stale) Mirror 10: 0xa97b33d4 (stale) CRC-32 checksum value for chunk [0, 0x27354): Mirror 1: 0x1a43ae82 Mirror 11: 0x1a43ae82 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27354: skipped 000601 1784124417.988017 mapread 0x00c741 thru 0x015e2a (0x96ea bytes) 000603 1784124417.999750 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x591:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27354): Mirror 3: 0xa97b33d4 (stale) Mirror 4: 0xa97b33d4 (stale) Mirror 5: 0xa97b33d4 (stale) Mirror 6: 0xa97b33d4 (stale) Mirror 7: 0xa97b33d4 (stale) Mirror 8: 0xa97b33d4 (stale) Mirror 9: 0xa97b33d4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27354): Mirror 3: 0xa97b33d4 (stale) Mirror 4: 0xa97b33d4 (stale) Mirror 5: 0xa97b33d4 (stale) Mirror 6: 0xa97b33d4 (stale) Mirror 7: 0xa97b33d4 (stale) Mirror 8: 0xa97b33d4 (stale) Mirror 9: 0xa97b33d4 (stale) CRC-32 checksum value for chunk [0, 0x27354): Mirror 1: 0x1a43ae82 Mirror 11: 0x1a43ae82 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27354: skipped 000604 1784124418.262062 read 0x009871 thru 0x011d52 (0x84e2 bytes) 000606 1784124418.270017 write 0x03b048 thru 0x03ffff (0x4fb8 bytes) 000607 1784124418.293359 write 0x003704 thru 0x01346a (0xfd67 bytes) 000608 1784124418.297320 mapwrite 0x01e4c6 thru 0x025c4c (0x7787 bytes) 000609 1784124418.319528 write 0x01df02 thru 0x023f0d (0x600c bytes) 000610 1784124418.320514 mapwrite 0x0376dd thru 0x03ffff (0x8923 bytes) 000611 1784124418.339101 MIRROR_RESYNC 9 mirrors 000613 1784124418.688883 MIRROR_EXTEND to 10 mirrors 000614 1784124418.885257 write 0x038c8f thru 0x03ffff (0x7371 bytes) 000617 1784124418.905869 mapwrite 0x02dc3b thru 0x03caef (0xeeb5 bytes) 000618 1784124418.925791 mapread 0x01defc thru 0x029157 (0xb25c bytes) 000621 1784124418.931739 write 0x020c71 thru 0x03078e (0xfb1e bytes) 000627 1784124418.934943 mapread 0x014bc5 thru 0x015574 (0x9b0 bytes) 000628 1784124418.938432 mapread 0x010809 thru 0x01a61d (0x9e15 bytes) 000630 1784124418.939613 mapread 0x00c23b thru 0x00f313 (0x30d9 bytes) 000632 1784124418.942837 write 0x0001b7 thru 0x00c249 (0xc093 bytes) 000634 1784124418.945372 write 0x0070d4 thru 0x008dba (0x1ce7 bytes) 000635 1784124418.946081 mapwrite 0x00fa1a thru 0x01e01c (0xe603 bytes) 000636 1784124418.960940 read 0x002642 thru 0x005c8c (0x364b bytes) 000637 1784124418.961684 mapread 0x03837b thru 0x03a5e8 (0x226e bytes) 000638 1784124418.962702 mapwrite 0x006efe thru 0x00f96f (0x8a72 bytes) 000639 1784124418.975006 mapread 0x0335ae thru 0x03ffff (0xca52 bytes) 000640 1784124418.977221 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x591:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x9391a4b4 (stale) Mirror 4: 0x9391a4b4 (stale) Mirror 5: 0x9391a4b4 (stale) Mirror 6: 0x9391a4b4 (stale) Mirror 7: 0x9391a4b4 (stale) Mirror 9: 0x9391a4b4 (stale) Mirror 11: 0x9391a4b4 (stale) Mirror 12: 0x9391a4b4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x9391a4b4 (stale) Mirror 4: 0x9391a4b4 (stale) Mirror 5: 0x9391a4b4 (stale) Mirror 6: 0x9391a4b4 (stale) Mirror 7: 0x9391a4b4 (stale) Mirror 9: 0x9391a4b4 (stale) Mirror 11: 0x9391a4b4 (stale) Mirror 12: 0x9391a4b4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000641 1784124419.129950 trunc from 0x040000 to 0x02c9f8 (0x13607 bytes) 000644 1784124419.140313 write 0x031916 thru 0x03b20b (0x98f6 bytes) 000647 1784124419.142224 trunc from 0x03b20c to 0x003bb4 (0x37657 bytes) 000650 1784124419.148600 write 0x02fe1a thru 0x03bd0b (0xbef2 bytes) 000654 1784124419.152704 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3bd0c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd0c): Mirror 3: 0xa370a4f4 (stale) Mirror 4: 0xa370a4f4 (stale) Mirror 5: 0xa370a4f4 (stale) Mirror 6: 0xa370a4f4 (stale) Mirror 7: 0xa370a4f4 (stale) Mirror 9: 0xa370a4f4 (stale) Mirror 11: 0xa370a4f4 (stale) Mirror 12: 0xa370a4f4 (stale) CRC-32 checksum value for chunk [0, 0x3bd0c): Mirror 1: 0x3d7ebb50 Mirror 13: 0x3d7ebb50 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd0c: skipped 000655 1784124419.473640 mapwrite 0x0360d5 thru 0x038644 (0x2570 bytes) 000656 1784124419.532073 mapwrite 0x03900f thru 0x03a0c9 (0x10bb bytes) 000658 1784124419.544381 read 0x010704 thru 0x019fae (0x98ab bytes) 000659 1784124419.548999 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3bd0c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd0c): Mirror 3: 0xa370a4f4 (stale) Mirror 4: 0xa370a4f4 (stale) Mirror 5: 0xa370a4f4 (stale) Mirror 6: 0xa370a4f4 (stale) Mirror 7: 0xa370a4f4 (stale) Mirror 9: 0xa370a4f4 (stale) Mirror 11: 0xa370a4f4 (stale) Mirror 12: 0xa370a4f4 (stale) Mirror 13: 0x3d7ebb50 (stale) CRC-32 checksum value for chunk [0, 0x3bd0c): Mirror 1: 0x461e5ce1 Mirror 14: 0x461e5ce1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd0c: skipped 000661 1784124419.869199 read 0x0382cf thru 0x03bd0b (0x3a3d bytes) 000662 1784124419.873073 mapwrite 0x02b582 thru 0x031c96 (0x6715 bytes) 000663 1784124419.935087 mapread 0x00f239 thru 0x011daf (0x2b77 bytes) 000664 1784124419.938722 mapread 0x035514 thru 0x036813 (0x1300 bytes) 000666 1784124419.942737 trunc from 0x03bd0c to 0x03c309 (0x5fe bytes) 000667 1784124419.954454 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3c30a): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c30a): Mirror 3: 0xd04d2816 (stale) Mirror 4: 0xd04d2816 (stale) Mirror 5: 0xd04d2816 (stale) Mirror 6: 0xd04d2816 (stale) Mirror 7: 0xd04d2816 (stale) Mirror 9: 0xd04d2816 (stale) Mirror 11: 0xd04d2816 (stale) Mirror 12: 0xd04d2816 (stale) Mirror 13: 0x3593b66d (stale) Mirror 14: 0x972fa991 (stale) CRC-32 checksum value for chunk [0, 0x3c30a): Mirror 1: 0x4371bc73 Mirror 15: 0x4371bc73 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c30a: skipped 000671 1784124420.339168 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x591:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c30a): Mirror 4: 0xd04d2816 (stale) Mirror 5: 0xd04d2816 (stale) Mirror 6: 0xd04d2816 (stale) Mirror 7: 0xd04d2816 (stale) Mirror 9: 0xd04d2816 (stale) Mirror 11: 0xd04d2816 (stale) Mirror 12: 0xd04d2816 (stale) Mirror 13: 0x3593b66d (stale) Mirror 14: 0x972fa991 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c30a): Mirror 4: 0xd04d2816 (stale) Mirror 5: 0xd04d2816 (stale) Mirror 6: 0xd04d2816 (stale) Mirror 7: 0xd04d2816 (stale) Mirror 9: 0xd04d2816 (stale) Mirror 11: 0xd04d2816 (stale) Mirror 12: 0xd04d2816 (stale) Mirror 13: 0x3593b66d (stale) Mirror 14: 0x972fa991 (stale) CRC-32 checksum value for chunk [0, 0x3c30a): Mirror 1: 0x4371bc73 Mirror 15: 0x4371bc73 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c30a: skipped 000672 1784124420.682579 trunc from 0x03c30a to 0x03bd13 (0x5f6 bytes) 000673 1784124420.704794 mapread 0x00167e thru 0x00a6b9 (0x903c bytes) 000674 1784124420.711274 read 0x037eb2 thru 0x03bd13 (0x3e62 bytes) 000675 1784124420.714326 read 0x01fd76 thru 0x02541b (0x56a6 bytes) 000676 1784124420.718345 mapwrite 0x033b0d thru 0x038a87 (0x4f7b bytes) 000678 1784124420.734549 mapwrite 0x01843a thru 0x0195c0 (0x1187 bytes) 000683 1784124420.748292 mapwrite 0x00e68a thru 0x0111fd (0x2b74 bytes) 000685 1784124420.762697 MIRROR_RESYNC 11 mirrors 000687 1784124421.202428 read 0x02bd1d thru 0x02fb1b (0x3dff bytes) 000688 1784124421.205741 read 0x01a2e9 thru 0x0277ac (0xd4c4 bytes) 000689 1784124421.209928 mapread 0x016dd1 thru 0x020a5f (0x9c8f bytes) 000690 1784124421.210733 write 0x03fcef thru 0x03ffff (0x311 bytes) 000691 1784124421.229139 mapread 0x036b88 thru 0x03ffff (0x9478 bytes) 000695 1784124421.239611 write 0x02e388 thru 0x03c064 (0xdcdd bytes) 000697 1784124421.243222 mapwrite 0x034b57 thru 0x03ffff (0xb4a9 bytes) 000698 1784124421.262255 mapread 0x02830c thru 0x03554a (0xd23f bytes) 000700 1784124421.273931 mapwrite 0x022feb thru 0x02410f (0x1125 bytes) 000701 1784124421.293174 trunc from 0x040000 to 0x02ceae (0x13151 bytes) 000702 1784124421.299998 trunc from 0x02ceaf to 0x037857 (0xa9a9 bytes) 000703 1784124421.304809 trunc from 0x037858 to 0x03bdb2 (0x455b bytes) 000704 1784124421.313475 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x591:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bdb3): Mirror 4: 0x9fa26450 (stale) Mirror 5: 0x9fa26450 (stale) Mirror 6: 0x9fa26450 (stale) Mirror 7: 0x9fa26450 (stale) Mirror 11: 0x9fa26450 (stale) Mirror 12: 0x9fa26450 (stale) Mirror 13: 0x9fa26450 (stale) Mirror 14: 0x9fa26450 (stale) Mirror 15: 0x9fa26450 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bdb3): Mirror 4: 0x9fa26450 (stale) Mirror 5: 0x9fa26450 (stale) Mirror 6: 0x9fa26450 (stale) Mirror 7: 0x9fa26450 (stale) Mirror 11: 0x9fa26450 (stale) Mirror 12: 0x9fa26450 (stale) Mirror 13: 0x9fa26450 (stale) Mirror 14: 0x9fa26450 (stale) Mirror 15: 0x9fa26450 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bdb3: skipped 000706 1784124421.511217 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x591:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bdb3): Mirror 5: 0x9fa26450 (stale) Mirror 6: 0x9fa26450 (stale) Mirror 7: 0x9fa26450 (stale) Mirror 11: 0x9fa26450 (stale) Mirror 12: 0x9fa26450 (stale) Mirror 13: 0x9fa26450 (stale) Mirror 14: 0x9fa26450 (stale) Mirror 15: 0x9fa26450 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bdb3): Mirror 5: 0x9fa26450 (stale) Mirror 6: 0x9fa26450 (stale) Mirror 7: 0x9fa26450 (stale) Mirror 11: 0x9fa26450 (stale) Mirror 12: 0x9fa26450 (stale) Mirror 13: 0x9fa26450 (stale) Mirror 14: 0x9fa26450 (stale) Mirror 15: 0x9fa26450 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bdb3: skipped 000707 1784124421.663314 read 0x019f27 thru 0x0221bb (0x8295 bytes) 000708 1784124421.665902 mapwrite 0x00a093 thru 0x0137c3 (0x9731 bytes) 000709 1784124421.683532 write 0x025fb7 thru 0x0293bf (0x3409 bytes) 000710 1784124421.687322 mapread 0x018cdb thru 0x021719 (0x8a3f bytes) 000711 1784124421.689985 mapread 0x020088 thru 0x021184 (0x10fd bytes) 000712 1784124421.690720 mapwrite 0x004fb4 thru 0x014e65 (0xfeb2 bytes) 000713 1784124421.708247 trunc from 0x03bdb3 to 0x031b8d (0xa225 bytes) 000714 1784124421.715016 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x591:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31b8e): Mirror 5: 0xeba6f479 (stale) Mirror 6: 0xeba6f479 (stale) Mirror 7: 0xeba6f479 (stale) Mirror 11: 0xeba6f479 (stale) Mirror 12: 0xeba6f479 (stale) Mirror 14: 0xeba6f479 (stale) Mirror 15: 0xeba6f479 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31b8e): Mirror 5: 0xeba6f479 (stale) Mirror 6: 0xeba6f479 (stale) Mirror 7: 0xeba6f479 (stale) Mirror 11: 0xeba6f479 (stale) Mirror 12: 0xeba6f479 (stale) Mirror 14: 0xeba6f479 (stale) Mirror 15: 0xeba6f479 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31b8e: skipped 000716 1784124421.856629 mapwrite 0x019783 thru 0x022fdc (0x985a bytes) 000718 1784124421.873649 mapwrite 0x010efe thru 0x011b38 (0xc3b bytes) 000719 1784124421.885469 read 0x003a37 thru 0x006332 (0x28fc bytes) 000720 1784124421.887697 mapread 0x0018a7 thru 0x003550 (0x1caa bytes) 000721 1784124421.890093 read 0x02d7da thru 0x031b8d (0x43b4 bytes) 000722 1784124421.892836 read 0x022dae thru 0x031b8d (0xede0 bytes) 000723 1784124421.893351 read 0x01dc17 thru 0x02ba7a (0xde64 bytes) 000724 1784124421.893822 trunc from 0x031b8e to 0x02952b (0x8662 bytes) 000725 1784124421.898186 mapread 0x01b06f thru 0x024c4f (0x9be1 bytes) 000726 1784124421.899084 trunc from 0x02952c to 0x0309ce (0x74a3 bytes) 000727 1784124421.903397 mapwrite 0x03c9c4 thru 0x03e9fa (0x2037 bytes) 000728 1784124421.919944 mapwrite 0x004d43 thru 0x009142 (0x4400 bytes) 000729 1784124421.932116 mapread 0x0177e5 thru 0x01e751 (0x6f6d bytes) 000730 1784124421.934567 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x591:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e9fb): Mirror 5: 0x7743a675 (stale) Mirror 6: 0x7743a675 (stale) Mirror 7: 0x7743a675 (stale) Mirror 12: 0x7743a675 (stale) Mirror 14: 0x7743a675 (stale) Mirror 15: 0x7743a675 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e9fb): Mirror 5: 0x7743a675 (stale) Mirror 6: 0x7743a675 (stale) Mirror 7: 0x7743a675 (stale) Mirror 12: 0x7743a675 (stale) Mirror 14: 0x7743a675 (stale) Mirror 15: 0x7743a675 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e9fb: skipped 000731 1784124422.051691 trunc from 0x03e9fb to 0x03ce69 (0x1b91 bytes) 000732 1784124422.056653 read 0x0326a7 thru 0x0336b9 (0x1013 bytes) 000733 1784124422.058997 write 0x03604f thru 0x039e97 (0x3e49 bytes) 000736 1784124422.064437 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3ce6a): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ce6a): Mirror 5: 0x80ad8fc (stale) Mirror 6: 0x80ad8fc (stale) Mirror 7: 0x80ad8fc (stale) Mirror 12: 0x80ad8fc (stale) Mirror 14: 0x80ad8fc (stale) Mirror 15: 0x80ad8fc (stale) CRC-32 checksum value for chunk [0, 0x3ce6a): Mirror 1: 0xb389e8d1 Mirror 16: 0xb389e8d1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ce6a: skipped 000739 1784124422.371809 mapwrite 0x0015f5 thru 0x0037b7 (0x21c3 bytes) 000742 1784124422.435528 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3ce6a): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ce6a): Mirror 5: 0x80ad8fc (stale) Mirror 6: 0x80ad8fc (stale) Mirror 7: 0x80ad8fc (stale) Mirror 12: 0x80ad8fc (stale) Mirror 14: 0x80ad8fc (stale) Mirror 15: 0x80ad8fc (stale) Mirror 16: 0xb389e8d1 (stale) CRC-32 checksum value for chunk [0, 0x3ce6a): Mirror 1: 0x4ce637dd Mirror 17: 0x4ce637dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ce6a: skipped 000744 1784124422.724421 MIRROR_RESYNC 9 mirrors 000745 1784124423.083319 read 0x00493d thru 0x011e2b (0xd4ef bytes) 000746 1784124423.086332 mapwrite 0x00a917 thru 0x019e3c (0xf526 bytes) 000748 1784124423.142674 mapwrite 0x030be5 thru 0x036917 (0x5d33 bytes) 000749 1784124423.159597 read 0x01eba7 thru 0x028695 (0x9aef bytes) 000750 1784124423.163430 mapwrite 0x00c217 thru 0x017a55 (0xb83f bytes) 000751 1784124423.177687 mapwrite 0x01d4f4 thru 0x026621 (0x912e bytes) 000753 1784124423.194011 trunc from 0x03ce6a to 0x036579 (0x68f0 bytes) 000754 1784124423.207092 trunc from 0x03657a to 0x0298da (0xcc9f bytes) 000755 1784124423.214062 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x298db): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x298db): Mirror 5: 0xe8fd8d3 (stale) Mirror 6: 0xe8fd8d3 (stale) Mirror 7: 0xe8fd8d3 (stale) Mirror 12: 0xe8fd8d3 (stale) Mirror 14: 0xe8fd8d3 (stale) Mirror 15: 0xe8fd8d3 (stale) Mirror 16: 0xe8fd8d3 (stale) Mirror 17: 0xe8fd8d3 (stale) CRC-32 checksum value for chunk [0, 0x298db): Mirror 1: 0x7d45830f Mirror 18: 0x7d45830f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x298db: skipped 000757 1784124423.517648 read 0x01217d thru 0x01b620 (0x94a4 bytes) 000758 1784124423.524174 MIRROR_RESYNC 10 mirrors 000760 1784124423.898703 write 0x033062 thru 0x037322 (0x42c1 bytes) 000761 1784124423.921677 mapwrite 0x03fe7b thru 0x03ffff (0x185 bytes) 000762 1784124423.949544 mapwrite 0x01401d thru 0x018a39 (0x4a1d bytes) 000766 1784124423.969017 read 0x038a1b thru 0x03ffff (0x75e5 bytes) 000767 1784124423.972369 trunc from 0x040000 to 0x03e63f (0x19c0 bytes) 000771 1784124423.977248 mapread 0x0184c5 thru 0x01a11c (0x1c58 bytes) 000773 1784124423.978284 read 0x01e6dc thru 0x02b03b (0xc960 bytes) 000774 1784124423.981240 mapwrite 0x022df1 thru 0x02a680 (0x7890 bytes) 000775 1784124423.994567 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3e640): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e640): Mirror 5: 0xd119b06c (stale) Mirror 6: 0xd119b06c (stale) Mirror 7: 0xd119b06c (stale) Mirror 12: 0xd119b06c (stale) Mirror 14: 0xd119b06c (stale) Mirror 15: 0xd119b06c (stale) Mirror 16: 0xd119b06c (stale) Mirror 17: 0xd119b06c (stale) Mirror 18: 0xd119b06c (stale) CRC-32 checksum value for chunk [0, 0x3e640): Mirror 1: 0x470246ac Mirror 19: 0x470246ac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e640: skipped 000776 1784124424.300519 write 0x01bb2b thru 0x02355c (0x7a32 bytes) 000777 1784124424.325860 write 0x00e993 thru 0x015a98 (0x7106 bytes) 000778 1784124424.329949 write 0x00fa30 thru 0x016425 (0x69f6 bytes) 000781 1784124424.334007 trunc from 0x03e640 to 0x0044e1 (0x3a15e bytes) 000782 1784124424.341095 mapwrite 0x029d23 thru 0x037558 (0xd836 bytes) 000783 1784124424.368748 mapread 0x01a814 thru 0x025b2f (0xb31c bytes) 000785 1784124424.374545 write 0x019d73 thru 0x019f8f (0x21d bytes) 000788 1784124424.376299 mapwrite 0x027894 thru 0x036637 (0xeda4 bytes) 000790 1784124424.390613 MIRROR_RESYNC 11 mirrors 000791 1784124424.788027 mapread 0x01a7ea thru 0x023312 (0x8b29 bytes) 000792 1784124424.793779 write 0x00a31c thru 0x00f150 (0x4e35 bytes) 000793 1784124424.816241 read 0x016d3c thru 0x01c839 (0x5afe bytes) 000795 1784124424.819838 write 0x00c41c thru 0x00f925 (0x350a bytes) 000796 1784124424.820551 mapread 0x00022c thru 0x002424 (0x21f9 bytes) 000798 1784124424.825954 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x591:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37559): Mirror 5: 0x3ae4d2a (stale) Mirror 6: 0x3ae4d2a (stale) Mirror 7: 0x3ae4d2a (stale) Mirror 12: 0x3ae4d2a (stale) Mirror 14: 0x3ae4d2a (stale) Mirror 15: 0x3ae4d2a (stale) Mirror 16: 0x3ae4d2a (stale) Mirror 17: 0x3ae4d2a (stale) Mirror 19: 0x3ae4d2a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37559): Mirror 5: 0x3ae4d2a (stale) Mirror 6: 0x3ae4d2a (stale) Mirror 7: 0x3ae4d2a (stale) Mirror 12: 0x3ae4d2a (stale) Mirror 14: 0x3ae4d2a (stale) Mirror 15: 0x3ae4d2a (stale) Mirror 16: 0x3ae4d2a (stale) Mirror 17: 0x3ae4d2a (stale) Mirror 19: 0x3ae4d2a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37559: skipped 000800 1784124425.004551 read 0x0070dc thru 0x013b40 (0xca65 bytes) 000802 1784124425.008386 read 0x02bd56 thru 0x037558 (0xb803 bytes) 000805 1784124425.012886 trunc from 0x037559 to 0x0236ca (0x13e8e bytes) 000807 1784124425.022846 mapread 0x001765 thru 0x008bbf (0x745b bytes) 000809 1784124425.023969 write 0x01b0eb thru 0x01eb04 (0x3a1a bytes) 000810 1784124425.024626 write 0x01f359 thru 0x0245ba (0x5262 bytes) 000811 1784124425.025322 trunc from 0x0245bb to 0x00dbb6 (0x16a04 bytes) 000813 1784124425.030811 read 0x008485 thru 0x00dbb6 (0x5732 bytes) 000814 1784124425.031454 write 0x003cc0 thru 0x00496e (0xcaf bytes) 000817 1784124425.033638 mapwrite 0x0134a1 thru 0x01fd9a (0xc8fa bytes) 000818 1784124425.059685 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x1fd9b): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fd9b): Mirror 5: 0x6ab73c7c (stale) Mirror 6: 0x6ab73c7c (stale) Mirror 7: 0x6ab73c7c (stale) Mirror 12: 0x6ab73c7c (stale) Mirror 14: 0x6ab73c7c (stale) Mirror 15: 0x6ab73c7c (stale) Mirror 16: 0x6ab73c7c (stale) Mirror 17: 0x6ab73c7c (stale) Mirror 19: 0x6ab73c7c (stale) CRC-32 checksum value for chunk [0, 0x1fd9b): Mirror 1: 0xcd2e83f5 Mirror 20: 0xcd2e83f5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fd9b: skipped 000819 1784124425.311984 trunc from 0x01fd9b to 0x02f659 (0xf8bf bytes) 000820 1784124425.344482 mapwrite 0x0348a6 thru 0x03fbd8 (0xb333 bytes) 000822 1784124425.386901 trunc from 0x03fbd9 to 0x00a897 (0x35341 bytes) 000824 1784124425.393461 read 0x0045aa thru 0x00a897 (0x62ee bytes) 000827 1784124425.397630 write 0x02efa2 thru 0x036c16 (0x7c75 bytes) 000828 1784124425.398560 write 0x005c62 thru 0x010fd9 (0xb378 bytes) 000829 1784124425.401132 mapread 0x020abf thru 0x02baa0 (0xafe2 bytes) 000830 1784124425.407152 write 0x007b8f thru 0x00e24e (0x66c0 bytes) 000832 1784124425.408894 mapwrite 0x01fb76 thru 0x029963 (0x9dee bytes) 000834 1784124425.426747 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x36c17): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36c17): Mirror 5: 0x5cc45f13 (stale) Mirror 6: 0x5cc45f13 (stale) Mirror 7: 0x5cc45f13 (stale) Mirror 12: 0x5cc45f13 (stale) Mirror 14: 0x5cc45f13 (stale) Mirror 15: 0x5cc45f13 (stale) Mirror 16: 0x5cc45f13 (stale) Mirror 17: 0x5cc45f13 (stale) Mirror 19: 0x5cc45f13 (stale) Mirror 20: 0x1cc6b81c (stale) CRC-32 checksum value for chunk [0, 0x36c17): Mirror 1: 0x776317b9 Mirror 21: 0x776317b9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36c17: skipped 000835 1784124425.692087 trunc from 0x036c17 to 0x01cba3 (0x1a073 bytes) 000836 1784124425.721802 write 0x018106 thru 0x01f3c7 (0x72c2 bytes) 000838 1784124425.725000 write 0x015e0e thru 0x01f62f (0x9822 bytes) 000839 1784124425.728094 mapread 0x015cc0 thru 0x01f62f (0x9970 bytes) 000840 1784124425.729183 mapwrite 0x0124cf thru 0x01fafb (0xd62d bytes) 000842 1784124425.760946 trunc from 0x01fafc to 0x03448b (0x14990 bytes) 000844 1784124425.766957 trunc from 0x03448c to 0x01b019 (0x19472 bytes) 000845 1784124425.771896 read 0x013c11 thru 0x018236 (0x4626 bytes) 000846 1784124425.772352 mapread 0x00f229 thru 0x01b019 (0xbdf1 bytes) 000847 1784124425.773354 trunc from 0x01b01a to 0x01ab5c (0x4bd bytes) 000848 1784124425.778764 mapread 0x00e116 thru 0x01ab5c (0xca47 bytes) 000849 1784124425.779973 trunc from 0x01ab5d to 0x017387 (0x37d5 bytes) 000850 1784124425.786207 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x17388): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17388): Mirror 5: 0x7772e1db (stale) Mirror 6: 0x7772e1db (stale) Mirror 7: 0x7772e1db (stale) Mirror 12: 0x7772e1db (stale) Mirror 14: 0x7772e1db (stale) Mirror 15: 0x7772e1db (stale) Mirror 16: 0x7772e1db (stale) Mirror 17: 0x7772e1db (stale) Mirror 19: 0x7772e1db (stale) Mirror 20: 0x7b8b0c57 (stale) Mirror 21: 0xe3f5d55d (stale) CRC-32 checksum value for chunk [0, 0x17388): Mirror 1: 0x4acc4247 Mirror 22: 0x4acc4247 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17388: skipped 000851 1784124426.051681 mapwrite 0x033efe thru 0x03ffff (0xc102 bytes) 000852 1784124426.110645 mapwrite 0x025c5b thru 0x027477 (0x181d bytes) 000854 1784124426.126525 mapread 0x0247d1 thru 0x0279d8 (0x3208 bytes) 000856 1784124426.129322 MIRROR_SPLIT mirror 9 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x591:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xb5e64ff (stale) Mirror 6: 0xb5e64ff (stale) Mirror 7: 0xb5e64ff (stale) Mirror 12: 0xb5e64ff (stale) Mirror 14: 0xb5e64ff (stale) Mirror 15: 0xb5e64ff (stale) Mirror 16: 0xb5e64ff (stale) Mirror 17: 0xb5e64ff (stale) Mirror 20: 0x518c2e90 (stale) Mirror 21: 0x15ed60fa (stale) Mirror 22: 0xebcd9e0e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xb5e64ff (stale) Mirror 6: 0xb5e64ff (stale) Mirror 7: 0xb5e64ff (stale) Mirror 12: 0xb5e64ff (stale) Mirror 14: 0xb5e64ff (stale) Mirror 15: 0xb5e64ff (stale) Mirror 16: 0xb5e64ff (stale) Mirror 17: 0xb5e64ff (stale) Mirror 20: 0x518c2e90 (stale) Mirror 21: 0x15ed60fa (stale) Mirror 22: 0xebcd9e0e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000858 1784124426.340321 mapwrite 0x019c6b thru 0x01f18c (0x5522 bytes) 000865 1784124426.361564 trunc from 0x040000 to 0x03c283 (0x3d7c bytes) 000868 1784124426.367138 mapread 0x026d9e thru 0x033b70 (0xcdd3 bytes) 000870 1784124426.373890 trunc from 0x03c284 to 0x02d629 (0xec5a bytes) 000871 1784124426.379767 mapwrite 0x010639 thru 0x014ebe (0x4886 bytes) 000872 1784124426.399796 read 0x0123c7 thru 0x020403 (0xe03d bytes) 000876 1784124426.401640 trunc from 0x02d62a to 0x00dd65 (0x1f8c4 bytes) 000878 1784124426.410615 write 0x03385b thru 0x0386d0 (0x4e76 bytes) 000879 1784124426.411902 trunc from 0x0386d1 to 0x03095f (0x7d71 bytes) 000881 1784124426.418923 mapread 0x0104af thru 0x013f7c (0x3ace bytes) 000883 1784124426.424058 mapwrite 0x03ab47 thru 0x03ffff (0x54b9 bytes) 000884 1784124426.452002 read 0x02b5e9 thru 0x0359c9 (0xa3e1 bytes) 000889 1784124426.462968 mapread 0x000337 thru 0x009632 (0x92fc bytes) 000890 1784124426.463967 mapread 0x036595 thru 0x03ab20 (0x458c bytes) 000891 1784124426.466980 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x591:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xb5e64ff (stale) Mirror 6: 0xb5e64ff (stale) Mirror 7: 0xb5e64ff (stale) Mirror 12: 0xb5e64ff (stale) Mirror 14: 0xb5e64ff (stale) Mirror 15: 0xb5e64ff (stale) Mirror 16: 0xb5e64ff (stale) Mirror 17: 0xb5e64ff (stale) Mirror 20: 0x518c2e90 (stale) Mirror 22: 0xebcd9e0e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xb5e64ff (stale) Mirror 6: 0xb5e64ff (stale) Mirror 7: 0xb5e64ff (stale) Mirror 12: 0xb5e64ff (stale) Mirror 14: 0xb5e64ff (stale) Mirror 15: 0xb5e64ff (stale) Mirror 16: 0xb5e64ff (stale) Mirror 17: 0xb5e64ff (stale) Mirror 20: 0x518c2e90 (stale) Mirror 22: 0xebcd9e0e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000892 1784124426.650072 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x591:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xb5e64ff (stale) Mirror 6: 0xb5e64ff (stale) Mirror 7: 0xb5e64ff (stale) Mirror 12: 0xb5e64ff (stale) Mirror 14: 0xb5e64ff (stale) Mirror 16: 0xb5e64ff (stale) Mirror 17: 0xb5e64ff (stale) Mirror 20: 0x518c2e90 (stale) Mirror 22: 0xebcd9e0e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xb5e64ff (stale) Mirror 6: 0xb5e64ff (stale) Mirror 7: 0xb5e64ff (stale) Mirror 12: 0xb5e64ff (stale) Mirror 14: 0xb5e64ff (stale) Mirror 16: 0xb5e64ff (stale) Mirror 17: 0xb5e64ff (stale) Mirror 20: 0x518c2e90 (stale) Mirror 22: 0xebcd9e0e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000894 1784124426.810165 write 0x01d4ac thru 0x0298c9 (0xc41e bytes) 000895 1784124426.814388 mapwrite 0x03c922 thru 0x03ffff (0x36de bytes) 000896 1784124426.837943 mapread 0x03bf63 thru 0x03e452 (0x24f0 bytes) 000897 1784124426.841068 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x591:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xb5e64ff (stale) Mirror 6: 0xb5e64ff (stale) Mirror 7: 0xb5e64ff (stale) Mirror 14: 0xb5e64ff (stale) Mirror 16: 0xb5e64ff (stale) Mirror 17: 0xb5e64ff (stale) Mirror 20: 0x518c2e90 (stale) Mirror 22: 0xebcd9e0e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xb5e64ff (stale) Mirror 6: 0xb5e64ff (stale) Mirror 7: 0xb5e64ff (stale) Mirror 14: 0xb5e64ff (stale) Mirror 16: 0xb5e64ff (stale) Mirror 17: 0xb5e64ff (stale) Mirror 20: 0x518c2e90 (stale) Mirror 22: 0xebcd9e0e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000898 1784124427.033674 mapwrite 0x02cca6 thru 0x038842 (0xbb9d bytes) 000899 1784124427.058524 write 0x02bd39 thru 0x02e6e4 (0x29ac bytes) 000900 1784124427.059319 mapwrite 0x00e37d thru 0x012082 (0x3d06 bytes) 000901 1784124427.077699 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xb5e64ff (stale) Mirror 6: 0xb5e64ff (stale) Mirror 7: 0xb5e64ff (stale) Mirror 14: 0xb5e64ff (stale) Mirror 16: 0xb5e64ff (stale) Mirror 17: 0xb5e64ff (stale) Mirror 20: 0x518c2e90 (stale) Mirror 22: 0xebcd9e0e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8ef485e1 Mirror 23: 0x8ef485e1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000902 1784124427.388666 mapwrite 0x002859 thru 0x011e53 (0xf5fb bytes) 000904 1784124427.464064 mapwrite 0x034fde thru 0x03d893 (0x88b6 bytes) 000905 1784124427.491316 mapread 0x032b30 thru 0x0337b7 (0xc88 bytes) 000907 1784124427.495000 MIRROR_RESYNC 10 mirrors 000911 1784124427.966128 MIRROR_RESYNC 10 mirrors 000913 1784124428.167337 mapwrite 0x0332ce thru 0x03fe4b (0xcb7e bytes) 000916 1784124428.223745 trunc from 0x040000 to 0x02d512 (0x12aed bytes) 000917 1784124428.235102 read 0x018d8b thru 0x01c4d9 (0x374f bytes) 000919 1784124428.238147 mapread 0x010620 thru 0x01bbfa (0xb5db bytes) 000920 1784124428.241380 mapwrite 0x000804 thru 0x0074e9 (0x6ce6 bytes) 000923 1784124428.257867 mapread 0x01cfb1 thru 0x0223af (0x53ff bytes) 000925 1784124428.261991 read 0x018099 thru 0x01cbcd (0x4b35 bytes) 000926 1784124428.262630 mapread 0x019586 thru 0x028ebf (0xf93a bytes) 000929 1784124428.267228 mapwrite 0x00d521 thru 0x01b01f (0xdaff bytes) 000930 1784124428.282128 read 0x001479 thru 0x0052ad (0x3e35 bytes) 000931 1784124428.282738 read 0x00573a thru 0x00e5ab (0x8e72 bytes) 000933 1784124428.283615 read 0x025400 thru 0x02d512 (0x8113 bytes) 000934 1784124428.286024 MIRROR_RESYNC 10 mirrors 000935 1784124428.654938 trunc from 0x02d513 to 0x020411 (0xd101 bytes) 000937 1784124428.682383 mapwrite 0x01e422 thru 0x029256 (0xae35 bytes) 000938 1784124428.713971 read 0x01744a thru 0x01f894 (0x844b bytes) 000939 1784124428.716144 mapwrite 0x012dab thru 0x01915d (0x63b3 bytes) 000941 1784124428.732524 trunc from 0x029257 to 0x03dab5 (0x1485f bytes) 000942 1784124428.737560 mapwrite 0x036716 thru 0x03f143 (0x8a2e bytes) 000943 1784124428.766453 trunc from 0x03f144 to 0x010e8b (0x2e2b8 bytes) 000946 1784124428.772472 read 0x00d480 thru 0x010e8b (0x3a0c bytes) 000947 1784124428.774595 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x591:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10e8c): Mirror 5: 0x6b47a75 (stale) Mirror 6: 0x6b47a75 (stale) Mirror 7: 0x6b47a75 (stale) Mirror 14: 0x6b47a75 (stale) Mirror 16: 0x6b47a75 (stale) Mirror 17: 0x6b47a75 (stale) Mirror 20: 0x6b47a75 (stale) Mirror 22: 0x6b47a75 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10e8c): Mirror 5: 0x6b47a75 (stale) Mirror 6: 0x6b47a75 (stale) Mirror 7: 0x6b47a75 (stale) Mirror 14: 0x6b47a75 (stale) Mirror 16: 0x6b47a75 (stale) Mirror 17: 0x6b47a75 (stale) Mirror 20: 0x6b47a75 (stale) Mirror 22: 0x6b47a75 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10e8c: skipped 000948 1784124428.913857 mapread 0x00e2ed thru 0x010e8b (0x2b9f bytes) 000949 1784124428.920100 mapwrite 0x037a45 thru 0x03a5e6 (0x2ba2 bytes) 000956 1784124428.945166 read 0x02da40 thru 0x035ce9 (0x82aa bytes) 000961 1784124428.949968 write 0x00a389 thru 0x00dd03 (0x397b bytes) 000964 1784124428.952076 trunc from 0x03a5e7 to 0x027836 (0x12db0 bytes) 000968 1784124428.959224 trunc from 0x027837 to 0x00e530 (0x19306 bytes) 000969 1784124428.964480 trunc from 0x00e531 to 0x02bd56 (0x1d826 bytes) 000970 1784124428.969979 trunc from 0x02bd57 to 0x01acbb (0x1109b bytes) 000971 1784124428.975109 read 0x00bb59 thru 0x00cfc9 (0x1471 bytes) 000973 1784124428.976019 read 0x014837 thru 0x016d1a (0x24e4 bytes) 000975 1784124428.979922 write 0x027026 thru 0x02924b (0x2226 bytes) 000977 1784124428.981428 mapread 0x01e0e5 thru 0x02841a (0xa336 bytes) 000978 1784124428.985120 trunc from 0x02924c to 0x0033f8 (0x25e53 bytes) 000979 1784124428.990917 trunc from 0x0033f9 to 0x0140c2 (0x10cca bytes) 000980 1784124428.997590 MIRROR_RESYNC 9 mirrors 000988 1784124429.420036 read 0x00c11f thru 0x00c7ce (0x6b0 bytes) 000990 1784124429.422714 read 0x003497 thru 0x011f3c (0xeaa6 bytes) 000991 1784124429.428287 mapwrite 0x034515 thru 0x0376b7 (0x31a3 bytes) 000992 1784124429.468631 mapwrite 0x006623 thru 0x008aa7 (0x2485 bytes) 000996 1784124429.487749 write 0x03d64a thru 0x03ffff (0x29b6 bytes) 000998 1784124429.489301 read 0x0259f6 thru 0x0315bf (0xbbca bytes) 000999 1784124429.492117 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x591:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xa6af90f2 (stale) Mirror 6: 0xa6af90f2 (stale) Mirror 7: 0xa6af90f2 (stale) Mirror 14: 0xa6af90f2 (stale) Mirror 16: 0xa6af90f2 (stale) Mirror 17: 0xa6af90f2 (stale) Mirror 20: 0xa6af90f2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xa6af90f2 (stale) Mirror 6: 0xa6af90f2 (stale) Mirror 7: 0xa6af90f2 (stale) Mirror 14: 0xa6af90f2 (stale) Mirror 16: 0xa6af90f2 (stale) Mirror 17: 0xa6af90f2 (stale) Mirror 20: 0xa6af90f2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped All operations completed A-OK! PASS 70a (28s) == sanity-flr test 71: check mirror extend parent fid ==== 10:07:11 (1784124431) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 72: check append on FLR file ========== 10:07:12 (1784124432) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x876:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x85a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0674042 s, 31.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x876:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x877:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x85a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606601 s, 17.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x876:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x877:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x85b:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x85a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x876:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x877:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x85b:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x85a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x878:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (3s) == sanity-flr test 200a: stress test ===================== 10:07:15 (1784124435) Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre2 Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1626592 .. Extending file size to 7705504 .. Extending file size to 5282208 .. Extending file size to 9057184 .. Extending file size to 7925344 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8416192 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9091136 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9105920 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59536 59537 59539 59541 59543 done 192.168.202.107@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg207-client.virtnet /mnt/lustre2 (opts:) 192.168.202.107@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg207-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 10:08:38 (1784124518) Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre2 Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7786528 .. Extending file size to 3400736 .. Extending file size to 5819872 .. Extending file size to 6892448 .. Extending file size to 8348096 .. Extending file size to 8954240 .. Extending file size to 8103488 .. Extending file size to 8269344 .. Extending file size to 8797568 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9192992 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62269 62270 62271 62274 62275 done 192.168.202.107@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg207-client.virtnet /mnt/lustre2 (opts:) 192.168.202.107@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg207-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (73s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 10:09:51 (1784124591) Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3456 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3456 2204800 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0529459 s, 39.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.201176 s, 10.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x7cae7550 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.107@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg207-client.virtnet /mnt/lustre2 (opts:) PASS 200c (10s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 10:10:02 (1784124602) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.060748 s, 34.5 MB/s PASS 202 (4s) == sanity-flr test 203: mirror file preserve mirror ID === 10:10:06 (1784124606) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5e1:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x871:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x871:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x872:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x88d:0x0] } PASS 203 (4s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 10:10:10 (1784124610) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.108336 s, 19.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x873:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x88e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x88f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (5s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 10:10:15 (1784124615) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.143816 s, 14.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x874:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x875:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x890:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x874:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x875:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x890:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x891:0x0] } PASS 204b (4s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 10:10:19 (1784124619) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0576802 s, 36.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x876:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x892:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x876:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x892:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x893:0x0] } PASS 204c (27s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 10:10:46 (1784124646) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg207-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801162 s, 13.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x877:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x895:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x894:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x877:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x895:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x894:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x878:0x0] } PASS 204d (49s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 10:11:35 (1784124695) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg207-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x896:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x897:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.076052 s, 27.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x896:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x897:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x879:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x896:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x897:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x879:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (50s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 10:12:25 (1784124745) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg207-server mds-ost sync done. Creating new pool test_204f oleg207-server: Pool lustre.test_204f created Adding targets to pool oleg207-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg207-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.305569 s, 6.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x898:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x87a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x898:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x87a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg207-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg207-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg207-server: Pool lustre.test_204f destroyed PASS 204f (83s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 10:13:48 (1784124828) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 10:13:54 (1784124834) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 10:14:00 (1784124840) oleg207-server: Pool lustre.test_206 created Adding targets to pool oleg207-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg207-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg207-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg207-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg207-server: Pool lustre.test_206 destroyed PASS 206 (20s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 10:14:21 (1784124861) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531928 s, 19.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054696 s, 19.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xaeb2a1ae Mirror 3: 0xaeb2a1ae PASS 207 (8s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 10:14:29 (1784124869) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 10:14:31 (1784124871) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 10:14:34 (1784124874) Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0780869 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x18b0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xa50000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xa50000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xa50000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xcc0000 Read this (4 bytes): BBBB 192.168.202.107@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg207-client.virtnet /mnt/lustre2 (opts:) PASS 209a (28s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 10:15:02 (1784124902) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0104 s, 788 kB/s the very first read read with warm cache PASS 209b (4s) == sanity-flr test 210a: handle broken mirrored lovea ==== 10:15:07 (1784124907) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314419 s, 33.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x606 lmm_fid: [0x200000404:0x606:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x606 lmm_fid: [0x200000404:0x606:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x885:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835462 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-15 10:15:08.000000000 -0400 Modify: 2026-07-15 10:15:08.000000000 -0400 Change: 2026-07-15 10:15:09.000000000 -0400 Birth: 2026-07-15 10:15:08.000000000 -0400 PASS 210a (6s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 10:15:13 (1784124913) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 10:15:16 (1784124916) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.413053 s, 99.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0762015 s, 53.8 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 15 10:15 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x608:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 15 10:15 /mnt/lustre/f211.sanity-flr PASS 211 (6s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 10:15:22 (1784124922) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (5s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 10:15:27 (1784124927) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (6s) == sanity-flr test complete, duration 976 sec ============ 10:15:33 (1784124933) === sanity-flr: start cleanup 10:15:34 (1784124934) === === sanity-flr: finish cleanup 10:15:39 (1784124939) ===