-----============= acceptance-small: sanity-flr ============----- Sat Sep 5 00:20:55 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg129-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 00:21:15 (1788582075) === oleg129-client.virtnet: executing check_config_client /mnt/lustre oleg129-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg129-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8b588909c000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8b588909c000.idle_timeout=debug disable quota as required oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 00:21:55 (1788582115) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7641] == sanity-flr test 0a: lfs mirror create with -N option == 00:21:57 (1788582117) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 00:22:14 (1788582134) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 00:22:18 (1788582138) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 00:22:21 (1788582141) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 00:22:44 (1788582164) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 00:22:47 (1788582167) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 00:22:50 (1788582170) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8b588909c000.stats=clear osc.lustre-OST0001-osc-ffff8b588909c000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 00:22:58 (1788582178) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 00:23:10 (1788582190) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 00:23:18 (1788582198) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 00:23:21 (1788582201) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 00:23:28 (1788582208) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 00:23:37 (1788582217) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 00:23:44 (1788582224) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 00:23:47 (1788582227) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 00:23:54 (1788582234) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 00:24:05 (1788582245) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 00:24:13 (1788582253) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 1.37794 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219365 s, 4.8 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 00:24:24 (1788582264) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.350387 s, 26.9 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 00:24:33 (1788582273) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 00:24:45 (1788582285) Stopping /mnt/lustre-ost1 (opts:) on oleg129-server oleg129-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b588909c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b588909c000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg129-server: oleg129-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg129-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg129-server oleg129-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8b588909c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b588909c000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg129-server: oleg129-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg129-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (73s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 00:25:58 (1788582358) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.255428 s, 16.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg129-server oleg129-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b588909c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b588909c000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg129-server: oleg129-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg129-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (43s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 00:26:41 (1788582401) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg129-server oleg129-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b588909c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b588909c000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg129-server: oleg129-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg129-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg129-server oleg129-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8b588909c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b588909c000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg129-server: oleg129-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg129-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (74s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 00:27:56 (1788582476) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.54038 s, 11.5 MB/s -rw-r--r-- 1 root root 52428800 Sep 5 00:28 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.163s user 0m0.000s sys 0m0.185s Stopping /mnt/lustre-ost1 (opts:) on oleg129-server oleg129-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b588909c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b588909c000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.278s user 0m0.000s sys 0m0.172s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg129-server: oleg129-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg129-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg129-server oleg129-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8b588909c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b588909c000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.357s user 0m0.001s sys 0m0.202s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg129-server: oleg129-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg129-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg129-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b588909c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b588909c000.ost_server_uuid in FULL state after 0 sec PASS 33b (98s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 00:29:33 (1788582573) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 00:29:37 (1788582577) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 00:29:40 (1788582580) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 00:29:44 (1788582584) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 00:29:50 (1788582590) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.450062 s, 28.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 00:30:02 (1788582602) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248086 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310268 s, 33.8 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 00:30:09 (1788582609) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 00:30:18 (1788582618) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00114063 s, 898 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000766238 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00101481 s, 1.0 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00104961 s, 976 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 00:30:28 (1788582628) create test files with size 15741948 .. b4111122cd763499b200213462d95189 - b19d7e39ae4d9f19f98709b793876d90 - f97d447cd50dc9190be8f5ed6113610d - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg129-client.virtnet /mnt/lustre (opts:) Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre PASS 37 (28s) == sanity-flr test 38: resync ============================ 00:30:56 (1788582656) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 00:31:11 (1788582671) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 00:31:18 (1788582678) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18441 s, 5.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475191 s, 22.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 00:31:28 (1788582688) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.120965 s, 17.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.279691 s, 15.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.256803 s, 8.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 00:31:45 (1788582705) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 00:31:49 (1788582709) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 00:31:52 (1788582712) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068245 s, 15.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 00:32:00 (1788582720) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 00:32:04 (1788582724) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 00:32:14 (1788582734) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.473544 s, 22.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg129-client.virtnet /mnt/lustre (opts:) Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 00:32:27 (1788582747) llite.lustre-ffff8b58a0fc4800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.366343 s, 28.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg129-client.virtnet /mnt/lustre (opts:) Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8b5890cdd800.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 00:32:43 (1788582763) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (47s) == sanity-flr test 46: Verify setstripe --copy option ==== 00:33:30 (1788582810) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (37s) == sanity-flr test 47: Verify mirror obj alloc =========== 00:34:07 (1788582847) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 00:34:10 (1788582850) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.251496 s, 12.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.184851 s, 17.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xfb11e735 (nosync,stale) Mirror 2: 0xfb11e735 (nosync,stale) ** verify mirror 2 doesn't change original checksum: f79e8fbf84af99a2767245c1c3619197 - mirror 1 checksum: 6edb725fce601510dc7c71f1e5f3bcce - mirror 2 checksum: f79e8fbf84af99a2767245c1c3619197 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-05T00:34:13-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-05T00:34:13-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: f79e8fbf84af99a2767245c1c3619197 - mirror 1 checksum: 6edb725fce601510dc7c71f1e5f3bcce - mirror 2 checksum: 6edb725fce601510dc7c71f1e5f3bcce - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-05T00:34:13-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-05T00:34:13-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 00:34:24 (1788582864) Failing mds1 on oleg129-server Stopping /mnt/lustre-mds1 (opts:) on oleg129-server 00:34:30 (1788582870) shut down facet: mds1 facet_host: oleg129-server facet_failover_host: oleg129-server Failover mds1 to oleg129-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-MDT0000 00:35:04 (1788582904) targets are mounted 00:35:04 (1788582904) facet_failover done oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205854 s, 50.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xee:0x0] } PASS 50A (57s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 00:35:21 (1788582921) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 00:35:24 (1788582924) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 00:35:28 (1788582928) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00460827 s, 3.6 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 00:35:35 (1788582935) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 00:35:39 (1788582939) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00928208 s, 441 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0869372 s, 47.1 kB/s PASS 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 00:35:49 (1788582949) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 00:36:11 (1788582971) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 00:36:38 (1788582998) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 00:37:02 (1788583022) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 00:37:26 (1788583046) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 00:37:33 (1788583053) mirror create pid 54346 mirror split pid 54347 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54346 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54347 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 00:38:39 (1788583119) Seed set to 3121 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1788583121.720356 write 0x003a00 thru 0x00b531 (0x7b32 bytes) 000003 1788583121.727987 punch from 0x0022ed to 0x005e47 (0x3b5b bytes) 000004 1788583121.748822 read 0x0061dc thru 0x00b531 (0x5356 bytes) 000006 1788583121.750634 read 0x001663 thru 0x003bb9 (0x2557 bytes) 000007 1788583121.772686 mapwrite 0x02322d thru 0x03088c (0xd660 bytes) 000008 1788583121.886083 write 0x018f02 thru 0x0220fa (0x91f9 bytes) 000009 1788583121.888247 write 0x03d956 thru 0x03ffff (0x26aa bytes) 000010 1788583121.890157 punch from 0x030ace to 0x03663f (0x5b72 bytes) 000011 1788583121.900959 write 0x033a61 thru 0x03ffff (0xc59f bytes) 000012 1788583121.912729 write 0x001b4b thru 0x003bf6 (0x20ac bytes) 000013 1788583121.914217 mapwrite 0x0140f6 thru 0x02127a (0xd185 bytes) 000015 1788583121.960161 punch from 0x01e88a to 0x02d951 (0xf0c8 bytes) 000017 1788583121.964217 mapread 0x007eaa thru 0x013500 (0xb657 bytes) 000018 1788583121.965047 mapread 0x0112af thru 0x0126e5 (0x1437 bytes) 000020 1788583121.966150 mapwrite 0x01bd2c thru 0x022e52 (0x7127 bytes) 000021 1788583122.025145 mapwrite 0x033386 thru 0x03ffff (0xcc7a bytes) truncating to largest ever: 0x2de14 000023 1788583122.055233 trunc from 0x040000 to 0x02de13 (0x121ec bytes) 000024 1788583122.069938 mapwrite 0x02fe40 thru 0x030dd7 (0xf98 bytes) 000025 1788583122.129853 write 0x0137f4 thru 0x013c24 (0x431 bytes) 000027 1788583122.137935 MIRROR_EXTEND to 2 mirrors 000028 1788583122.586442 write 0x0038ce thru 0x00a610 (0x6d43 bytes) 000029 1788583122.673946 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) 000030 1788583122.836636 write 0x0147a0 thru 0x02432e (0xfb8f bytes) 000033 1788583122.852200 mapwrite 0x03d0b0 thru 0x03ffff (0x2f50 bytes) 000034 1788583122.954137 read 0x03488b thru 0x03ffff (0xb775 bytes) 000037 1788583122.963940 trunc from 0x040000 to 0x00ef3f (0x310c0 bytes) 000038 1788583122.977860 write 0x01dd9c thru 0x02c2fb (0xe560 bytes) 000039 1788583122.979466 write 0x013c90 thru 0x01e5a2 (0xa913 bytes) truncating to largest ever: 0x31d8e 000041 1788583122.986672 trunc from 0x02c2fc to 0x031d8d (0x5a92 bytes) 000042 1788583123.006111 read 0x028e2a thru 0x02d723 (0x48fa bytes) 000044 1788583123.015603 mapwrite 0x01f330 thru 0x021ddc (0x2aad bytes) truncating to largest ever: 0x33d7c 000046 1788583123.055883 trunc from 0x031d8e to 0x033d7b (0x1fee bytes) 000047 1788583123.069098 mapwrite 0x013e1b thru 0x01aa64 (0x6c4a bytes) 000048 1788583123.110122 punch from 0x01f201 to 0x021fa5 (0x2da5 bytes) 000049 1788583123.114925 mapwrite 0x035b70 thru 0x03ec90 (0x9121 bytes) 000050 1788583123.182937 write 0x03d922 thru 0x03ffff (0x26de bytes) 000051 1788583123.184329 trunc from 0x040000 to 0x0231a2 (0x1ce5d bytes) 000052 1788583123.196940 trunc from 0x0231a3 to 0x00ff1c (0x13286 bytes) 000053 1788583123.213035 trunc from 0x00ff1d to 0x00f5e8 (0x934 bytes) 000054 1788583123.225279 mapwrite 0x034f4b thru 0x03ffff (0xb0b5 bytes) 000055 1788583123.352482 trunc from 0x040000 to 0x00de74 (0x3218b bytes) 000058 1788583123.370133 write 0x02e4ad thru 0x02f9e0 (0x1534 bytes) 000059 1788583123.371980 punch from 0x00df3f to 0x015d77 (0x7e39 bytes) 000060 1788583123.380062 trunc from 0x02f9e1 to 0x02a01b (0x59c5 bytes) 000064 1788583123.404156 mapread 0x022acc thru 0x025a04 (0x2f39 bytes) 000065 1788583123.423308 mapwrite 0x0147f8 thru 0x018d60 (0x4569 bytes) 000066 1788583123.494110 punch from 0x0268eb to 0x02a01b (0x3731 bytes) 000067 1788583123.498743 mapread 0x0243da thru 0x02a01b (0x5c42 bytes) 000068 1788583123.523801 write 0x03130f thru 0x035a93 (0x4785 bytes) 000070 1788583123.527081 mapread 0x00ee2d thru 0x019af8 (0xaccc bytes) 000071 1788583123.568929 write 0x002241 thru 0x00cf8b (0xad4b bytes) 000072 1788583123.573891 punch from 0x0246c1 to 0x02ece9 (0xa629 bytes) 000075 1788583123.584282 read 0x005c9c thru 0x00f844 (0x9ba9 bytes) 000076 1788583123.597453 mapread 0x025237 thru 0x029467 (0x4231 bytes) truncating to largest ever: 0x3b4b3 000078 1788583123.599034 trunc from 0x035a94 to 0x03b4b2 (0x5a1f bytes) 000081 1788583123.615039 trunc from 0x03b4b3 to 0x00c52e (0x2ef84 bytes) 000082 1788583123.631687 mapread 0x0023c2 thru 0x00c52e (0xa16d bytes) 000086 1788583123.635080 punch from 0x004d4b to 0x00c52e (0x77e4 bytes) 000087 1788583123.638643 mapread 0x001f16 thru 0x00c52e (0xa619 bytes) 000089 1788583123.671176 punch from 0x005d7b to 0x00c52e (0x67b4 bytes) 000090 1788583123.675459 punch from 0x0056ed to 0x00c52e (0x6e42 bytes) 000092 1788583123.680317 mapwrite 0x022657 thru 0x024c5c (0x2606 bytes) 000098 1788583123.740583 write 0x00200a thru 0x00e6d4 (0xc6cb bytes) 000099 1788583123.745881 trunc from 0x024c5d to 0x01ea3d (0x621f bytes) 000100 1788583123.761669 trunc from 0x01ea3e to 0x02514f (0x6712 bytes) 000101 1788583123.777909 mapwrite 0x034a0f thru 0x03ffff (0xb5f1 bytes) 000102 1788583123.849865 read 0x014705 thru 0x021042 (0xc93e bytes) 000103 1788583123.861638 write 0x034a03 thru 0x03dc40 (0x923e bytes) 000105 1788583123.863441 read 0x000fcd thru 0x003625 (0x2659 bytes) 000107 1788583123.865001 read 0x006f13 thru 0x00d31c (0x640a bytes) 000108 1788583123.865932 mapwrite 0x023eb5 thru 0x030447 (0xc593 bytes) 000109 1788583123.901481 write 0x0047e0 thru 0x012939 (0xe15a bytes) 000110 1788583123.902798 read 0x03bcda thru 0x03f882 (0x3ba9 bytes) 000112 1788583123.903791 mapwrite 0x0036e0 thru 0x008496 (0x4db7 bytes) 000113 1788583123.943657 read 0x006e7e thru 0x00db9d (0x6d20 bytes) 000114 1788583123.948633 MIRROR_EXTEND to 2 mirrors 000115 1788583124.497483 MIRROR_RESYNC 2 mirrors 000116 1788583124.741127 write 0x022cf4 thru 0x024dbe (0x20cb bytes) 000117 1788583124.813009 trunc from 0x040000 to 0x0021e7 (0x3de18 bytes) truncating to largest ever: 0x3d214 000118 1788583124.833534 trunc from 0x0021e8 to 0x03d213 (0x3b02c bytes) 000119 1788583124.845479 trunc from 0x03d214 to 0x017d31 (0x254e2 bytes) 000120 1788583124.857540 mapwrite 0x02191e thru 0x0241a5 (0x2888 bytes) 000122 1788583124.942239 write 0x024cf8 thru 0x0270f1 (0x23fa bytes) 000123 1788583124.943950 punch from 0x01097a to 0x0127b1 (0x1e38 bytes) 000126 1788583124.955192 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) 000129 1788583125.063995 read 0x020001 thru 0x0270f1 (0x70f1 bytes) 000130 1788583125.072858 trunc from 0x0270f2 to 0x018783 (0xe96e bytes) 000132 1788583125.089460 read 0x015f92 thru 0x018783 (0x27f2 bytes) 000133 1788583125.099432 mapread 0x00f731 thru 0x018783 (0x9053 bytes) 000134 1788583125.100336 read 0x010761 thru 0x018783 (0x8023 bytes) 000135 1788583125.100881 punch from 0x0164d9 to 0x018783 (0x22ab bytes) 000136 1788583125.104653 read 0x014616 thru 0x018783 (0x416e bytes) 000138 1788583125.108523 mapread 0x005448 thru 0x014ef7 (0xfab0 bytes) 000140 1788583125.109474 mapwrite 0x03c433 thru 0x03c983 (0x551 bytes) 000141 1788583125.157482 trunc from 0x03c984 to 0x02b6a5 (0x112de bytes) 000143 1788583125.175329 mapwrite 0x029812 thru 0x03912e (0xf91d bytes) 000146 1788583125.287866 MIRROR_EXTEND to 2 mirrors 000147 1788583125.751467 trunc from 0x03912f to 0x029472 (0xfcbc bytes) 000149 1788583125.837983 write 0x02145b thru 0x026c24 (0x57ca bytes) 000150 1788583125.849580 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) 000152 1788583125.976925 mapwrite 0x00a63f thru 0x015792 (0xb154 bytes) 000153 1788583126.041452 trunc from 0x029473 to 0x02609f (0x33d3 bytes) 000154 1788583126.058869 write 0x0263c0 thru 0x034228 (0xde69 bytes) 000156 1788583126.067290 mapread 0x009483 thru 0x014110 (0xac8e bytes) 000157 1788583126.068538 trunc from 0x034229 to 0x037fa8 (0x3d80 bytes) 000160 1788583126.085669 read 0x033aeb thru 0x037fa8 (0x44be bytes) 000161 1788583126.091320 read 0x015650 thru 0x01ca61 (0x7412 bytes) 000163 1788583126.092548 mapread 0x0082be thru 0x00dc75 (0x59b8 bytes) 000164 1788583126.094073 mapread 0x007132 thru 0x01543c (0xe30b bytes) 000165 1788583126.095180 write 0x00f820 thru 0x01a519 (0xacfa bytes) 000168 1788583126.108820 trunc from 0x037fa9 to 0x024b3e (0x1346a bytes) 000170 1788583126.127879 write 0x001893 thru 0x00e328 (0xca96 bytes) 000171 1788583126.139057 trunc from 0x024b3f to 0x004b71 (0x1ffcd bytes) 000174 1788583126.160556 trunc from 0x004b72 to 0x01402b (0xf4ba bytes) 000175 1788583126.178067 punch from 0x013b57 to 0x01402b (0x4d5 bytes) 000176 1788583126.188296 read 0x00c40c thru 0x010d8b (0x4980 bytes) 000177 1788583126.200354 trunc from 0x01402c to 0x00bbc7 (0x8464 bytes) 000179 1788583126.215219 punch from 0x007fea to 0x00bbc7 (0x3bde bytes) 000180 1788583126.219793 mapread 0x009a91 thru 0x00bbc7 (0x2137 bytes) 000181 1788583126.231996 mapwrite 0x03d963 thru 0x03ffff (0x269d bytes) 000182 1788583126.288432 write 0x01aa20 thru 0x028e3e (0xe41f bytes) 000183 1788583126.290893 punch from 0x0388cf to 0x03ffff (0x7731 bytes) 000184 1788583126.300684 punch from 0x00cb8f to 0x00d0b3 (0x525 bytes) 000185 1788583126.304037 trunc from 0x040000 to 0x01e8f7 (0x21708 bytes) 000186 1788583126.323007 trunc from 0x01e8f8 to 0x03952b (0x1ac34 bytes) 000187 1788583126.336007 read 0x024e41 thru 0x02afc1 (0x6181 bytes) 000188 1788583126.344898 trunc from 0x03952c to 0x036f9f (0x258c bytes) 000189 1788583126.363086 read 0x00bd35 thru 0x00c0c8 (0x394 bytes) 000190 1788583126.364064 read 0x0352d4 thru 0x036f9f (0x1ccc bytes) 000191 1788583126.369349 MIRROR_EXTEND to 2 mirrors 000194 1788583126.949963 read 0x03031b thru 0x030979 (0x65f bytes) 000195 1788583126.954355 read 0x03414e thru 0x036f9f (0x2e52 bytes) 000196 1788583126.977126 mapwrite 0x02cd53 thru 0x03ae85 (0xe133 bytes) 000197 1788583127.173371 mapread 0x038f9e thru 0x03ae85 (0x1ee8 bytes) 000198 1788583127.183908 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ae86): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ae86): Mirror 2: 0x52e6c2c5 (stale) CRC-32 checksum value for chunk [0, 0x3ae86): Mirror 1: 0xd62b2e19 Mirror 3: 0xd62b2e19 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ae86: skipped 000199 1788583127.728913 punch from 0x004540 to 0x013ee8 (0xf9a9 bytes) 000200 1788583127.790901 write 0x029ee4 thru 0x02edb4 (0x4ed1 bytes) 000201 1788583127.799940 trunc from 0x03ae86 to 0x009818 (0x3166d bytes) 000202 1788583127.837490 punch from 0x002c2a to 0x009818 (0x6bef bytes) 000204 1788583127.842528 punch from 0x003db2 to 0x009818 (0x5a67 bytes) 000205 1788583127.848022 trunc from 0x009819 to 0x01672f (0xcf17 bytes) 000206 1788583127.866127 trunc from 0x016730 to 0x03d134 (0x26a05 bytes) 000207 1788583127.884731 read 0x029bba thru 0x035ce8 (0xc12f bytes) 000208 1788583127.892036 mapread 0x023e30 thru 0x02536c (0x153d bytes) 000210 1788583127.907027 punch from 0x025720 to 0x02beda (0x67bb bytes) 000212 1788583127.912856 write 0x033fd2 thru 0x03abdf (0x6c0e bytes) 000213 1788583127.922781 mapread 0x02ddda thru 0x03cd66 (0xef8d bytes) 000214 1788583127.926596 read 0x000c46 thru 0x00beb7 (0xb272 bytes) 000217 1788583127.951060 punch from 0x03209f to 0x035f2f (0x3e91 bytes) 000219 1788583127.974941 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d135): Mirror 3: 0x716e4dd7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d135): Mirror 3: 0x716e4dd7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d135: skipped 000220 1788583128.349909 trunc from 0x03d135 to 0x019a32 (0x23702 bytes) 000222 1788583128.367456 trunc from 0x019a33 to 0x009f53 (0xfadf bytes) 000223 1788583128.405704 trunc from 0x009f54 to 0x02777d (0x1d82a bytes) 000224 1788583128.426138 read 0x012bed thru 0x01a001 (0x7415 bytes) 000225 1788583128.436813 MIRROR_RESYNC 2 mirrors 000229 1788583129.137789 MIRROR_EXTEND to 3 mirrors 000230 1788583129.764662 write 0x011a8e thru 0x0157b3 (0x3d26 bytes) 000232 1788583129.832505 write 0x0011b6 thru 0x004866 (0x36b1 bytes) 000234 1788583129.842905 mapwrite 0x02b13e thru 0x02b764 (0x627 bytes) 000237 1788583129.901995 mapread 0x011660 thru 0x015a1b (0x43bc bytes) 000238 1788583129.903900 mapwrite 0x014c18 thru 0x01770b (0x2af4 bytes) 000241 1788583129.957840 mapwrite 0x038365 thru 0x03a368 (0x2004 bytes) 000243 1788583130.029488 mapread 0x0218cf thru 0x02e979 (0xd0ab bytes) 000244 1788583130.048626 read 0x0154af thru 0x01dbb8 (0x870a bytes) 000245 1788583130.049214 write 0x012d73 thru 0x015264 (0x24f2 bytes) 000246 1788583130.052751 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a369): Mirror 3: 0xe64735cb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a369): Mirror 3: 0xe64735cb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a369: skipped 000247 1788583130.359364 punch from 0x019dd6 to 0x020128 (0x6353 bytes) 000248 1788583130.368501 mapread 0x007cc5 thru 0x014df1 (0xd12d bytes) 000249 1788583130.402549 read 0x02e0eb thru 0x033140 (0x5056 bytes) 000251 1788583130.417325 mapread 0x02617c thru 0x02d462 (0x72e7 bytes) 000252 1788583130.425593 punch from 0x01df84 to 0x027cdb (0x9d58 bytes) 000253 1788583130.431418 read 0x034ef8 thru 0x03a368 (0x5471 bytes) 000257 1788583130.442997 read 0x031206 thru 0x035595 (0x4390 bytes) 000258 1788583130.443614 trunc from 0x03a369 to 0x0319d3 (0x8995 bytes) 000259 1788583130.459068 mapwrite 0x00a69a thru 0x00d597 (0x2efe bytes) 000260 1788583130.497515 MIRROR_RESYNC 2 mirrors 000261 1788583131.029482 read 0x011a9f thru 0x0160b5 (0x4617 bytes) 000263 1788583131.073799 write 0x0183a4 thru 0x019110 (0xd6d bytes) 000268 1788583131.183010 mapwrite 0x01b1f9 thru 0x0220ed (0x6ef5 bytes) 000272 1788583131.237939 mapwrite 0x03d3f7 thru 0x03ffff (0x2c09 bytes) 000273 1788583131.316193 punch from 0x027ad5 to 0x02b149 (0x3675 bytes) 000274 1788583131.320799 punch from 0x008ecb to 0x015ea5 (0xcfdb bytes) 000275 1788583131.324583 punch from 0x0084aa to 0x00a7f3 (0x234a bytes) 000276 1788583131.328606 mapread 0x01bdd5 thru 0x01de49 (0x2075 bytes) 000277 1788583131.330087 mapread 0x01f017 thru 0x02299c (0x3986 bytes) 000278 1788583131.331039 mapread 0x000784 thru 0x0030f5 (0x2972 bytes) 000279 1788583131.343552 MIRROR_RESYNC 2 mirrors 000280 1788583132.078985 punch from 0x0156d1 to 0x023b91 (0xe4c1 bytes) 000281 1788583132.126987 trunc from 0x040000 to 0x030ff7 (0xf008 bytes) 000283 1788583132.174499 mapwrite 0x02f988 thru 0x03589c (0x5f15 bytes) 000284 1788583132.249175 write 0x0255fc thru 0x02e5df (0x8fe4 bytes) 000285 1788583132.254456 trunc from 0x03589d to 0x02615c (0xf740 bytes) 000286 1788583132.274124 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) 000287 1788583132.395982 punch from 0x00f3b4 to 0x012b19 (0x3766 bytes) 000288 1788583132.399523 trunc from 0x02615d to 0x000243 (0x25f19 bytes) 000289 1788583132.410911 punch from 0x00021e to 0x000243 (0x26 bytes) 000290 1788583132.429540 MIRROR_EXTEND to 2 mirrors 000291 1788583133.076639 mapread 0x00023b thru 0x000243 (0x9 bytes) 000292 1788583133.083562 read 0x00008b thru 0x000243 (0x1b9 bytes) 000293 1788583133.083936 punch from 0x00004f to 0x000243 (0x1f5 bytes) 000294 1788583133.168523 mapread 0x000042 thru 0x000243 (0x202 bytes) 000295 1788583133.174043 read 0x00009c thru 0x000243 (0x1a8 bytes) 000296 1788583133.174693 punch from 0x000120 to 0x000243 (0x124 bytes) 000297 1788583133.187051 read 0x0001c9 thru 0x000243 (0x7b bytes) 000298 1788583133.200136 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) 000299 1788583133.378784 trunc from 0x000244 to 0x027fde (0x27d9b bytes) 000301 1788583133.417569 write 0x0018c6 thru 0x00255d (0xc98 bytes) 000302 1788583133.446767 trunc from 0x027fdf to 0x016f94 (0x1104a bytes) 000304 1788583133.477675 mapread 0x00261a thru 0x0037be (0x11a5 bytes) 000305 1788583133.489564 write 0x036b08 thru 0x03a8c1 (0x3dba bytes) 000307 1788583133.495527 mapread 0x0263bb thru 0x02ff2a (0x9b70 bytes) 000308 1788583133.517365 MIRROR_EXTEND to 2 mirrors 000309 1788583134.368965 trunc from 0x03a8c2 to 0x023cbc (0x16c05 bytes) 000310 1788583134.450079 trunc from 0x023cbd to 0x00cb3e (0x1717e bytes) 000311 1788583134.478906 mapread 0x004d2f thru 0x00cb3e (0x7e10 bytes) 000313 1788583134.494139 punch from 0x0017fe to 0x00c19a (0xa99d bytes) 000316 1788583134.500004 trunc from 0x00cb3f to 0x0177af (0xac71 bytes) 000317 1788583134.513919 mapread 0x00b5ae thru 0x015980 (0xa3d3 bytes) 000318 1788583134.532504 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) 000321 1788583134.646820 read 0x00e1f4 thru 0x0177af (0x95bc bytes) 000322 1788583134.662126 punch from 0x00b742 to 0x00ddf3 (0x26b2 bytes) 000325 1788583134.668291 mapread 0x011831 thru 0x0177af (0x5f7f bytes) 000326 1788583134.669521 mapwrite 0x021bab thru 0x02c01c (0xa472 bytes) 000327 1788583134.751234 mapwrite 0x02e567 thru 0x03b002 (0xca9c bytes) 000328 1788583134.821492 MIRROR_EXTEND to 2 mirrors 000329 1788583135.419798 mapread 0x025e24 thru 0x03573e (0xf91b bytes) 000330 1788583135.446675 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) 000331 1788583135.553439 write 0x01c4e5 thru 0x01f1ee (0x2d0a bytes) 000332 1788583135.588684 mapwrite 0x002de3 thru 0x009dea (0x7008 bytes) 000333 1788583135.696811 trunc from 0x03b003 to 0x000f8b (0x3a077 bytes) 000335 1788583135.727082 mapread 0x0001d1 thru 0x000f8b (0xdbb bytes) 000336 1788583135.734215 read 0x000259 thru 0x000f8b (0xd33 bytes) 000337 1788583135.735502 mapwrite 0x03a70e thru 0x03aa71 (0x364 bytes) 000338 1788583135.889959 write 0x01180d thru 0x0155fd (0x3df1 bytes) 000339 1788583135.906018 mapwrite 0x0275d2 thru 0x027989 (0x3b8 bytes) 000340 1788583136.018714 trunc from 0x03aa72 to 0x02bafd (0xef74 bytes) 000341 1788583136.047345 trunc from 0x02bafe to 0x0087a6 (0x23357 bytes) 000343 1788583136.076211 trunc from 0x0087a7 to 0x005a51 (0x2d55 bytes) 000344 1788583136.103632 mapwrite 0x0332ad thru 0x03d1f9 (0x9f4d bytes) 000347 1788583136.228704 trunc from 0x03d1fa to 0x0030f5 (0x3a104 bytes) 000349 1788583136.251599 trunc from 0x0030f6 to 0x01d135 (0x1a040 bytes) 000351 1788583136.279481 mapwrite 0x02e191 thru 0x03b6c2 (0xd532 bytes) 000352 1788583136.404959 MIRROR_EXTEND to 2 mirrors 000354 1788583137.154419 write 0x033be2 thru 0x03be8f (0x82ae bytes) 000357 1788583137.260144 mapwrite 0x02e50f thru 0x034635 (0x6127 bytes) 000358 1788583137.344315 read 0x02f37a thru 0x0332a4 (0x3f2b bytes) 000359 1788583137.344855 write 0x03d828 thru 0x03ffff (0x27d8 bytes) 000360 1788583137.345690 mapread 0x03ccb5 thru 0x03ce0d (0x159 bytes) 000362 1788583137.348742 write 0x028e75 thru 0x036e4b (0xdfd7 bytes) 000363 1788583137.353179 mapread 0x033e76 thru 0x03c8b3 (0x8a3e bytes) 000365 1788583137.354984 mapread 0x022ae0 thru 0x029c0f (0x7130 bytes) 000366 1788583137.366697 mapread 0x02be5a thru 0x031908 (0x5aaf bytes) 000368 1788583137.371940 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x68b00f0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9097925e Mirror 3: 0x9097925e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000369 1788583138.202828 punch from 0x015f70 to 0x01884d (0x28de bytes) 000370 1788583138.281000 mapread 0x03b9da thru 0x03ffff (0x4626 bytes) 000372 1788583138.303119 mapread 0x023b4f thru 0x029180 (0x5632 bytes) 000374 1788583138.312379 trunc from 0x040000 to 0x00209a (0x3df65 bytes) 000375 1788583138.356767 mapread 0x000eb8 thru 0x00209a (0x11e3 bytes) 000376 1788583138.362835 read 0x000d12 thru 0x00209a (0x1389 bytes) 000377 1788583138.363830 read 0x0006c5 thru 0x00209a (0x19d6 bytes) 000378 1788583138.364926 punch from 0x000a09 to 0x00209a (0x1692 bytes) 000379 1788583138.373660 mapwrite 0x02de65 thru 0x0352b7 (0x7453 bytes) 000380 1788583138.538929 punch from 0x028128 to 0x0331df (0xb0b8 bytes) 000382 1788583138.546878 read 0x01fb58 thru 0x021818 (0x1cc1 bytes) 000383 1788583138.547920 read 0x00a85f thru 0x01059a (0x5d3c bytes) 000384 1788583138.549132 mapwrite 0x00a875 thru 0x00ecc9 (0x4455 bytes) 000385 1788583138.591663 mapwrite 0x01c075 thru 0x02777a (0xb706 bytes) 000387 1788583138.639769 trunc from 0x0352b8 to 0x010fe5 (0x242d2 bytes) 000388 1788583138.660356 MIRROR_RESYNC 3 mirrors 000390 1788583139.329605 mapread 0x00f172 thru 0x010fe5 (0x1e74 bytes) 000391 1788583139.347863 MIRROR_EXTEND to 4 mirrors 000392 1788583140.072202 trunc from 0x010fe6 to 0x0167c9 (0x57e4 bytes) 000393 1788583140.202551 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x167ca): Mirror 3: 0x81d7a55a (stale) Mirror 4: 0x81d7a55a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x167ca): Mirror 3: 0x81d7a55a (stale) Mirror 4: 0x81d7a55a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x167ca: skipped 000394 1788583140.708046 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x167ca): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x167ca): Mirror 3: 0x81d7a55a (stale) Mirror 4: 0x81d7a55a (stale) CRC-32 checksum value for chunk [0, 0x167ca): Mirror 1: 0x81d7a55a Mirror 5: 0x81d7a55a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x167ca: skipped 000395 1788583141.709733 mapwrite 0x011baa thru 0x019b13 (0x7f6a bytes) 000396 1788583141.934798 punch from 0x011fea to 0x019b13 (0x7b2a bytes) 000402 1788583141.959133 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x19b14): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19b14): Mirror 3: 0xe23bc6fa (stale) Mirror 4: 0xe23bc6fa (stale) Mirror 5: 0xe23bc6fa (stale) CRC-32 checksum value for chunk [0, 0x19b14): Mirror 1: 0xed4d446e Mirror 6: 0xed4d446e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19b14: skipped 000403 1788583143.084784 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x19b14): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19b14): Mirror 3: 0xe23bc6fa (stale) Mirror 4: 0xe23bc6fa (stale) Mirror 5: 0xe23bc6fa (stale) CRC-32 checksum value for chunk [0, 0x19b14): Mirror 1: 0xed4d446e Mirror 6: 0xed4d446e Mirror 7: 0xed4d446e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19b14: skipped 000404 1788583144.127477 mapread 0x016c72 thru 0x019b13 (0x2ea2 bytes) 000405 1788583144.217703 write 0x02ebf8 thru 0x03c36a (0xd773 bytes) 000406 1788583144.370840 trunc from 0x03c36b to 0x02e99c (0xd9ce bytes) 000407 1788583144.401370 punch from 0x02a7b2 to 0x02e99c (0x41eb bytes) 000410 1788583144.427505 mapread 0x023945 thru 0x024ad2 (0x118e bytes) 000412 1788583144.448536 mapread 0x0279f3 thru 0x02e99c (0x6faa bytes) 000413 1788583144.461719 read 0x019cbd thru 0x0206e4 (0x6a28 bytes) 000414 1788583144.471255 read 0x00647c thru 0x0143d6 (0xdf5b bytes) 000415 1788583144.516985 MIRROR_RESYNC 6 mirrors 000416 1788583145.608365 mapread 0x01c5e9 thru 0x02044c (0x3e64 bytes) 000417 1788583145.627084 read 0x00151c thru 0x00a7e7 (0x92cc bytes) 000421 1788583145.636442 mapwrite 0x001fc1 thru 0x00dbf7 (0xbc37 bytes) 000423 1788583145.755936 mapwrite 0x019c3f thru 0x021067 (0x7429 bytes) 000425 1788583145.799223 trunc from 0x02e99d to 0x02aa49 (0x3f53 bytes) 000426 1788583145.823056 mapread 0x001535 thru 0x01006b (0xeb37 bytes) 000427 1788583145.830531 punch from 0x025f48 to 0x02706b (0x1124 bytes) 000428 1788583145.838668 mapwrite 0x022959 thru 0x024515 (0x1bbd bytes) 000429 1788583145.900847 write 0x032dfd thru 0x035dcd (0x2fd1 bytes) 000430 1788583145.905487 punch from 0x02da98 to 0x02e598 (0xb01 bytes) 000432 1788583145.923296 write 0x01c5e8 thru 0x01d333 (0xd4c bytes) 000433 1788583145.924482 punch from 0x01e9e4 to 0x02e2e8 (0xf905 bytes) 000434 1788583145.933638 punch from 0x013e2a to 0x015e90 (0x2067 bytes) 000437 1788583145.938361 punch from 0x02f215 to 0x0329f1 (0x37dd bytes) 000439 1788583145.944582 write 0x01ef43 thru 0x027e48 (0x8f06 bytes) 000440 1788583145.965766 punch from 0x02ddd8 to 0x03465f (0x6888 bytes) 000441 1788583145.976876 mapread 0x032144 thru 0x03473f (0x25fc bytes) 000443 1788583145.990105 mapwrite 0x02a683 thru 0x037902 (0xd280 bytes) 000444 1788583146.067150 write 0x013bb7 thru 0x0221b6 (0xe600 bytes) 000445 1788583146.071775 mapwrite 0x02e840 thru 0x030bc2 (0x2383 bytes) 000446 1788583146.123911 trunc from 0x037903 to 0x032e9b (0x4a67 bytes) 000448 1788583146.145930 trunc from 0x032e9c to 0x033b9f (0xd04 bytes) 000450 1788583146.175307 mapread 0x0053a8 thru 0x014d08 (0xf961 bytes) 000451 1788583146.177063 punch from 0x024fb9 to 0x02675e (0x17a6 bytes) 000452 1788583146.186605 punch from 0x030bae to 0x033b9f (0x2ff2 bytes) 000454 1788583146.196992 read 0x00ccf6 thru 0x01a4d5 (0xd7e0 bytes) 000455 1788583146.198132 trunc from 0x033ba0 to 0x02f3cb (0x47d4 bytes) 000456 1788583146.222148 mapread 0x02329a thru 0x025fe7 (0x2d4e bytes) 000457 1788583146.236817 write 0x000942 thru 0x0078bc (0x6f7b bytes) 000458 1788583146.238260 write 0x035387 thru 0x03ffff (0xac79 bytes) 000459 1788583146.242978 mapwrite 0x017965 thru 0x0264f6 (0xeb92 bytes) 000460 1788583146.299056 write 0x009d84 thru 0x0161db (0xc458 bytes) 000461 1788583146.305350 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf5c2af61 (stale) Mirror 4: 0xf5c2af61 (stale) Mirror 5: 0xf5c2af61 (stale) Mirror 6: 0xf5c2af61 (stale) Mirror 7: 0xf5c2af61 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x561dac2e Mirror 8: 0x561dac2e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000462 1788583147.951109 mapwrite 0x027f9c thru 0x02f1f3 (0x7258 bytes) 000463 1788583148.195645 mapread 0x036e1e thru 0x03f79a (0x897d bytes) 000464 1788583148.230901 write 0x022283 thru 0x02a2a9 (0x8027 bytes) 000467 1788583148.256988 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf5c2af61 (stale) Mirror 4: 0xf5c2af61 (stale) Mirror 5: 0xf5c2af61 (stale) Mirror 6: 0xf5c2af61 (stale) Mirror 7: 0xf5c2af61 (stale) Mirror 8: 0x561dac2e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x25d014e7 Mirror 9: 0x25d014e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000468 1788583149.337497 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf5c2af61 (stale) Mirror 4: 0xf5c2af61 (stale) Mirror 5: 0xf5c2af61 (stale) Mirror 6: 0xf5c2af61 (stale) Mirror 7: 0xf5c2af61 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf5c2af61 (stale) Mirror 4: 0xf5c2af61 (stale) Mirror 5: 0xf5c2af61 (stale) Mirror 6: 0xf5c2af61 (stale) Mirror 7: 0xf5c2af61 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x25d014e7 Mirror 9: 0x25d014e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000469 1788583149.934270 mapread 0x001c80 thru 0x00b375 (0x96f6 bytes) 000471 1788583149.984687 punch from 0x012547 to 0x022440 (0xfefa bytes) 000473 1788583150.100246 mapread 0x009578 thru 0x00bbf8 (0x2681 bytes) 000474 1788583150.119297 punch from 0x012cce to 0x019b7a (0x6ead bytes) 000475 1788583150.143335 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf5c2af61 (stale) Mirror 4: 0xf5c2af61 (stale) Mirror 5: 0xf5c2af61 (stale) Mirror 6: 0xf5c2af61 (stale) Mirror 7: 0xf5c2af61 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf5c2af61 (stale) Mirror 4: 0xf5c2af61 (stale) Mirror 5: 0xf5c2af61 (stale) Mirror 6: 0xf5c2af61 (stale) Mirror 7: 0xf5c2af61 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000476 1788583150.759505 punch from 0x02b325 to 0x03a34e (0xf02a bytes) 000478 1788583150.773855 mapwrite 0x03f836 thru 0x03ffff (0x7ca bytes) 000479 1788583150.869549 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf5c2af61 (stale) Mirror 4: 0xf5c2af61 (stale) Mirror 5: 0xf5c2af61 (stale) Mirror 6: 0xf5c2af61 (stale) Mirror 7: 0xf5c2af61 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x11d34bd Mirror 8: 0x11d34bd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000480 1788583151.976070 punch from 0x01b876 to 0x01b94c (0xd7 bytes) 000481 1788583152.079984 mapread 0x01fee4 thru 0x02bc39 (0xbd56 bytes) 000482 1788583152.106983 read 0x03c820 thru 0x03ffff (0x37e0 bytes) 000485 1788583152.114231 mapwrite 0x01dafc thru 0x02709b (0x95a0 bytes) 000486 1788583152.160158 mapread 0x03ca9e thru 0x03ffff (0x3562 bytes) 000487 1788583152.160969 mapwrite 0x0090d3 thru 0x0133eb (0xa319 bytes) 000489 1788583152.216818 trunc from 0x040000 to 0x01361b (0x2c9e4 bytes) 000490 1788583152.233774 mapwrite 0x010063 thru 0x01dfb8 (0xdf56 bytes) 000491 1788583152.302983 punch from 0x0074d1 to 0x00d0a2 (0x5bd2 bytes) 000493 1788583152.311827 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1dfb9): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dfb9): Mirror 3: 0x7cdc650b (stale) Mirror 4: 0x7cdc650b (stale) Mirror 5: 0x7cdc650b (stale) Mirror 6: 0x7cdc650b (stale) Mirror 7: 0x7cdc650b (stale) Mirror 8: 0xc3bf188e (stale) CRC-32 checksum value for chunk [0, 0x1dfb9): Mirror 1: 0x52972800 Mirror 9: 0x52972800 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dfb9: skipped 000494 1788583152.901642 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1dfb9): Mirror 3: 0x7cdc650b (stale) Mirror 5: 0x7cdc650b (stale) Mirror 6: 0x7cdc650b (stale) Mirror 7: 0x7cdc650b (stale) Mirror 8: 0xc3bf188e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dfb9): Mirror 3: 0x7cdc650b (stale) Mirror 5: 0x7cdc650b (stale) Mirror 6: 0x7cdc650b (stale) Mirror 7: 0x7cdc650b (stale) Mirror 8: 0xc3bf188e (stale) CRC-32 checksum value for chunk [0, 0x1dfb9): Mirror 1: 0x52972800 Mirror 9: 0x52972800 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dfb9: skipped 000495 1788583153.394993 trunc from 0x01dfb9 to 0x003c70 (0x1a348 bytes) 000496 1788583153.493543 mapread 0x0026ee thru 0x003c70 (0x1583 bytes) 000498 1788583153.507967 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c71): Mirror 3: 0xb978631e (stale) Mirror 5: 0xb978631e (stale) Mirror 6: 0xb978631e (stale) Mirror 7: 0xb978631e (stale) Mirror 9: 0x563bbda0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c71): Mirror 3: 0xb978631e (stale) Mirror 5: 0xb978631e (stale) Mirror 6: 0xb978631e (stale) Mirror 7: 0xb978631e (stale) Mirror 9: 0x563bbda0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c71: skipped 000500 1788583153.756894 mapwrite 0x026a89 thru 0x02fe91 (0x9409 bytes) 000501 1788583153.835036 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2fe92): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fe92): Mirror 3: 0x245f6930 (stale) Mirror 5: 0x245f6930 (stale) Mirror 6: 0x245f6930 (stale) Mirror 7: 0x245f6930 (stale) Mirror 9: 0x30c55c96 (stale) CRC-32 checksum value for chunk [0, 0x2fe92): Mirror 1: 0x936a7295 Mirror 10: 0x936a7295 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fe92: skipped 000502 1788583154.741021 read 0x001f3b thru 0x00c9b3 (0xaa79 bytes) 000505 1788583154.776168 mapread 0x00f26f thru 0x01281e (0x35b0 bytes) 000506 1788583154.805793 mapread 0x02ca44 thru 0x02fa45 (0x3002 bytes) 000508 1788583154.837739 trunc from 0x02fe92 to 0x005e42 (0x2a04f bytes) 000509 1788583154.930138 trunc from 0x005e43 to 0x02910a (0x232c8 bytes) 000510 1788583154.959030 read 0x014ef6 thru 0x015728 (0x833 bytes) 000511 1788583154.972488 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2910b): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2910b): Mirror 3: 0xb8f9676e (stale) Mirror 5: 0xb8f9676e (stale) Mirror 6: 0xb8f9676e (stale) Mirror 7: 0xb8f9676e (stale) Mirror 9: 0xe796295e (stale) Mirror 10: 0x62f4178f (stale) CRC-32 checksum value for chunk [0, 0x2910b): Mirror 1: 0xdfb90ae4 Mirror 11: 0xdfb90ae4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2910b: skipped 000512 1788583155.937117 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2910b): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2910b): Mirror 3: 0xb8f9676e (stale) Mirror 5: 0xb8f9676e (stale) Mirror 6: 0xb8f9676e (stale) Mirror 7: 0xb8f9676e (stale) Mirror 9: 0xe796295e (stale) Mirror 10: 0x62f4178f (stale) CRC-32 checksum value for chunk [0, 0x2910b): Mirror 1: 0xdfb90ae4 Mirror 11: 0xdfb90ae4 Mirror 12: 0xdfb90ae4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2910b: skipped 000513 1788583157.430949 mapread 0x026189 thru 0x02910a (0x2f82 bytes) 000514 1788583157.454315 mapread 0x002ab3 thru 0x010fbf (0xe50d bytes) 000516 1788583157.464145 read 0x020991 thru 0x021c9b (0x130b bytes) 000517 1788583157.473598 MIRROR_RESYNC 9 mirrors 000519 1788583158.923764 write 0x03fc36 thru 0x03ffff (0x3ca bytes) 000520 1788583159.034495 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa053b85 (stale) Mirror 5: 0xa053b85 (stale) Mirror 6: 0xa053b85 (stale) Mirror 9: 0xa053b85 (stale) Mirror 10: 0xa053b85 (stale) Mirror 11: 0xa053b85 (stale) Mirror 12: 0xa053b85 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa053b85 (stale) Mirror 5: 0xa053b85 (stale) Mirror 6: 0xa053b85 (stale) Mirror 9: 0xa053b85 (stale) Mirror 10: 0xa053b85 (stale) Mirror 11: 0xa053b85 (stale) Mirror 12: 0xa053b85 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000521 1788583159.559011 read 0x021c94 thru 0x023c1b (0x1f88 bytes) 000522 1788583159.563926 read 0x02d764 thru 0x03393e (0x61db bytes) 000526 1788583159.578219 read 0x0372ae thru 0x0388e8 (0x163b bytes) 000527 1788583159.578905 mapwrite 0x03d4ab thru 0x03ffff (0x2b55 bytes) 000528 1788583159.619861 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa053b85 (stale) Mirror 5: 0xa053b85 (stale) Mirror 9: 0xa053b85 (stale) Mirror 10: 0xa053b85 (stale) Mirror 11: 0xa053b85 (stale) Mirror 12: 0xa053b85 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa053b85 (stale) Mirror 5: 0xa053b85 (stale) Mirror 9: 0xa053b85 (stale) Mirror 10: 0xa053b85 (stale) Mirror 11: 0xa053b85 (stale) Mirror 12: 0xa053b85 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000529 1788583160.191947 mapread 0x0053e4 thru 0x01147d (0xc09a bytes) 000530 1788583160.266040 read 0x002e3e thru 0x00edc6 (0xbf89 bytes) 000531 1788583160.282598 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xa053b85 (stale) Mirror 9: 0xa053b85 (stale) Mirror 10: 0xa053b85 (stale) Mirror 11: 0xa053b85 (stale) Mirror 12: 0xa053b85 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xa053b85 (stale) Mirror 9: 0xa053b85 (stale) Mirror 10: 0xa053b85 (stale) Mirror 11: 0xa053b85 (stale) Mirror 12: 0xa053b85 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000533 1788583160.738853 write 0x03964c thru 0x03ffff (0x69b4 bytes) 000534 1788583160.745506 trunc from 0x040000 to 0x037ca4 (0x835b bytes) 000535 1788583160.763576 mapwrite 0x02c452 thru 0x03a54f (0xe0fe bytes) 000536 1788583160.846871 trunc from 0x03a550 to 0x0376f5 (0x2e5a bytes) 000537 1788583160.872327 read 0x0012bc thru 0x00c440 (0xb185 bytes) 000539 1788583160.886810 punch from 0x018f81 to 0x01f02b (0x60ab bytes) 000541 1788583160.895026 trunc from 0x0376f6 to 0x000313 (0x373e2 bytes) 000542 1788583160.915752 punch from 0x000267 to 0x000313 (0xad bytes) 000543 1788583160.923836 MIRROR_RESYNC 6 mirrors 000544 1788583161.686167 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) 000546 1788583161.983523 trunc from 0x000314 to 0x029c43 (0x29930 bytes) 000547 1788583162.068356 trunc from 0x029c44 to 0x027826 (0x241d bytes) 000549 1788583162.090138 write 0x024b55 thru 0x026947 (0x1df3 bytes) 000550 1788583162.097719 mapread 0x023348 thru 0x024d45 (0x19fe bytes) 000551 1788583162.102203 write 0x032dce thru 0x038bb9 (0x5dec bytes) 000554 1788583162.104311 punch from 0x0209ff to 0x02a94e (0x9f50 bytes) 000556 1788583162.127294 mapwrite 0x026e19 thru 0x02aea0 (0x4088 bytes) 000557 1788583162.202375 punch from 0x002055 to 0x00d627 (0xb5d3 bytes) 000559 1788583162.212071 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38bba): Mirror 9: 0x526440b3 (stale) Mirror 10: 0x526440b3 (stale) Mirror 12: 0x526440b3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38bba): Mirror 9: 0x526440b3 (stale) Mirror 10: 0x526440b3 (stale) Mirror 12: 0x526440b3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38bba: skipped 000560 1788583162.760118 mapread 0x021940 thru 0x02e374 (0xca35 bytes) 000561 1788583162.796740 read 0x01643b thru 0x016c4c (0x812 bytes) 000562 1788583162.802066 mapwrite 0x018e12 thru 0x026c54 (0xde43 bytes) 000564 1788583162.915828 read 0x000a5d thru 0x00cc50 (0xc1f4 bytes) 000565 1788583162.930184 punch from 0x003454 to 0x00c22f (0x8ddc bytes) 000568 1788583162.943215 punch from 0x01f2e9 to 0x02c636 (0xd34e bytes) 000569 1788583162.950756 mapwrite 0x02f552 thru 0x030263 (0xd12 bytes) 000570 1788583163.031333 trunc from 0x038bba to 0x0138ee (0x252cb bytes) 000572 1788583163.059239 mapwrite 0x032ef4 thru 0x033f45 (0x1052 bytes) 000573 1788583163.158071 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x33f46): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33f46): Mirror 9: 0x89636ba5 (stale) Mirror 10: 0x89636ba5 (stale) Mirror 12: 0x89636ba5 (stale) CRC-32 checksum value for chunk [0, 0x33f46): Mirror 1: 0x77ede072 Mirror 13: 0x77ede072 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33f46: skipped 000575 1788583164.180113 trunc from 0x033f46 to 0x017917 (0x1c62e bytes) 000577 1788583164.347136 read 0x010e87 thru 0x012ca6 (0x1e20 bytes) 000578 1788583164.354845 read 0x00351d thru 0x011b4d (0xe631 bytes) 000579 1788583164.371680 MIRROR_RESYNC 5 mirrors 000580 1788583166.007922 trunc from 0x017918 to 0x02c9c7 (0x150b0 bytes) 000581 1788583166.230350 mapread 0x00b397 thru 0x00bf57 (0xbc1 bytes) 000582 1788583166.259554 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2c9c8): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c9c8): Mirror 9: 0xe644f47a (stale) Mirror 10: 0xe644f47a (stale) Mirror 12: 0xe644f47a (stale) Mirror 13: 0xe644f47a (stale) CRC-32 checksum value for chunk [0, 0x2c9c8): Mirror 1: 0xe644f47a Mirror 14: 0xe644f47a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c9c8: skipped 000583 1788583167.723799 mapwrite 0x0095f8 thru 0x00d6e3 (0x40ec bytes) 000584 1788583168.131812 write 0x023809 thru 0x032ae7 (0xf2df bytes) 000585 1788583168.140271 mapread 0x0296c5 thru 0x032ae7 (0x9423 bytes) 000586 1788583168.141815 trunc from 0x032ae8 to 0x003d94 (0x2ed53 bytes) 000587 1788583168.243599 write 0x03d3ec thru 0x03ffff (0x2c14 bytes) 000588 1788583168.245767 punch from 0x02ad21 to 0x02d550 (0x2830 bytes) 000589 1788583168.264371 write 0x018dd3 thru 0x01de7a (0x50a8 bytes) 000590 1788583168.284139 punch from 0x021299 to 0x02f1b4 (0xdf1c bytes) 000591 1788583168.308818 mapread 0x030d2c thru 0x0373b3 (0x6688 bytes) 000592 1788583168.325924 mapread 0x0261e8 thru 0x035709 (0xf522 bytes) 000593 1788583168.349455 mapwrite 0x00504a thru 0x01078e (0xb745 bytes) 000595 1788583168.514872 read 0x0346e2 thru 0x0378da (0x31f9 bytes) 000596 1788583168.515467 punch from 0x02cb1b to 0x02e9c7 (0x1ead bytes) 000598 1788583168.528084 MIRROR_RESYNC 6 mirrors 000599 1788583170.592248 mapwrite 0x03d8fc thru 0x03ffff (0x2704 bytes) 000602 1788583170.880506 punch from 0x00eef2 to 0x00f7b4 (0x8c3 bytes) 000605 1788583170.892475 punch from 0x0397f8 to 0x03ffff (0x6808 bytes) 000606 1788583170.901885 write 0x0261fd thru 0x02f763 (0x9567 bytes) 000607 1788583170.936802 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xcbe94f1b (stale) Mirror 10: 0xcbe94f1b (stale) Mirror 12: 0xcbe94f1b (stale) Mirror 13: 0xcbe94f1b (stale) Mirror 14: 0xcbe94f1b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa14c0202 Mirror 15: 0xa14c0202 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000608 1788583171.774486 mapwrite 0x02c20d thru 0x031eda (0x5cce bytes) 000609 1788583171.921191 punch from 0x026fa6 to 0x032b83 (0xbbde bytes) 000612 1788583171.927668 mapwrite 0x02db9c thru 0x02deca (0x32f bytes) 000614 1788583171.994773 trunc from 0x040000 to 0x01284b (0x2d7b4 bytes) 000615 1788583172.021091 punch from 0x002c9d to 0x002e44 (0x1a8 bytes) 000616 1788583172.025034 punch from 0x00ed5f to 0x01284b (0x3aed bytes) 000617 1788583172.033315 mapread 0x00865b thru 0x01104d (0x89f3 bytes) 000620 1788583172.050687 read 0x00a2c3 thru 0x01284b (0x8589 bytes) 000621 1788583172.051869 trunc from 0x01284c to 0x02d5a5 (0x1ad5a bytes) 000622 1788583172.071308 punch from 0x0261b3 to 0x02d5a5 (0x73f3 bytes) 000624 1788583172.075524 write 0x00a401 thru 0x0198a7 (0xf4a7 bytes) 000625 1788583172.085129 mapwrite 0x039c0d thru 0x03b1b8 (0x15ac bytes) 000626 1788583172.170836 read 0x032278 thru 0x03b1b8 (0x8f41 bytes) 000627 1788583172.175762 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3b1b9): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b1b9): Mirror 9: 0x8d054449 (stale) Mirror 10: 0x8d054449 (stale) Mirror 12: 0x8d054449 (stale) Mirror 13: 0x8d054449 (stale) Mirror 14: 0x8d054449 (stale) Mirror 15: 0x53807384 (stale) CRC-32 checksum value for chunk [0, 0x3b1b9): Mirror 1: 0x56993894 Mirror 16: 0x56993894 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b1b9: skipped 000628 1788583172.815817 mapwrite 0x00b019 thru 0x019c5b (0xec43 bytes) 000630 1788583172.942019 punch from 0x00f269 to 0x016fb0 (0x7d48 bytes) 000631 1788583172.946532 MIRROR_RESYNC 8 mirrors 000632 1788583173.987184 mapwrite 0x03c49b thru 0x03ffff (0x3b65 bytes) 000633 1788583174.107534 punch from 0x0127b9 to 0x01c78c (0x9fd4 bytes) 000634 1788583174.113186 mapread 0x02b97e thru 0x034c6c (0x92ef bytes) 000635 1788583174.126142 mapread 0x0114d9 thru 0x013b55 (0x267d bytes) 000637 1788583174.138968 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x7ed0867b (stale) Mirror 10: 0x7ed0867b (stale) Mirror 12: 0x7ed0867b (stale) Mirror 13: 0x7ed0867b (stale) Mirror 14: 0x7ed0867b (stale) Mirror 16: 0x7ed0867b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x7ed0867b (stale) Mirror 10: 0x7ed0867b (stale) Mirror 12: 0x7ed0867b (stale) Mirror 13: 0x7ed0867b (stale) Mirror 14: 0x7ed0867b (stale) Mirror 16: 0x7ed0867b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000638 1788583174.523452 mapread 0x027e8e thru 0x035fe3 (0xe156 bytes) 000639 1788583174.551943 write 0x00572f thru 0x00aa11 (0x52e3 bytes) 000640 1788583174.559748 punch from 0x03c639 to 0x03ffff (0x39c7 bytes) 000644 1788583174.572332 read 0x017597 thru 0x0196c1 (0x212b bytes) 000645 1788583174.577531 mapwrite 0x00aac4 thru 0x010bc1 (0x60fe bytes) 000646 1788583174.618322 trunc from 0x040000 to 0x01063b (0x2f9c4 bytes) 000647 1788583174.633770 trunc from 0x01063c to 0x01f6e4 (0xf0a9 bytes) 000649 1788583174.648747 mapwrite 0x024c1e thru 0x02fcfe (0xb0e1 bytes) 000650 1788583174.735902 mapwrite 0x00e566 thru 0x018d2f (0xa7ca bytes) 000651 1788583174.794626 punch from 0x01a966 to 0x020f7a (0x6615 bytes) 000652 1788583174.800002 read 0x00b0ad thru 0x00f3bb (0x430f bytes) 000654 1788583174.810501 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fcff): Mirror 10: 0x94b96dd5 (stale) Mirror 12: 0x94b96dd5 (stale) Mirror 13: 0x94b96dd5 (stale) Mirror 14: 0x94b96dd5 (stale) Mirror 16: 0x94b96dd5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fcff): Mirror 10: 0x94b96dd5 (stale) Mirror 12: 0x94b96dd5 (stale) Mirror 13: 0x94b96dd5 (stale) Mirror 14: 0x94b96dd5 (stale) Mirror 16: 0x94b96dd5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fcff: skipped 000656 1788583175.138793 punch from 0x02f8ad to 0x02fcfe (0x452 bytes) 000658 1788583175.145088 punch from 0x01f99f to 0x02c58f (0xcbf1 bytes) 000660 1788583175.149705 write 0x01cfce thru 0x02404a (0x707d bytes) 000662 1788583175.159559 mapread 0x019f17 thru 0x020988 (0x6a72 bytes) 000663 1788583175.178299 trunc from 0x02fcff to 0x01fcf2 (0x1000c bytes) 000664 1788583175.201044 mapwrite 0x008c59 thru 0x016735 (0xdadd bytes) 000666 1788583175.265621 read 0x00d1dd thru 0x00e9bf (0x17e3 bytes) 000667 1788583175.266337 write 0x00c03f thru 0x011414 (0x53d6 bytes) 000669 1788583175.269523 read 0x0106e2 thru 0x01f35f (0xec7e bytes) 000670 1788583175.270270 write 0x038e49 thru 0x03a6c3 (0x187b bytes) 000671 1788583175.276105 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a6c4): Mirror 12: 0x8155164e (stale) Mirror 13: 0x8155164e (stale) Mirror 14: 0x8155164e (stale) Mirror 16: 0x8155164e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a6c4): Mirror 12: 0x8155164e (stale) Mirror 13: 0x8155164e (stale) Mirror 14: 0x8155164e (stale) Mirror 16: 0x8155164e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a6c4: skipped 000673 1788583175.657136 punch from 0x03066f to 0x03a6c3 (0xa055 bytes) 000675 1788583175.668926 mapread 0x02ea03 thru 0x0300f1 (0x16ef bytes) 000678 1788583175.687575 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a6c4): Mirror 12: 0x8155164e (stale) Mirror 13: 0x8155164e (stale) Mirror 16: 0x8155164e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a6c4): Mirror 12: 0x8155164e (stale) Mirror 13: 0x8155164e (stale) Mirror 16: 0x8155164e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a6c4: skipped 000680 1788583176.057939 mapwrite 0x02b102 thru 0x038c75 (0xdb74 bytes) 000684 1788583176.157197 trunc from 0x03a6c4 to 0x01a50e (0x201b5 bytes) 000686 1788583176.188224 punch from 0x00bb03 to 0x016914 (0xae12 bytes) 000687 1788583176.195847 punch from 0x00410b to 0x012b50 (0xea46 bytes) 000688 1788583176.203706 write 0x004ea2 thru 0x0114bd (0xc61c bytes) 000690 1788583176.216273 mapread 0x002529 thru 0x00d199 (0xac71 bytes) 000691 1788583176.230223 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a50f): Mirror 13: 0x576b0ecd (stale) Mirror 16: 0x576b0ecd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a50f): Mirror 13: 0x576b0ecd (stale) Mirror 16: 0x576b0ecd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a50f: skipped 000693 1788583176.822956 mapread 0x0191c0 thru 0x01a50e (0x134f bytes) 000694 1788583176.853812 trunc from 0x01a50f to 0x008d39 (0x117d5 bytes) 000695 1788583176.886088 mapwrite 0x01b667 thru 0x02342b (0x7dc5 bytes) 000696 1788583177.101932 read 0x01a231 thru 0x02342b (0x91fb bytes) 000697 1788583177.102801 write 0x01af3a thru 0x02a53b (0xf602 bytes) 000698 1788583177.104969 mapwrite 0x036956 thru 0x03a821 (0x3ecc bytes) 000699 1788583177.181999 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a822): Mirror 13: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a822): Mirror 13: 0x5b287548 (stale) Mirror 16: 0x5b287548 (stale) CRC-32 checksum value for chunk [0, 0x3a822): Mirror 1: 0x4d754647 Mirror 17: 0x4d754647 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a822: skipped 000703 1788583178.020278 mapwrite 0x03e42f thru 0x03ffff (0x1bd1 bytes) 000704 1788583178.351123 mapread 0x031a54 thru 0x0384ee (0x6a9b bytes) 000705 1788583178.388752 punch from 0x0028c1 to 0x00d491 (0xabd1 bytes) 000706 1788583178.401608 read 0x006771 thru 0x012105 (0xb995 bytes) 000707 1788583178.421085 write 0x034d97 thru 0x03ffff (0xb269 bytes) 000711 1788583178.432667 punch from 0x027495 to 0x0356ce (0xe23a bytes) 000712 1788583178.473435 read 0x01db6a thru 0x02964a (0xbae1 bytes) 000713 1788583178.483571 punch from 0x024133 to 0x030acd (0xc99b bytes) 000714 1788583178.499144 MIRROR_RESYNC 4 mirrors 000716 1788583179.922052 mapwrite 0x00ee7f thru 0x01c283 (0xd405 bytes) 000718 1788583180.113754 MIRROR_RESYNC 4 mirrors 000719 1788583180.984932 write 0x0051f5 thru 0x010702 (0xb50e bytes) 000720 1788583181.037908 read 0x00dc77 thru 0x01c016 (0xe3a0 bytes) 000721 1788583181.047601 mapread 0x00ce64 thru 0x018407 (0xb5a4 bytes) 000722 1788583181.048547 read 0x03bbc9 thru 0x03d607 (0x1a3f bytes) 000723 1788583181.057340 trunc from 0x040000 to 0x015ee1 (0x2a11e bytes) truncating to largest ever: 0x3f639 000725 1788583181.074370 trunc from 0x015ee2 to 0x03f638 (0x29757 bytes) 000726 1788583181.082130 mapwrite 0x0139cc thru 0x014971 (0xfa6 bytes) 000727 1788583181.115973 MIRROR_RESYNC 4 mirrors 000729 1788583181.775142 mapread 0x00f452 thru 0x01c8b6 (0xd465 bytes) 000730 1788583181.813565 trunc from 0x03f639 to 0x037624 (0x8014 bytes) 000733 1788583181.959051 mapread 0x01be8f thru 0x01c20e (0x380 bytes) 000735 1788583181.978922 write 0x031636 thru 0x03e8d3 (0xd29e bytes) 000736 1788583181.984153 trunc from 0x03e8d4 to 0x039697 (0x523c bytes) 000738 1788583182.021867 write 0x0198d9 thru 0x020951 (0x7079 bytes) 000739 1788583182.024901 mapwrite 0x00699d thru 0x015d70 (0xf3d4 bytes) 000740 1788583182.136586 write 0x03b9b7 thru 0x03da35 (0x207f bytes) 000741 1788583182.138106 trunc from 0x03da36 to 0x025268 (0x187cd bytes) 000743 1788583182.161377 write 0x0061cc thru 0x013619 (0xd44e bytes) 000744 1788583182.167915 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25269): Mirror 13: 0xf63a43a5 (stale) Mirror 17: 0xf63a43a5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25269): Mirror 13: 0xf63a43a5 (stale) Mirror 17: 0xf63a43a5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25269: skipped 000745 1788583182.460980 mapwrite 0x00f259 thru 0x011b4f (0x28f7 bytes) 000746 1788583182.525589 punch from 0x011c5b to 0x016126 (0x44cc bytes) 000747 1788583182.535653 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25269): Mirror 17: 0xf63a43a5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25269): Mirror 17: 0xf63a43a5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25269: skipped 000748 1788583182.789640 write 0x032388 thru 0x03c917 (0xa590 bytes) 000750 1788583182.793139 punch from 0x02a6c5 to 0x02c16b (0x1aa7 bytes) 000751 1788583182.807125 punch from 0x02e439 to 0x03bd96 (0xd95e bytes) 000752 1788583182.811989 mapread 0x005ad4 thru 0x0108ab (0xadd8 bytes) 000753 1788583182.835948 write 0x0184a7 thru 0x01890d (0x467 bytes) 000754 1788583182.844530 read 0x014087 thru 0x01cafa (0x8a74 bytes) 000755 1788583182.857356 read 0x03bac7 thru 0x03c917 (0xe51 bytes) 000756 1788583182.879955 mapwrite 0x016ea8 thru 0x023313 (0xc46c bytes) 000757 1788583182.943623 mapread 0x0218de thru 0x02fde3 (0xe506 bytes) 000759 1788583182.951218 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3c918): Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c918): Mirror 17: 0x112155dd (stale) CRC-32 checksum value for chunk [0, 0x3c918): Mirror 1: 0x781af3bb Mirror 18: 0x781af3bb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c918: skipped 000760 1788583183.772202 mapwrite 0x00a0e5 thru 0x011461 (0x737d bytes) 000761 1788583183.945958 mapwrite 0x00c6d1 thru 0x00ca64 (0x394 bytes) 000762 1788583183.985653 mapwrite 0x000f53 thru 0x0071e1 (0x628f bytes) 000763 1788583184.039464 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c918): Mirror 18: 0x781af3bb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c918): Mirror 18: 0x781af3bb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c918: skipped 000765 1788583184.326163 mapwrite 0x02f885 thru 0x031702 (0x1e7e bytes) 000766 1788583184.403352 read 0x01b79b thru 0x024e03 (0x9669 bytes) 000767 1788583184.414111 read 0x01000a thru 0x01cc4f (0xcc46 bytes) 000769 1788583184.429330 write 0x0294f8 thru 0x02a9c4 (0x14cd bytes) 000770 1788583184.430424 read 0x003656 thru 0x011b88 (0xe533 bytes) 000771 1788583184.437262 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) 000773 1788583184.673152 trunc from 0x03c918 to 0x025205 (0x17712 bytes) 000774 1788583184.691182 mapwrite 0x01f1f0 thru 0x02eafd (0xf90e bytes) 000776 1788583184.798211 punch from 0x012bd0 to 0x01e54a (0xb97b bytes) 000777 1788583184.803660 write 0x03b8d3 thru 0x03ffff (0x472d bytes) 000779 1788583184.806438 mapread 0x010950 thru 0x018c57 (0x8308 bytes) 000780 1788583184.819023 read 0x034a2f thru 0x03ecd3 (0xa2a5 bytes) 000781 1788583184.826156 mapwrite 0x00e598 thru 0x018222 (0x9c8b bytes) 000782 1788583184.870618 write 0x002772 thru 0x00e7c0 (0xc04f bytes) 000783 1788583184.876072 trunc from 0x040000 to 0x034645 (0xb9ba bytes) 000784 1788583184.891184 mapwrite 0x01d0c5 thru 0x0267bd (0x96f9 bytes) 000787 1788583184.938170 read 0x013e66 thru 0x01ab3c (0x6cd7 bytes) 000788 1788583184.939060 write 0x0166e5 thru 0x01e724 (0x8040 bytes) 000789 1788583184.941201 trunc from 0x034646 to 0x001930 (0x32d15 bytes) 000790 1788583184.976645 read 0x0014e0 thru 0x001930 (0x451 bytes) 000791 1788583184.988364 punch from 0x0009a1 to 0x001930 (0xf90 bytes) 000793 1788583185.004788 read 0x00030f thru 0x001930 (0x1622 bytes) 000794 1788583185.013145 trunc from 0x001931 to 0x01944e (0x17b1e bytes) 000795 1788583185.036272 mapwrite 0x0299ac thru 0x0326a4 (0x8cf9 bytes) 000797 1788583185.204640 punch from 0x000dcd to 0x006564 (0x5798 bytes) 000799 1788583185.227615 MIRROR_EXTEND to 2 mirrors 000800 1788583186.616969 trunc from 0x0326a5 to 0x007a49 (0x2ac5b bytes) 000802 1788583186.775786 mapwrite 0x030d52 thru 0x032fdd (0x228c bytes) 000803 1788583186.876648 punch from 0x016fdf to 0x0227d2 (0xb7f4 bytes) 000804 1788583186.883560 mapwrite 0x00bd0f thru 0x00dbef (0x1ee1 bytes) 000805 1788583186.946768 read 0x01790e thru 0x018557 (0xc4a bytes) 000807 1788583186.959005 mapwrite 0x02e6aa thru 0x036472 (0x7dc9 bytes) 000809 1788583187.072731 mapwrite 0x00ab24 thru 0x0143cc (0x98a9 bytes) 000810 1788583187.154982 write 0x02a229 thru 0x0311ae (0x6f86 bytes) 000811 1788583187.161822 write 0x014b2d thru 0x01543e (0x912 bytes) 000812 1788583187.164872 punch from 0x01730a to 0x01999b (0x2692 bytes) 000813 1788583187.184140 write 0x027d50 thru 0x0297a1 (0x1a52 bytes) 000814 1788583187.200248 punch from 0x022df5 to 0x028d67 (0x5f73 bytes) 000815 1788583187.211065 write 0x026715 thru 0x02eb25 (0x8411 bytes) 000817 1788583187.233317 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36473): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36473): Mirror 2: 0x6c4f9dca (stale) CRC-32 checksum value for chunk [0, 0x36473): Mirror 1: 0x559799cb Mirror 3: 0x559799cb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36473: skipped 000818 1788583188.083360 read 0x010882 thru 0x01f7e7 (0xef66 bytes) 000819 1788583188.101200 write 0x01060c thru 0x01413b (0x3b30 bytes) 000821 1788583188.179316 mapread 0x01615f thru 0x023044 (0xcee6 bytes) 000822 1788583188.209348 mapwrite 0x0100f0 thru 0x017ee2 (0x7df3 bytes) 000823 1788583188.257679 read 0x0013bb thru 0x00265e (0x12a4 bytes) 000824 1788583188.268620 mapwrite 0x004a59 thru 0x0105e0 (0xbb88 bytes) 000825 1788583188.319377 MIRROR_RESYNC 3 mirrors 000826 1788583189.019074 read 0x030102 thru 0x036472 (0x6371 bytes) 000828 1788583189.033906 read 0x01f35d thru 0x02cb23 (0xd7c7 bytes) 000829 1788583189.051615 trunc from 0x036473 to 0x000e7f (0x355f3 bytes) 000831 1788583189.239914 punch from 0x000cf4 to 0x000e7f (0x18c bytes) 000832 1788583189.246861 read 0x000d04 thru 0x000e7f (0x17c bytes) 000836 1788583189.255028 mapread 0x000c25 thru 0x000e7f (0x25b bytes) 000837 1788583189.257829 mapwrite 0x01b8fa thru 0x01d7db (0x1ee2 bytes) 000838 1788583189.402024 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1d7dc): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d7dc): Mirror 2: 0x87e51b8a (stale) Mirror 3: 0x87e51b8a (stale) CRC-32 checksum value for chunk [0, 0x1d7dc): Mirror 1: 0x9974642a Mirror 4: 0x9974642a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d7dc: skipped 000840 1788583190.242990 write 0x02913e thru 0x029b91 (0xa54 bytes) 000841 1788583190.333784 punch from 0x011f0a to 0x012d12 (0xe09 bytes) 000842 1788583190.349936 read 0x006b3f thru 0x00cdcc (0x628e bytes) 000843 1788583190.360899 write 0x00d5e8 thru 0x01be2f (0xe848 bytes) 000846 1788583190.373899 punch from 0x0043b5 to 0x0096e4 (0x5330 bytes) 000847 1788583190.407702 MIRROR_RESYNC 4 mirrors 000848 1788583191.460492 read 0x01c15d thru 0x022b5b (0x69ff bytes) 000849 1788583191.473774 MIRROR_RESYNC 4 mirrors 000850 1788583192.109245 read 0x01b9d3 thru 0x029b91 (0xe1bf bytes) 000853 1788583192.118960 write 0x01775b thru 0x01818c (0xa32 bytes) 000854 1788583192.300072 mapread 0x028638 thru 0x029b91 (0x155a bytes) 000855 1788583192.316159 write 0x03724e thru 0x03ffff (0x8db2 bytes) 000856 1788583192.321835 mapread 0x01a325 thru 0x029d7c (0xfa58 bytes) 000857 1788583192.383757 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb954905a (stale) Mirror 3: 0xb954905a (stale) Mirror 4: 0xb954905a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb47c2beb Mirror 5: 0xb47c2beb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000858 1788583193.293670 punch from 0x00a462 to 0x00e240 (0x3ddf bytes) 000859 1788583193.353940 punch from 0x03dfab to 0x03ffff (0x2055 bytes) 000860 1788583193.363112 mapread 0x033322 thru 0x03865c (0x533b bytes) 000861 1788583193.381321 mapread 0x0043f9 thru 0x00abca (0x67d2 bytes) 000862 1788583193.394666 trunc from 0x040000 to 0x006cd4 (0x3932b bytes) 000863 1788583193.439259 punch from 0x006392 to 0x006cd4 (0x943 bytes) 000864 1788583193.448253 read 0x003454 thru 0x006cd4 (0x3881 bytes) 000865 1788583193.453789 write 0x00387e thru 0x00cee3 (0x9666 bytes) 000866 1788583193.455069 read 0x005bd6 thru 0x007a5e (0x1e89 bytes) 000867 1788583193.455628 punch from 0x00b77d to 0x00cee3 (0x1767 bytes) 000868 1788583193.469687 write 0x0315c1 thru 0x03729d (0x5cdd bytes) 000869 1788583193.475843 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3729e): Mirror 2: 0x42e1affc (stale) Mirror 3: 0x42e1affc (stale) Mirror 5: 0x6608a979 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3729e): Mirror 2: 0x42e1affc (stale) Mirror 3: 0x42e1affc (stale) Mirror 5: 0x6608a979 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3729e: skipped 000872 1788583193.927433 punch from 0x02148b to 0x02fb4b (0xe6c1 bytes) 000874 1788583193.937266 trunc from 0x03729e to 0x021c0a (0x15693 bytes) 000875 1788583193.976952 MIRROR_RESYNC 4 mirrors 000876 1788583195.380799 mapread 0x01146b thru 0x018d05 (0x789b bytes) 000877 1788583195.404097 punch from 0x011fa2 to 0x01280a (0x869 bytes) 000878 1788583195.478982 trunc from 0x021c0b to 0x00e96d (0x1329d bytes) 000880 1788583195.506288 read 0x005f4d thru 0x0073e3 (0x1497 bytes) 000881 1788583195.512932 punch from 0x004819 to 0x005984 (0x116c bytes) 000882 1788583195.517661 mapwrite 0x01d6e7 thru 0x02c2e5 (0xebff bytes) 000883 1788583195.599915 trunc from 0x02c2e6 to 0x014581 (0x17d64 bytes) 000884 1788583195.627092 mapwrite 0x01277d thru 0x01e99e (0xc222 bytes) 000885 1788583195.723846 read 0x01b00d thru 0x01e99e (0x3992 bytes) 000886 1788583195.725172 trunc from 0x01e99f to 0x039053 (0x1a6b5 bytes) 000887 1788583195.753101 MIRROR_RESYNC 4 mirrors 000888 1788583196.785344 write 0x02c052 thru 0x030350 (0x42ff bytes) 000889 1788583196.845309 trunc from 0x039054 to 0x01700e (0x22045 bytes) 000890 1788583196.863829 mapwrite 0x03940d thru 0x03ffff (0x6bf3 bytes) 000891 1788583197.015265 trunc from 0x040000 to 0x026a58 (0x195a7 bytes) 000893 1788583197.058182 MIRROR_RESYNC 4 mirrors 000896 1788583198.017090 write 0x006e9b thru 0x01387e (0xc9e4 bytes) 000898 1788583198.081458 trunc from 0x026a59 to 0x019767 (0xd2f1 bytes) 000899 1788583198.106282 mapwrite 0x033a37 thru 0x03bf77 (0x8541 bytes) 000900 1788583198.217027 mapread 0x0075bc thru 0x00ba12 (0x4457 bytes) 000901 1788583198.219841 mapwrite 0x036a0d thru 0x03ffff (0x95f3 bytes) 000902 1788583198.278217 punch from 0x02c0b5 to 0x0361c8 (0xa114 bytes) 000905 1788583198.290690 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x742fe2f6 (stale) Mirror 5: 0x742fe2f6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x742fe2f6 (stale) Mirror 5: 0x742fe2f6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000906 1788583198.628304 mapwrite 0x033943 thru 0x03b818 (0x7ed6 bytes) 000907 1788583198.721022 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x742fe2f6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x742fe2f6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000908 1788583198.957012 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) 000909 1788583199.071546 mapwrite 0x0026e2 thru 0x00be0b (0x972a bytes) 000911 1788583199.128435 mapread 0x030117 thru 0x034d33 (0x4c1d bytes) 000913 1788583199.143574 read 0x036aa7 thru 0x03ffff (0x9559 bytes) 000914 1788583199.150229 mapwrite 0x02ff38 thru 0x03ec3f (0xed08 bytes) 000915 1788583199.194332 mapread 0x02e410 thru 0x02f143 (0xd34 bytes) 000917 1788583199.195528 punch from 0x008518 to 0x014266 (0xbd4f bytes) 000918 1788583199.200024 write 0x006266 thru 0x013ba4 (0xd93f bytes) 000919 1788583199.206086 punch from 0x00894e to 0x01214a (0x97fd bytes) 000920 1788583199.227321 punch from 0x00a34f to 0x01015b (0x5e0d bytes) 000921 1788583199.232819 mapwrite 0x018bcc thru 0x0219c2 (0x8df7 bytes) 000922 1788583199.297834 read 0x00a3ee thru 0x013a57 (0x966a bytes) 000924 1788583199.306619 write 0x000242 thru 0x00999f (0x975e bytes) 000925 1788583199.308158 punch from 0x01da90 to 0x01ec56 (0x11c7 bytes) 000927 1788583199.320880 mapread 0x0297ee thru 0x036176 (0xc989 bytes) 000929 1788583199.324375 mapread 0x00b96f thru 0x01a0f2 (0xe784 bytes) 000930 1788583199.325914 write 0x00f57f thru 0x01f115 (0xfb97 bytes) 000931 1788583199.327534 write 0x00e6a2 thru 0x00e76f (0xce bytes) 000932 1788583199.333356 MIRROR_EXTEND to 2 mirrors 000934 1788583200.132212 MIRROR_EXTEND to 3 mirrors 000935 1788583200.790317 trunc from 0x040000 to 0x01143c (0x2ebc3 bytes) 000936 1788583200.880990 trunc from 0x01143d to 0x021759 (0x1031d bytes) 000937 1788583200.907926 mapread 0x00a8f7 thru 0x015751 (0xae5b bytes) 000939 1788583200.934253 read 0x02138e thru 0x021759 (0x3cc bytes) 000941 1788583200.939308 read 0x01f3bb thru 0x021759 (0x239f bytes) 000942 1788583200.949351 mapread 0x01e0aa thru 0x021759 (0x36b0 bytes) 000944 1788583200.950782 mapwrite 0x03c71b thru 0x03fe34 (0x371a bytes) 000945 1788583201.063893 read 0x035589 thru 0x038442 (0x2eba bytes) 000946 1788583201.064529 mapwrite 0x006283 thru 0x012182 (0xbf00 bytes) 000948 1788583201.115704 read 0x0202a5 thru 0x02e3e4 (0xe140 bytes) 000950 1788583201.117273 read 0x03fc62 thru 0x03fe34 (0x1d3 bytes) 000951 1788583201.118209 mapread 0x016b4b thru 0x022bb7 (0xc06d bytes) 000952 1788583201.120174 write 0x03eca1 thru 0x03ffff (0x135f bytes) 000953 1788583201.121794 mapread 0x03d2e2 thru 0x03ffff (0x2d1e bytes) 000954 1788583201.128014 MIRROR_RESYNC 3 mirrors 000955 1788583202.205303 mapwrite 0x025e85 thru 0x033c07 (0xdd83 bytes) 000957 1788583202.402801 punch from 0x00a2ce to 0x0121e9 (0x7f1c bytes) 000958 1788583202.413459 read 0x03f122 thru 0x03ffff (0xede bytes) 000959 1788583202.421887 mapwrite 0x021761 thru 0x02a613 (0x8eb3 bytes) 000960 1788583202.538661 punch from 0x01f880 to 0x025de9 (0x656a bytes) 000961 1788583202.553142 mapwrite 0x01d425 thru 0x02d2a2 (0xfe7e bytes) 000963 1788583202.681871 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x984f9eea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x984f9eea (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000964 1788583203.040223 write 0x0353cb thru 0x03fddc (0xaa12 bytes) 000965 1788583203.050829 write 0x02fe8b thru 0x035c99 (0x5e0f bytes) 000966 1788583203.055968 mapwrite 0x03021f thru 0x03ecbe (0xeaa0 bytes) 000967 1788583203.100200 trunc from 0x040000 to 0x01e6a1 (0x2195e bytes) 000969 1788583203.139065 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) 000970 1788583203.242710 write 0x024fdf thru 0x026f6b (0x1f8d bytes) 000971 1788583203.244087 write 0x004ac1 thru 0x00b283 (0x67c3 bytes) 000972 1788583203.254863 mapread 0x02372c thru 0x026f6b (0x3840 bytes) 000975 1788583203.261374 mapread 0x02321d thru 0x026f6b (0x3d4f bytes) 000977 1788583203.263058 trunc from 0x026f6c to 0x03f580 (0x18615 bytes) 000982 1788583203.285814 read 0x020f77 thru 0x02cc31 (0xbcbb bytes) 000984 1788583203.298515 mapread 0x03722d thru 0x037845 (0x619 bytes) 000986 1788583203.304320 read 0x02158e thru 0x026b7d (0x55f0 bytes) 000987 1788583203.305089 mapwrite 0x03c759 thru 0x03ffff (0x38a7 bytes) 000989 1788583203.367524 read 0x02d3e9 thru 0x031675 (0x428d bytes) 000990 1788583203.368115 write 0x02ef8c thru 0x02f7b5 (0x82a bytes) 000991 1788583203.368966 mapwrite 0x02f190 thru 0x0314d1 (0x2342 bytes) 000992 1788583203.404908 write 0x006c4b thru 0x012c81 (0xc037 bytes) 000994 1788583203.406570 mapwrite 0x03d64c thru 0x03ffff (0x29b4 bytes) 000995 1788583203.442201 mapwrite 0x0395be thru 0x039d8c (0x7cf bytes) 000997 1788583203.475304 mapread 0x028cb4 thru 0x033f3e (0xb28b bytes) 000998 1788583203.477154 trunc from 0x040000 to 0x015d10 (0x2a2ef bytes) 000999 1788583203.491201 read 0x00edbd thru 0x015d10 (0x6f54 bytes) 001000 1788583203.492245 mapread 0x00d2f7 thru 0x011794 (0x449e bytes) All operations completed A-OK! PASS 70a (88s) == sanity-flr test 71: check mirror extend parent fid ==== 00:40:07 (1788583207) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 00:40:10 (1788583210) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.24524 s, 8.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203629 s, 5.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f8:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 00:40:20 (1788583220) Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre2 Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3647168 .. Extending file size to 725504 .. done Extending file size to 7286944 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8193248 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8718688 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7724960 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8498624 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8885504 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9145440 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8917024 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 57463 57464 57467 57468 57469 192.168.201.129@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg129-client.virtnet /mnt/lustre2 (opts:) 192.168.201.129@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg129-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 00:41:40 (1788583300) Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre2 Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 900480 .. Extending file size to 963264 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5478944 .. Extending file size to 5588768 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8918656 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7079328 .. Extending file size to 7131584 .. Extending file size to 7341376 .. Extending file size to 7764960 .. done Extending file size to 8264544 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8618816 .. Extending file size to 8717472 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9277920 .. Extending file size to 8936896 .. Waiting 59041 59042 59043 59045 59048 done 192.168.201.129@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg129-client.virtnet /mnt/lustre2 (opts:) 192.168.201.129@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg129-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (84s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 00:43:04 (1788583384) Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.140382 s, 14.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.31306 s, 6.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xc34c4f6c (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.129@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg129-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 00:43:25 (1788583405) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.134908 s, 15.5 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 00:43:33 (1788583413) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x30d:0x0] } PASS 203 (10s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 00:43:43 (1788583423) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.245787 s, 8.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 00:43:52 (1788583432) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.193899 s, 10.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x310:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x310:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x311:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 00:44:02 (1788583442) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.384355 s, 5.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x312:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x312:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x313:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 00:44:37 (1788583477) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg129-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173763 s, 6.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x315:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x314:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x315:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x314:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } PASS 204d (59s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 00:45:36 (1788583536) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg129-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.362628 s, 5.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x316:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x316:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (63s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 00:46:40 (1788583600) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg129-server mds-ost sync done. Creating new pool test_204f oleg129-server: Pool lustre.test_204f created Adding targets to pool oleg129-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg129-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.270539 s, 7.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x317:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x317:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg129-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg129-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg129-server: Pool lustre.test_204f destroyed PASS 204f (84s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 00:48:04 (1788583684) PASS 205a (11s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 00:48:15 (1788583695) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 00:48:26 (1788583706) oleg129-server: Pool lustre.test_206 created Adding targets to pool oleg129-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg129-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg129-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg129-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg129-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 00:48:55 (1788583735) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473124 s, 22.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501257 s, 20.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x344c7a Mirror 3: 0x344c7a PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 00:49:08 (1788583748) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 00:49:12 (1788583752) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 00:49:16 (1788583756) Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.144824 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xfb0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1fc0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1fc0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1fc0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x960000 Read this (4 bytes): BBBB 192.168.201.129@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg129-client.virtnet /mnt/lustre2 (opts:) PASS 209a (35s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 00:49:51 (1788583791) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.012413 s, 660 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 00:49:58 (1788583798) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712212 s, 14.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x242 lmm_fid: [0x200000404:0x242:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x242 lmm_fid: [0x200000404:0x242:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x321:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834498 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 00:50:00.000000000 -0400 Modify: 2026-09-05 00:50:00.000000000 -0400 Change: 2026-09-05 00:50:02.000000000 -0400 Birth: 2026-09-05 00:50:00.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 00:50:06 (1788583806) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 00:50:10 (1788583810) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.557808 s, 73.4 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0402612 s, 102 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 5 00:50 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x244:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 5 00:50 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 00:50:17 (1788583817) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 00:50:26 (1788583826) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1778 sec =========== 00:50:35 (1788583835) === sanity-flr: start cleanup 00:50:37 (1788583837) === === sanity-flr: finish cleanup 00:50:43 (1788583843) ===