-----============= acceptance-small: sanity-flr ============----- Wed Jul 15 01:23:49 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg419-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 01:23:59 (1784093039) === oleg419-client.virtnet: executing check_config_client /mnt/lustre oleg419-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg419-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff99f1c7e0e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff99f1c7e0e800.idle_timeout=debug disable quota as required oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 01:24:18 (1784093058) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7584] == sanity-flr test 0a: lfs mirror create with -N option == 01:24:19 (1784093059) PASS 0a (9s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:24:28 (1784093068) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:24:30 (1784093070) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:24:32 (1784093072) PASS 0d (15s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:24:47 (1784093087) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:24:49 (1784093089) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 01:24:51 (1784093091) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff99f1c7e0e800.stats=clear osc.lustre-OST0001-osc-ffff99f1c7e0e800.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:24:55 (1784093095) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (6s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:25:01 (1784093101) PASS 0j (5s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:25:06 (1784093106) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 0m: mirror extend with -N= option ===== 01:25:07 (1784093107) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:25:11 (1784093111) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 01:25:15 (1784093115) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:25:18 (1784093118) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:25:20 (1784093120) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (4s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:25:24 (1784093124) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:25:29 (1784093129) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:25:32 (1784093132) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.532596 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066158 s, 15.8 MB/s PASS 21 (4s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:25:36 (1784093136) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.0707996 s, 88.9 MB/s PASS 22 (4s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 01:25:40 (1784093140) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:25:44 (1784093144) Stopping /mnt/lustre-ost1 (opts:) on oleg419-server oleg419-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99f1c7e0e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99f1c7e0e800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg419-server: oleg419-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg419-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg419-server oleg419-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99f1c7e0e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99f1c7e0e800.ost_server_uuid in DISCONN state after 1 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg419-server: oleg419-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg419-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (33s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:26:17 (1784093177) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.140688 s, 44.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg419-server oleg419-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99f1c7e0e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99f1c7e0e800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg419-server: oleg419-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg419-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (21s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:26:38 (1784093198) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg419-server oleg419-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99f1c7e0e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99f1c7e0e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg419-server: oleg419-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg419-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg419-server oleg419-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99f1c7e0e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99f1c7e0e800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg419-server: oleg419-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg419-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (40s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:27:18 (1784093238) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.39866 s, 21.9 MB/s -rw-r--r-- 1 root root 52428800 Jul 15 01:27 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.539s user 0m0.001s sys 0m0.175s Stopping /mnt/lustre-ost1 (opts:) on oleg419-server oleg419-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99f1c7e0e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99f1c7e0e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.569s user 0m0.000s sys 0m0.176s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg419-server: oleg419-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg419-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg419-server oleg419-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99f1c7e0e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99f1c7e0e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.726s user 0m0.001s sys 0m0.201s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg419-server: oleg419-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg419-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg419-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99f1c7e0e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99f1c7e0e800.ost_server_uuid in FULL state after 0 sec PASS 33b (48s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 01:28:06 (1784093286) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 01:28:08 (1784093288) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 01:28:10 (1784093290) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 01:28:11 (1784093291) PASS 35 (4s) == sanity-flr test 36a: write to mirrored files ========== 01:28:15 (1784093295) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.375196 s, 33.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (7s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:28:22 (1784093302) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186492 s, 56.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224237 s, 46.8 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:28:26 (1784093306) PASS 36c (5s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:28:30 (1784093310) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00102639 s, 998 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00132715 s, 772 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00157578 s, 650 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00108114 s, 947 kB/s flv=10 olv=10 PASS 36d (5s) == sanity-flr test 37: mirror I/O API verification ======= 01:28:35 (1784093315) create test files with size 15734419 .. 277e9defece101445444b8e3b5750d8b - b1b2e7f2a58c8ecec07a0d78ca9b1b7f - 25b3cf509a75e89b3a8d9e77b26f1398 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg419-client.virtnet /mnt/lustre (opts:) Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre PASS 37 (9s) == sanity-flr test 38: resync ============================ 01:28:45 (1784093325) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (8s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:28:52 (1784093332) PASS 39 (4s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:28:56 (1784093336) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503954 s, 20.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155323 s, 67.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 01:29:00 (1784093340) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0552728 s, 37.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.172257 s, 24.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.112888 s, 18.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (7s) == sanity-flr test 42: lfs mirror verify ================= 01:29:07 (1784093347) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 01:29:09 (1784093349) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 01:29:10 (1784093350) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145036 s, 72.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 01:29:14 (1784093354) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 01:29:16 (1784093356) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 01:29:20 (1784093360) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.130131 s, 80.6 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg419-client.virtnet /mnt/lustre (opts:) Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (15s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 01:29:35 (1784093375) llite.lustre-ffff99f1c8dd3000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.139569 s, 75.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 Stopping client oleg419-client.virtnet /mnt/lustre (opts:) Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 llite.lustre-ffff99f1c86cb800.xattr_cache=1 PASS 44e (6s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 01:29:41 (1784093381) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (18s) == sanity-flr test 46: Verify setstripe --copy option ==== 01:29:59 (1784093399) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (16s) == sanity-flr test 47: Verify mirror obj alloc =========== 01:30:15 (1784093415) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 01:30:16 (1784093416) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0850421 s, 37.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0656822 s, 47.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x845666f7 (nosync,stale) Mirror 2: 0x845666f7 (nosync,stale) ** verify mirror 2 doesn't change original checksum: f772fc53726cf56c1f477f7bf4d3a3f1 - mirror 1 checksum: 1fdae3b8956b92350bd4958eb8ed3518 - mirror 2 checksum: f772fc53726cf56c1f477f7bf4d3a3f1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-15T01:30:18-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-15T01:30:18-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: f772fc53726cf56c1f477f7bf4d3a3f1 - mirror 1 checksum: 1fdae3b8956b92350bd4958eb8ed3518 - mirror 2 checksum: 1fdae3b8956b92350bd4958eb8ed3518 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-15T01:30:18-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-15T01:30:18-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } PASS 48 (4s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 01:30:21 (1784093421) Failing mds1 on oleg419-server Stopping /mnt/lustre-mds1 (opts:) on oleg419-server 01:30:24 (1784093424) shut down facet: mds1 facet_host: oleg419-server facet_failover_host: oleg419-server Failover mds1 to oleg419-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-MDT0000 01:30:39 (1784093439) targets are mounted 01:30:39 (1784093439) facet_failover done oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155551 s, 67.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } PASS 50A (26s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 01:30:47 (1784093447) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 01:30:48 (1784093448) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 01:30:50 (1784093450) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00486824 s, 3.4 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 01:30:53 (1784093453) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 01:30:54 (1784093454) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 01:30:56 (1784093456) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (9s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 01:31:05 (1784093465) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (21s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 01:31:26 (1784093486) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (19s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:31:45 (1784093505) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:32:03 (1784093523) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 01:32:06 (1784093526) mirror create pid 53824 mirror split pid 53826 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 53824 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 53826 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 01:33:08 (1784093588) Seed set to 3589 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x1314e 000002 1784093589.798551 trunc from 00000000 to 0x01314d (0x1314e bytes) 000004 1784093589.810071 mapread 0x010954 thru 0x01314d (0x27fa bytes) truncating to largest ever: 0x24109 000006 1784093589.828876 trunc from 0x01314e to 0x024108 (0x10fbb bytes) truncating to largest ever: 0x3b9a0 000009 1784093589.840322 trunc from 0x024109 to 0x03b99f (0x17897 bytes) 000010 1784093589.848446 trunc from 0x03b9a0 to 0x03b509 (0x496 bytes) 000013 1784093589.858926 write 0x009213 thru 0x01773b (0xe529 bytes) 000014 1784093589.865813 read 0x02e4db thru 0x03b509 (0xd02f bytes) 000016 1784093589.872316 read 0x011d06 thru 0x017842 (0x5b3d bytes) 000017 1784093589.872998 trunc from 0x03b50a to 0x02084b (0x1acbe bytes) 000018 1784093589.881617 mapread 0x00bf81 thru 0x012040 (0x60c0 bytes) 000024 1784093589.885341 mapwrite 0x00bd58 thru 0x010315 (0x45be bytes) 000030 1784093589.927978 MIRROR_EXTEND to 2 mirrors 000031 1784093590.212756 trunc from 0x02084c to 0x01be1d (0x4a2e bytes) 000033 1784093590.251469 trunc from 0x01be1e to 0x01a3ce (0x1a4f bytes) 000036 1784093590.262697 mapwrite 0x029cd1 thru 0x02b343 (0x1673 bytes) 000037 1784093590.302695 write 0x0090d1 thru 0x011ba1 (0x8ad1 bytes) 000039 1784093590.314207 write 0x02b80a thru 0x0362c5 (0xaabc bytes) 000040 1784093590.315639 read 0x00a4b9 thru 0x00f82b (0x5373 bytes) 000041 1784093590.316320 mapwrite 0x0213ec thru 0x022dd2 (0x19e7 bytes) 000042 1784093590.353573 mapread 0x02ec3e thru 0x033a2c (0x4def bytes) 000043 1784093590.357834 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x362c6): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x362c6): Mirror 2: 0x83a0e387 (stale) CRC-32 checksum value for chunk [0, 0x362c6): Mirror 1: 0xe883567 Mirror 3: 0xe883567 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x362c6: skipped 000044 1784093590.693516 read 0x02ee9e thru 0x0362c5 (0x7428 bytes) 000047 1784093590.698046 mapread 0x02e992 thru 0x0362c5 (0x7934 bytes) 000049 1784093590.699183 write 0x03183a thru 0x03fbe7 (0xe3ae bytes) 000052 1784093590.731735 trunc from 0x03fbe8 to 0x010e82 (0x2ed65 bytes) 000053 1784093590.747215 write 0x018a2b thru 0x01bc3a (0x3210 bytes) 000055 1784093590.748759 write 0x02f216 thru 0x030e5a (0x1c45 bytes) 000057 1784093590.749965 mapwrite 0x003cb5 thru 0x00c326 (0x8672 bytes) 000059 1784093590.781433 write 0x0399da thru 0x03ffff (0x6626 bytes) 000063 1784093590.783897 mapread 0x0144e9 thru 0x01530c (0xe24 bytes) 000064 1784093590.789905 read 0x029d0d thru 0x0317a5 (0x7a99 bytes) 000066 1784093590.795621 write 0x0287b1 thru 0x03363d (0xae8d bytes) 000070 1784093590.798462 mapwrite 0x016a00 thru 0x01ce88 (0x6489 bytes) 000071 1784093590.821179 trunc from 0x040000 to 0x0232c2 (0x1cd3d bytes) 000073 1784093590.832994 read 0x01c238 thru 0x020405 (0x41ce bytes) 000074 1784093590.833709 trunc from 0x0232c3 to 0x00313c (0x20186 bytes) 000075 1784093590.842829 mapwrite 0x026a5e thru 0x035c29 (0xf1cc bytes) 000077 1784093590.894518 mapread 0x012a96 thru 0x01a4fd (0x7a68 bytes) 000080 1784093590.897733 read 0x02a937 thru 0x02cf7d (0x2647 bytes) 000081 1784093590.898462 mapwrite 0x010499 thru 0x013658 (0x31c0 bytes) 000083 1784093590.929066 read 0x02bfa5 thru 0x02f9a0 (0x39fc bytes) 000086 1784093590.932050 mapread 0x01f3a4 thru 0x023902 (0x455f bytes) 000087 1784093590.935575 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35c2a): Mirror 3: 0xca19b1c7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35c2a): Mirror 3: 0xca19b1c7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35c2a: skipped 000088 1784093591.073967 read 0x02a624 thru 0x035c29 (0xb606 bytes) 000089 1784093591.079858 read 0x02ebe7 thru 0x035c29 (0x7043 bytes) 000090 1784093591.080508 read 0x005207 thru 0x012c43 (0xda3d bytes) 000092 1784093591.087929 mapread 0x018294 thru 0x022d4a (0xaab7 bytes) 000095 1784093591.090259 read 0x0285e1 thru 0x033ba0 (0xb5c0 bytes) 000096 1784093591.091182 mapread 0x02218f thru 0x0223c6 (0x238 bytes) 000097 1784093591.092187 mapwrite 0x0168b7 thru 0x01ee8b (0x85d5 bytes) 000098 1784093591.116059 read 0x02c3b6 thru 0x02d6e5 (0x1330 bytes) 000099 1784093591.116472 trunc from 0x035c2a to 0x0379db (0x1db2 bytes) 000100 1784093591.127566 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x379dc): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x379dc): Mirror 3: 0x97d18407 (stale) CRC-32 checksum value for chunk [0, 0x379dc): Mirror 1: 0x694c0557 Mirror 4: 0x694c0557 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x379dc: skipped 000101 1784093591.483677 MIRROR_RESYNC 3 mirrors 000103 1784093591.732315 MIRROR_RESYNC 3 mirrors 000104 1784093591.906035 mapread 0x017162 thru 0x01ef5b (0x7dfa bytes) 000106 1784093591.915990 mapwrite 0x0360fd thru 0x03de68 (0x7d6c bytes) 000107 1784093592.001843 trunc from 0x03de69 to 0x032202 (0xbc66 bytes) 000108 1784093592.014965 read 0x00b874 thru 0x00eeae (0x363b bytes) 000109 1784093592.019979 mapwrite 0x000a85 thru 0x00502b (0x45a7 bytes) 000110 1784093592.050595 mapwrite 0x01ce67 thru 0x025a50 (0x8bea bytes) 000111 1784093592.092454 trunc from 0x032203 to 0x01d80d (0x149f5 bytes) 000116 1784093592.104286 mapwrite 0x005d04 thru 0x014d89 (0xf086 bytes) 000117 1784093592.140823 read 0x01180c thru 0x011d10 (0x505 bytes) 000119 1784093592.141681 mapwrite 0x0325e8 thru 0x03ab81 (0x859a bytes) 000120 1784093592.190394 read 0x02ae34 thru 0x030c25 (0x5df2 bytes) 000121 1784093592.191280 mapread 0x005350 thru 0x013d7a (0xea2b bytes) 000122 1784093592.192592 write 0x00e907 thru 0x016c11 (0x830b bytes) 000123 1784093592.193609 mapwrite 0x02a7f1 thru 0x02a81b (0x2b bytes) 000126 1784093592.222424 mapread 0x02fef1 thru 0x037d3a (0x7e4a bytes) 000127 1784093592.223967 mapwrite 0x015c1d thru 0x01a378 (0x475c bytes) 000128 1784093592.252196 trunc from 0x03ab82 to 0x01606a (0x24b17 bytes) 000129 1784093592.266817 trunc from 0x01606b to 0x02b587 (0x1551d bytes) 000130 1784093592.280008 MIRROR_RESYNC 3 mirrors 000131 1784093592.579649 mapread 0x018ce3 thru 0x022cfa (0xa018 bytes) 000132 1784093592.597951 read 0x01c209 thru 0x029756 (0xd54e bytes) 000133 1784093592.603216 read 0x006f5d thru 0x0144fe (0xd5a2 bytes) 000134 1784093592.609971 mapread 0x003f42 thru 0x012a73 (0xeb32 bytes) 000137 1784093592.611345 mapread 0x02295d thru 0x02b587 (0x8c2b bytes) 000138 1784093592.612141 write 0x030a5b thru 0x033a9f (0x3045 bytes) 000139 1784093592.648888 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x33aa0): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33aa0): Mirror 3: 0x828ab946 (stale) Mirror 4: 0x828ab946 (stale) CRC-32 checksum value for chunk [0, 0x33aa0): Mirror 1: 0xb820580b Mirror 5: 0xb820580b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33aa0: skipped 000140 1784093593.011347 write 0x008a0b thru 0x0185bf (0xfbb5 bytes) 000142 1784093593.058592 write 0x02b477 thru 0x03aed9 (0xfa63 bytes) 000143 1784093593.063981 mapwrite 0x00eda4 thru 0x015e01 (0x705e bytes) 000144 1784093593.090757 mapread 0x006d77 thru 0x007f29 (0x11b3 bytes) 000145 1784093593.098176 mapwrite 0x014dc5 thru 0x01a014 (0x5250 bytes) 000146 1784093593.126609 trunc from 0x03aeda to 0x006e72 (0x34067 bytes) truncating to largest ever: 0x3c620 000149 1784093593.149859 trunc from 0x006e73 to 0x03c61f (0x357ad bytes) 000151 1784093593.160248 trunc from 0x03c620 to 0x022ba7 (0x19a78 bytes) 000152 1784093593.171727 write 0x02ba62 thru 0x030fed (0x558c bytes) 000154 1784093593.173299 read 0x00f83a thru 0x01a4c2 (0xac89 bytes) 000155 1784093593.181406 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x30fee): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30fee): Mirror 3: 0x90510b6a (stale) Mirror 4: 0x90510b6a (stale) Mirror 5: 0x9db6125e (stale) CRC-32 checksum value for chunk [0, 0x30fee): Mirror 1: 0x60297554 Mirror 6: 0x60297554 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30fee: skipped 000156 1784093593.548533 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x30fee): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30fee): Mirror 3: 0x90510b6a (stale) Mirror 4: 0x90510b6a (stale) Mirror 5: 0x9db6125e (stale) CRC-32 checksum value for chunk [0, 0x30fee): Mirror 1: 0x60297554 Mirror 6: 0x60297554 Mirror 7: 0x60297554 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30fee: skipped 000157 1784093593.941572 read 0x0165b7 thru 0x0187f8 (0x2242 bytes) 000158 1784093593.951815 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x30fee): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30fee): Mirror 3: 0x90510b6a (stale) Mirror 4: 0x90510b6a (stale) Mirror 5: 0x9db6125e (stale) CRC-32 checksum value for chunk [0, 0x30fee): Mirror 1: 0x60297554 Mirror 6: 0x60297554 Mirror 7: 0x60297554 Mirror 8: 0x60297554 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30fee: skipped 000159 1784093594.402299 write 0x0295bc thru 0x02fa29 (0x646e bytes) 000161 1784093594.459582 mapread 0x0082f3 thru 0x0090e3 (0xdf1 bytes) 000163 1784093594.466565 read 0x0071ad thru 0x007aa3 (0x8f7 bytes) 000167 1784093594.472575 read 0x023f47 thru 0x02876d (0x4827 bytes) 000168 1784093594.479143 write 0x008593 thru 0x017df6 (0xf864 bytes) 000171 1784093594.484589 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30fee): Mirror 3: 0x90510b6a (stale) Mirror 4: 0x90510b6a (stale) Mirror 6: 0x60297554 (stale) Mirror 7: 0x60297554 (stale) Mirror 8: 0x60297554 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30fee): Mirror 3: 0x90510b6a (stale) Mirror 4: 0x90510b6a (stale) Mirror 6: 0x60297554 (stale) Mirror 7: 0x60297554 (stale) Mirror 8: 0x60297554 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30fee: skipped 000174 1784093594.720360 mapread 0x00e7c1 thru 0x01e253 (0xfa93 bytes) 000176 1784093594.745555 mapwrite 0x030511 thru 0x03ffff (0xfaef bytes) 000177 1784093594.819223 MIRROR_RESYNC 6 mirrors 000178 1784093595.432214 trunc from 0x040000 to 0x02bda8 (0x14257 bytes) 000179 1784093595.501879 MIRROR_RESYNC 6 mirrors 000181 1784093596.065965 write 0x030ed3 thru 0x032657 (0x1785 bytes) 000183 1784093596.109261 trunc from 0x032658 to 0x008a3e (0x29c19 bytes) 000184 1784093596.127173 mapread 0x005255 thru 0x007ed0 (0x2c7c bytes) 000186 1784093596.146524 read 0x0020f0 thru 0x008a3e (0x694f bytes) 000188 1784093596.148715 mapread 0x003702 thru 0x0062ed (0x2bec bytes) 000190 1784093596.152037 read 0x00174d thru 0x008a3e (0x72f2 bytes) 000191 1784093596.153120 mapwrite 0x02f2d3 thru 0x036be7 (0x7915 bytes) 000192 1784093596.216775 MIRROR_RESYNC 6 mirrors 000193 1784093596.685870 mapwrite 0x007ae7 thru 0x00e6df (0x6bf9 bytes) 000194 1784093596.768660 write 0x02e7f2 thru 0x03d568 (0xed77 bytes) 000195 1784093596.773417 trunc from 0x03d569 to 0x032bbf (0xa9a9 bytes) 000198 1784093596.803230 MIRROR_RESYNC 6 mirrors 000199 1784093597.239233 write 0x0174df thru 0x024f87 (0xdaa9 bytes) 000201 1784093597.271270 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x32bc0): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32bc0): Mirror 3: 0xc308c32 (stale) Mirror 4: 0xc308c32 (stale) Mirror 6: 0xc308c32 (stale) Mirror 7: 0xc308c32 (stale) Mirror 8: 0xc308c32 (stale) CRC-32 checksum value for chunk [0, 0x32bc0): Mirror 1: 0xc7b57d6 Mirror 9: 0xc7b57d6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32bc0: skipped 000203 1784093597.698846 mapwrite 0x001b7b thru 0x00d873 (0xbcf9 bytes) 000204 1784093597.808178 trunc from 0x032bc0 to 0x02aa19 (0x81a6 bytes) 000206 1784093597.821963 write 0x017d4c thru 0x019e5c (0x2111 bytes) 000208 1784093597.827974 trunc from 0x02aa1a to 0x02caf3 (0x20da bytes) 000209 1784093597.837035 read 0x020793 thru 0x02caf3 (0xc361 bytes) 000213 1784093597.845669 write 0x039e0c thru 0x03ab30 (0xd25 bytes) 000214 1784093597.847002 trunc from 0x03ab31 to 0x00909e (0x31a92 bytes) 000216 1784093597.857237 read 0x003f35 thru 0x00909e (0x516a bytes) 000219 1784093597.859426 trunc from 0x00909f to 0x0088c8 (0x7d6 bytes) 000221 1784093597.868772 trunc from 0x0088c9 to 0x026a9f (0x1e1d7 bytes) 000226 1784093597.879984 mapwrite 0x02313d thru 0x024d25 (0x1be9 bytes) 000227 1784093597.907061 mapwrite 0x0102a4 thru 0x01765e (0x73bb bytes) 000229 1784093597.934143 mapwrite 0x004de5 thru 0x008219 (0x3435 bytes) 000232 1784093597.954998 read 0x011c71 thru 0x016ae5 (0x4e75 bytes) 000233 1784093597.955651 mapread 0x003ade thru 0x010916 (0xce39 bytes) 000235 1784093597.957168 mapwrite 0x01e6aa thru 0x02e4db (0xfe32 bytes) 000236 1784093597.995021 mapwrite 0x00d7a3 thru 0x012738 (0x4f96 bytes) 000238 1784093598.020418 write 0x01ddf5 thru 0x01e31d (0x529 bytes) 000242 1784093598.023290 trunc from 0x02e4dc to 0x039314 (0xae39 bytes) 000243 1784093598.032638 trunc from 0x039315 to 0x03b16b (0x1e57 bytes) 000244 1784093598.043881 mapread 0x011bed thru 0x0134b4 (0x18c8 bytes) 000247 1784093598.046277 read 0x02702e thru 0x029cfb (0x2cce bytes) 000248 1784093598.050457 mapwrite 0x01dc13 thru 0x021154 (0x3542 bytes) 000249 1784093598.074622 mapread 0x00405f thru 0x009826 (0x57c8 bytes) 000250 1784093598.075624 trunc from 0x03b16c to 0x01b494 (0x1fcd7 bytes) 000251 1784093598.083537 read 0x011ed1 thru 0x01271f (0x84f bytes) 000253 1784093598.084576 mapread 0x00d9df thru 0x01ab3e (0xd160 bytes) 000255 1784093598.086141 trunc from 0x01b495 to 0x01b033 (0x461 bytes) 000257 1784093598.095477 mapwrite 0x017da9 thru 0x0217c8 (0x9a20 bytes) 000262 1784093598.140089 write 0x02db1e thru 0x03218e (0x4671 bytes) 000263 1784093598.141510 mapwrite 0x026bbc thru 0x02a501 (0x3946 bytes) 000264 1784093598.179149 write 0x003011 thru 0x00962d (0x661d bytes) 000265 1784093598.183348 MIRROR_RESYNC 7 mirrors 000266 1784093598.669635 MIRROR_RESYNC 7 mirrors 000267 1784093598.883169 trunc from 0x03218f to 0x02afcb (0x71c3 bytes) 000269 1784093598.933698 mapwrite 0x007bdb thru 0x01478a (0xcbb0 bytes) 000271 1784093598.985935 MIRROR_RESYNC 7 mirrors 000276 1784093599.482100 mapread 0x006f8d thru 0x00e9ee (0x7a62 bytes) 000277 1784093599.493634 trunc from 0x02afcc to 0x00db8f (0x1d43c bytes) 000278 1784093599.538125 mapread 0x00a985 thru 0x00ad2e (0x3aa bytes) 000280 1784093599.544345 read 0x0045f9 thru 0x00db8f (0x9597 bytes) 000285 1784093599.547634 mapread 0x008609 thru 0x00c11d (0x3b15 bytes) 000286 1784093599.548542 trunc from 0x00db90 to 0x026711 (0x18b82 bytes) 000287 1784093599.562228 mapread 0x004012 thru 0x00c94f (0x893e bytes) 000288 1784093599.563218 write 0x033484 thru 0x03ffff (0xcb7c bytes) 000289 1784093599.564516 mapread 0x03a6d1 thru 0x03ffff (0x592f bytes) 000291 1784093599.565924 trunc from 0x040000 to 0x02cbf5 (0x1340a bytes) 000293 1784093599.574847 write 0x03e036 thru 0x03ffff (0x1fca bytes) 000294 1784093599.575806 mapwrite 0x017073 thru 0x01857b (0x1509 bytes) 000295 1784093599.608535 write 0x015d95 thru 0x01e803 (0x8a6f bytes) 000296 1784093599.609844 mapwrite 0x0283cd thru 0x034285 (0xbeb9 bytes) 000297 1784093599.647445 mapread 0x011872 thru 0x013199 (0x1928 bytes) 000298 1784093599.651415 MIRROR_RESYNC 7 mirrors 000299 1784093600.211940 trunc from 0x040000 to 0x004a49 (0x3b5b6 bytes) 000300 1784093600.243590 write 0x022afd thru 0x030428 (0xd92c bytes) 000301 1784093600.245414 mapwrite 0x013329 thru 0x01508a (0x1d62 bytes) 000302 1784093600.280096 write 0x001d28 thru 0x0023d0 (0x6a9 bytes) 000303 1784093600.285958 trunc from 0x030429 to 0x013831 (0x1cbf7 bytes) 000304 1784093600.299977 write 0x0188da thru 0x019d42 (0x1469 bytes) 000307 1784093600.302009 read 0x012698 thru 0x018188 (0x5af1 bytes) 000308 1784093600.306838 read 0x019392 thru 0x019d42 (0x9b1 bytes) 000309 1784093600.307460 write 0x03929a thru 0x03ffff (0x6d66 bytes) 000311 1784093600.309331 mapwrite 0x015ee1 thru 0x015f84 (0xa4 bytes) 000312 1784093600.337619 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xef57242a (stale) Mirror 4: 0xef57242a (stale) Mirror 6: 0xef57242a (stale) Mirror 8: 0xef57242a (stale) Mirror 9: 0xef57242a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xef57242a (stale) Mirror 4: 0xef57242a (stale) Mirror 6: 0xef57242a (stale) Mirror 8: 0xef57242a (stale) Mirror 9: 0xef57242a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000315 1784093600.573893 mapread 0x00f0fd thru 0x013f5f (0x4e63 bytes) 000316 1784093600.590771 mapread 0x00ec10 thru 0x01c45c (0xd84d bytes) 000317 1784093600.600360 read 0x01b175 thru 0x024c3d (0x9ac9 bytes) 000318 1784093600.606224 write 0x02d369 thru 0x02e265 (0xefd bytes) 000319 1784093600.612618 mapread 0x019ca5 thru 0x025c8e (0xbfea bytes) 000320 1784093600.618043 trunc from 0x040000 to 0x038428 (0x7bd7 bytes) 000321 1784093600.628101 trunc from 0x038429 to 0x017924 (0x20b04 bytes) 000322 1784093600.640110 write 0x02431f thru 0x02ad8c (0x6a6e bytes) 000323 1784093600.641485 trunc from 0x02ad8d to 0x001541 (0x2984b bytes) 000325 1784093600.653091 write 0x0112e6 thru 0x0132a4 (0x1fbf bytes) 000326 1784093600.654158 read 0x00a548 thru 0x00bb99 (0x1652 bytes) 000328 1784093600.660072 read 0x0088c1 thru 0x0132a4 (0xa9e4 bytes) 000329 1784093600.663716 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x132a5): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x132a5): Mirror 3: 0xe6119b42 (stale) Mirror 4: 0xe6119b42 (stale) Mirror 6: 0xe6119b42 (stale) Mirror 8: 0xe6119b42 (stale) Mirror 9: 0xe6119b42 (stale) CRC-32 checksum value for chunk [0, 0x132a5): Mirror 1: 0xf508684b Mirror 10: 0xf508684b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x132a5: skipped 000330 1784093601.042856 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x132a5): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x132a5): Mirror 3: 0xe6119b42 (stale) Mirror 4: 0xe6119b42 (stale) Mirror 6: 0xe6119b42 (stale) Mirror 8: 0xe6119b42 (stale) Mirror 9: 0xe6119b42 (stale) CRC-32 checksum value for chunk [0, 0x132a5): Mirror 1: 0xf508684b Mirror 10: 0xf508684b Mirror 11: 0xf508684b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x132a5: skipped 000332 1784093601.405615 mapread 0x000e1b thru 0x006123 (0x5309 bytes) 000334 1784093601.427679 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x132a5): Mirror 3: 0xe6119b42 (stale) Mirror 4: 0xe6119b42 (stale) Mirror 8: 0xe6119b42 (stale) Mirror 9: 0xe6119b42 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x132a5): Mirror 3: 0xe6119b42 (stale) Mirror 4: 0xe6119b42 (stale) Mirror 8: 0xe6119b42 (stale) Mirror 9: 0xe6119b42 (stale) CRC-32 checksum value for chunk [0, 0x132a5): Mirror 1: 0xf508684b Mirror 10: 0xf508684b Mirror 11: 0xf508684b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x132a5: skipped 000335 1784093601.662187 mapread 0x00434d thru 0x008554 (0x4208 bytes) 000336 1784093601.672480 mapread 0x009a4f thru 0x0132a4 (0x9856 bytes) 000337 1784093601.678557 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x132a5): Mirror 3: 0xe6119b42 (stale) Mirror 4: 0xe6119b42 (stale) Mirror 8: 0xe6119b42 (stale) Mirror 9: 0xe6119b42 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x132a5): Mirror 3: 0xe6119b42 (stale) Mirror 4: 0xe6119b42 (stale) Mirror 8: 0xe6119b42 (stale) Mirror 9: 0xe6119b42 (stale) CRC-32 checksum value for chunk [0, 0x132a5): Mirror 1: 0xf508684b Mirror 10: 0xf508684b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x132a5: skipped 000339 1784093601.903095 write 0x02aaba thru 0x02d765 (0x2cac bytes) 000340 1784093601.931096 write 0x0369d3 thru 0x03bb3b (0x5169 bytes) 000342 1784093601.935522 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bb3c): Mirror 4: 0xf9fe1a32 (stale) Mirror 8: 0xf9fe1a32 (stale) Mirror 9: 0xf9fe1a32 (stale) Mirror 10: 0x7702cb6c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb3c): Mirror 4: 0xf9fe1a32 (stale) Mirror 8: 0xf9fe1a32 (stale) Mirror 9: 0xf9fe1a32 (stale) Mirror 10: 0x7702cb6c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb3c: skipped 000345 1784093602.112598 MIRROR_RESYNC 5 mirrors 000346 1784093602.509741 MIRROR_EXTEND to 6 mirrors 000348 1784093602.804538 write 0x012a75 thru 0x01f7f3 (0xcd7f bytes) 000349 1784093602.845826 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bb3c): Mirror 4: 0x435df32f (stale) Mirror 8: 0x435df32f (stale) Mirror 9: 0x435df32f (stale) Mirror 11: 0x435df32f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb3c): Mirror 4: 0x435df32f (stale) Mirror 8: 0x435df32f (stale) Mirror 9: 0x435df32f (stale) Mirror 11: 0x435df32f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb3c: skipped 000353 1784093603.018061 read 0x007002 thru 0x010296 (0x9295 bytes) 000354 1784093603.025886 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3bb3c): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb3c): Mirror 4: 0x435df32f (stale) Mirror 8: 0x435df32f (stale) Mirror 9: 0x435df32f (stale) Mirror 11: 0x435df32f (stale) CRC-32 checksum value for chunk [0, 0x3bb3c): Mirror 1: 0x82fd584a Mirror 12: 0x82fd584a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb3c: skipped 000355 1784093603.424612 mapread 0x01287b thru 0x0211ac (0xe932 bytes) 000357 1784093603.457109 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3bb3c): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb3c): Mirror 4: 0x435df32f (stale) Mirror 8: 0x435df32f (stale) Mirror 9: 0x435df32f (stale) Mirror 11: 0x435df32f (stale) CRC-32 checksum value for chunk [0, 0x3bb3c): Mirror 1: 0x82fd584a Mirror 12: 0x82fd584a Mirror 13: 0x82fd584a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb3c: skipped 000358 1784093603.823835 read 0x030e0a thru 0x03bb3b (0xad32 bytes) 000359 1784093603.827917 write 0x003ca0 thru 0x012e74 (0xf1d5 bytes) 000360 1784093603.859324 MIRROR_RESYNC 7 mirrors 000361 1784093604.316554 trunc from 0x03bb3c to 0x0215ff (0x1a53c bytes) 000363 1784093604.347139 mapread 0x01be51 thru 0x0215ff (0x57af bytes) 000364 1784093604.356605 read 0x01f938 thru 0x0215ff (0x1cc8 bytes) 000365 1784093604.357203 trunc from 0x021600 to 0x034031 (0x12a32 bytes) 000366 1784093604.370478 mapwrite 0x000edd thru 0x00572c (0x4850 bytes) 000368 1784093604.398524 read 0x015f81 thru 0x01da89 (0x7b09 bytes) 000369 1784093604.404653 mapwrite 0x0327a9 thru 0x03791b (0x5173 bytes) 000370 1784093604.438560 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3791c): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3791c): Mirror 4: 0xd17b5e71 (stale) Mirror 8: 0xd17b5e71 (stale) Mirror 9: 0xd17b5e71 (stale) Mirror 11: 0xd17b5e71 (stale) Mirror 12: 0xd17b5e71 (stale) Mirror 13: 0xd17b5e71 (stale) CRC-32 checksum value for chunk [0, 0x3791c): Mirror 1: 0x94e7a0a2 Mirror 14: 0x94e7a0a2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3791c: skipped 000373 1784093604.825885 write 0x00eff4 thru 0x01df65 (0xef72 bytes) 000375 1784093604.865356 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3791c): Mirror 4: 0xd17b5e71 (stale) Mirror 8: 0xd17b5e71 (stale) Mirror 11: 0xd17b5e71 (stale) Mirror 12: 0xd17b5e71 (stale) Mirror 13: 0xd17b5e71 (stale) Mirror 14: 0x94e7a0a2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3791c): Mirror 4: 0xd17b5e71 (stale) Mirror 8: 0xd17b5e71 (stale) Mirror 11: 0xd17b5e71 (stale) Mirror 12: 0xd17b5e71 (stale) Mirror 13: 0xd17b5e71 (stale) Mirror 14: 0x94e7a0a2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3791c: skipped 000376 1784093605.069247 mapread 0x026479 thru 0x02fd0d (0x9895 bytes) 000377 1784093605.081486 trunc from 0x03791c to 0x003e09 (0x33b12 bytes) 000378 1784093605.099452 mapread 0x001e15 thru 0x003e09 (0x1ff5 bytes) 000380 1784093605.105535 write 0x004d2d thru 0x00e6bb (0x998f bytes) 000381 1784093605.107014 mapread 0x006ac3 thru 0x00e6bb (0x7bf9 bytes) 000382 1784093605.108125 mapread 0x00b428 thru 0x00e6bb (0x3294 bytes) 000383 1784093605.109124 write 0x005d58 thru 0x010c36 (0xaedf bytes) 000384 1784093605.110670 mapwrite 0x022791 thru 0x029797 (0x7007 bytes) 000385 1784093605.149858 mapread 0x00b80f thru 0x00c0f6 (0x8e8 bytes) 000386 1784093605.150949 write 0x02c637 thru 0x03a793 (0xe15d bytes) 000387 1784093605.152698 read 0x01d54f thru 0x02b2e6 (0xdd98 bytes) 000393 1784093605.158787 write 0x01af2e thru 0x02a897 (0xf96a bytes) 000394 1784093605.163159 write 0x031365 thru 0x038ab6 (0x7752 bytes) 000397 1784093605.169279 MIRROR_RESYNC 7 mirrors 000399 1784093605.669970 write 0x032765 thru 0x03be22 (0x96be bytes) 000401 1784093605.705588 mapwrite 0x0251ec thru 0x031e1a (0xcc2f bytes) 000402 1784093605.739830 trunc from 0x03be23 to 0x00621f (0x35c03 bytes) 000405 1784093605.758485 trunc from 0x006220 to 0x01f7fd (0x195de bytes) 000406 1784093605.770500 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f7fe): Mirror 4: 0x42bc49a5 (stale) Mirror 8: 0x42bc49a5 (stale) Mirror 11: 0x42bc49a5 (stale) Mirror 12: 0x42bc49a5 (stale) Mirror 14: 0x42bc49a5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f7fe): Mirror 4: 0x42bc49a5 (stale) Mirror 8: 0x42bc49a5 (stale) Mirror 11: 0x42bc49a5 (stale) Mirror 12: 0x42bc49a5 (stale) Mirror 14: 0x42bc49a5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f7fe: skipped 000408 1784093605.984855 mapwrite 0x01a9c9 thru 0x01f4cc (0x4b04 bytes) 000410 1784093606.025848 write 0x030c8f thru 0x0338d7 (0x2c49 bytes) 000411 1784093606.027051 trunc from 0x0338d8 to 0x0033d0 (0x30507 bytes) 000412 1784093606.035557 read 0x00233d thru 0x0033d0 (0x1094 bytes) 000413 1784093606.039191 mapread 0x0006b1 thru 0x0033d0 (0x2d20 bytes) 000416 1784093606.043706 read 0x0008f0 thru 0x0033d0 (0x2ae1 bytes) 000417 1784093606.047201 MIRROR_RESYNC 6 mirrors 000422 1784093606.446571 trunc from 0x0033d1 to 0x031164 (0x2dd94 bytes) 000423 1784093606.491958 MIRROR_RESYNC 6 mirrors 000425 1784093606.925677 trunc from 0x031165 to 0x0319a8 (0x844 bytes) 000427 1784093606.973740 read 0x012870 thru 0x01fa87 (0xd218 bytes) 000429 1784093606.979808 trunc from 0x0319a9 to 0x024f21 (0xca87 bytes) 000431 1784093606.991196 mapread 0x01ca10 thru 0x024f21 (0x8512 bytes) 000432 1784093607.001645 mapread 0x0077cb thru 0x007852 (0x88 bytes) 000433 1784093607.007279 trunc from 0x024f22 to 0x00bbc1 (0x19360 bytes) truncating to largest ever: 0x3e996 000434 1784093607.019443 trunc from 0x00bbc2 to 0x03e995 (0x32dd4 bytes) 000435 1784093607.029474 write 0x020429 thru 0x0291c6 (0x8d9e bytes) 000436 1784093607.036441 mapwrite 0x008176 thru 0x00e77a (0x6605 bytes) 000438 1784093607.071820 read 0x03026a thru 0x030706 (0x49d bytes) 000439 1784093607.077645 read 0x03e414 thru 0x03e995 (0x582 bytes) 000440 1784093607.081195 MIRROR_RESYNC 6 mirrors 000441 1784093607.511773 write 0x004bc6 thru 0x009d0d (0x5148 bytes) 000443 1784093607.542264 mapread 0x03b161 thru 0x03b76f (0x60f bytes) 000446 1784093607.547125 trunc from 0x03e996 to 0x024d4a (0x19c4b bytes) 000447 1784093607.564663 read 0x019ce5 thru 0x01e782 (0x4a9e bytes) 000448 1784093607.569539 write 0x001c57 thru 0x00e005 (0xc3af bytes) 000449 1784093607.575313 read 0x0077fc thru 0x01730a (0xfb0f bytes) 000450 1784093607.583606 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x24d4b): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24d4b): Mirror 4: 0x69d46a67 (stale) Mirror 8: 0x69d46a67 (stale) Mirror 11: 0x69d46a67 (stale) Mirror 12: 0x69d46a67 (stale) Mirror 14: 0x69d46a67 (stale) CRC-32 checksum value for chunk [0, 0x24d4b): Mirror 1: 0x712e4e41 Mirror 15: 0x712e4e41 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24d4b: skipped 000451 1784093607.948656 write 0x005600 thru 0x006240 (0xc41 bytes) 000452 1784093607.979600 mapread 0x003f5f thru 0x013abd (0xfb5f bytes) 000453 1784093607.995425 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x24d4b): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24d4b): Mirror 4: 0x69d46a67 (stale) Mirror 8: 0x69d46a67 (stale) Mirror 11: 0x69d46a67 (stale) Mirror 12: 0x69d46a67 (stale) Mirror 14: 0x69d46a67 (stale) Mirror 15: 0x712e4e41 (stale) CRC-32 checksum value for chunk [0, 0x24d4b): Mirror 1: 0x94332759 Mirror 16: 0x94332759 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24d4b: skipped 000457 1784093608.389117 trunc from 0x024d4b to 0x017849 (0xd501 bytes) 000459 1784093608.421722 read 0x006542 thru 0x0098a0 (0x335f bytes) 000460 1784093608.427949 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1784a): Mirror 4: 0xb38b87c4 (stale) Mirror 8: 0xb38b87c4 (stale) Mirror 11: 0xb38b87c4 (stale) Mirror 12: 0xb38b87c4 (stale) Mirror 14: 0xb38b87c4 (stale) Mirror 15: 0x51f17046 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1784a): Mirror 4: 0xb38b87c4 (stale) Mirror 8: 0xb38b87c4 (stale) Mirror 11: 0xb38b87c4 (stale) Mirror 12: 0xb38b87c4 (stale) Mirror 14: 0xb38b87c4 (stale) Mirror 15: 0x51f17046 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1784a: skipped 000464 1784093608.617787 write 0x0187a5 thru 0x0222fe (0x9b5a bytes) 000466 1784093608.619758 write 0x00505b thru 0x013f2c (0xeed2 bytes) 000469 1784093608.626464 write 0x010092 thru 0x018519 (0x8488 bytes) 000470 1784093608.627922 read 0x01b9fc thru 0x01ea43 (0x3048 bytes) 000471 1784093608.628532 read 0x017936 thru 0x01b85a (0x3f25 bytes) 000472 1784093608.631604 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x222ff): Mirror 4: 0x8ec1dc28 (stale) Mirror 8: 0x8ec1dc28 (stale) Mirror 11: 0x8ec1dc28 (stale) Mirror 12: 0x8ec1dc28 (stale) Mirror 15: 0xa9f14b03 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x222ff): Mirror 4: 0x8ec1dc28 (stale) Mirror 8: 0x8ec1dc28 (stale) Mirror 11: 0x8ec1dc28 (stale) Mirror 12: 0x8ec1dc28 (stale) Mirror 15: 0xa9f14b03 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x222ff: skipped 000473 1784093608.797601 mapread 0x00c0e4 thru 0x01a26a (0xe187 bytes) 000474 1784093608.814315 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x222ff): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x222ff): Mirror 4: 0x8ec1dc28 (stale) Mirror 8: 0x8ec1dc28 (stale) Mirror 11: 0x8ec1dc28 (stale) Mirror 12: 0x8ec1dc28 (stale) Mirror 15: 0xa9f14b03 (stale) CRC-32 checksum value for chunk [0, 0x222ff): Mirror 1: 0x47cd8fa2 Mirror 16: 0x47cd8fa2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x222ff: skipped 000475 1784093609.131184 trunc from 0x0222ff to 0x002bc8 (0x1f736 bytes) 000476 1784093609.174926 mapwrite 0x026e9b thru 0x030e78 (0x9fde bytes) 000478 1784093609.218711 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30e79): Mirror 4: 0xb351b643 (stale) Mirror 11: 0xb351b643 (stale) Mirror 12: 0xb351b643 (stale) Mirror 15: 0xb1477753 (stale) Mirror 16: 0x1745b09c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30e79): Mirror 4: 0xb351b643 (stale) Mirror 11: 0xb351b643 (stale) Mirror 12: 0xb351b643 (stale) Mirror 15: 0xb1477753 (stale) Mirror 16: 0x1745b09c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30e79: skipped 000479 1784093609.386002 mapread 0x01716a thru 0x017f92 (0xe29 bytes) 000480 1784093609.389690 mapread 0x0301ee thru 0x030e78 (0xc8b bytes) 000481 1784093609.393467 mapwrite 0x02c141 thru 0x035958 (0x9818 bytes) 000482 1784093609.435123 write 0x028338 thru 0x0382ee (0xffb7 bytes) 000485 1784093609.437690 write 0x0010f2 thru 0x0106cd (0xf5dc bytes) 000486 1784093609.447148 trunc from 0x0382ef to 0x00f80c (0x28ae2 bytes) 000490 1784093609.456128 mapread 0x00dd7e thru 0x00f80c (0x1a8f bytes) 000491 1784093609.457109 write 0x024f83 thru 0x0341d8 (0xf256 bytes) 000493 1784093609.459242 mapread 0x01a82c thru 0x01fc05 (0x53da bytes) 000494 1784093609.464983 read 0x014574 thru 0x019229 (0x4cb6 bytes) 000495 1784093609.465633 read 0x00142f thru 0x00fdeb (0xe9bd bytes) 000497 1784093609.469463 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x341d9): Mirror 11: 0x2cf8df7e (stale) Mirror 12: 0x2cf8df7e (stale) Mirror 15: 0xa8f3e35 (stale) Mirror 16: 0x18e78584 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x341d9): Mirror 11: 0x2cf8df7e (stale) Mirror 12: 0x2cf8df7e (stale) Mirror 15: 0xa8f3e35 (stale) Mirror 16: 0x18e78584 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x341d9: skipped 000498 1784093609.651961 MIRROR_RESYNC 5 mirrors 000501 1784093610.060291 read 0x01ea9d thru 0x01f7b6 (0xd1a bytes) 000502 1784093610.066118 MIRROR_EXTEND to 6 mirrors 000503 1784093610.370010 mapread 0x030ed7 thru 0x0341d8 (0x3302 bytes) 000505 1784093610.381499 read 0x002c05 thru 0x00e594 (0xb990 bytes) 000507 1784093610.387766 trunc from 0x0341d9 to 0x03dc03 (0x9a2b bytes) 000509 1784093610.432045 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dc04): Mirror 12: 0xfb38b101 (stale) Mirror 15: 0xfb38b101 (stale) Mirror 16: 0xfb38b101 (stale) Mirror 17: 0xfb38b101 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dc04): Mirror 12: 0xfb38b101 (stale) Mirror 15: 0xfb38b101 (stale) Mirror 16: 0xfb38b101 (stale) Mirror 17: 0xfb38b101 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dc04: skipped 000510 1784093610.629507 trunc from 0x03dc04 to 0x030383 (0xd880 bytes) 000511 1784093610.641154 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x30384): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30384): Mirror 12: 0x5f843d6d (stale) Mirror 15: 0x5f843d6d (stale) Mirror 16: 0x5f843d6d (stale) Mirror 17: 0x5f843d6d (stale) CRC-32 checksum value for chunk [0, 0x30384): Mirror 1: 0x5f843d6d Mirror 18: 0x5f843d6d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30384: skipped 000514 1784093611.207431 MIRROR_RESYNC 6 mirrors 000515 1784093611.726610 read 0x0121a3 thru 0x016564 (0x43c2 bytes) 000516 1784093611.732460 write 0x032b22 thru 0x03f0d1 (0xc5b0 bytes) 000518 1784093611.785831 trunc from 0x03f0d2 to 0x008149 (0x36f88 bytes) 000519 1784093611.802737 read 0x0059a6 thru 0x008149 (0x27a4 bytes) 000520 1784093611.811694 mapread 0x004d19 thru 0x008149 (0x3431 bytes) 000521 1784093611.812743 read 0x007211 thru 0x008149 (0xf39 bytes) 000522 1784093611.813200 mapwrite 0x0239ba thru 0x028258 (0x489f bytes) 000523 1784093611.860086 mapread 0x003bf8 thru 0x00a055 (0x645e bytes) 000524 1784093611.861473 mapread 0x0209a4 thru 0x0212fa (0x957 bytes) 000533 1784093611.866844 read 0x013df6 thru 0x019881 (0x5a8c bytes) 000534 1784093611.870201 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28259): Mirror 12: 0x1f32dabd (stale) Mirror 15: 0x1f32dabd (stale) Mirror 16: 0x1f32dabd (stale) Mirror 17: 0x1f32dabd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28259): Mirror 12: 0x1f32dabd (stale) Mirror 15: 0x1f32dabd (stale) Mirror 16: 0x1f32dabd (stale) Mirror 17: 0x1f32dabd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28259: skipped 000535 1784093612.064189 write 0x01ca00 thru 0x01fe4c (0x344d bytes) 000537 1784093612.072295 read 0x002f52 thru 0x0033d8 (0x487 bytes) 000538 1784093612.077952 read 0x01df89 thru 0x025190 (0x7208 bytes) 000541 1784093612.087920 write 0x01d5bf thru 0x0293d1 (0xbe13 bytes) 000542 1784093612.089718 read 0x01ba0b thru 0x01f7c4 (0x3dba bytes) 000543 1784093612.094789 MIRROR_RESYNC 5 mirrors 000544 1784093612.521133 read 0x0103ce thru 0x01bd25 (0xb958 bytes) 000546 1784093612.526996 trunc from 0x0293d2 to 0x0120c2 (0x1730f bytes) 000547 1784093612.581902 mapread 0x0118ed thru 0x0120c2 (0x7d6 bytes) 000549 1784093612.592848 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x120c3): Mirror 12: 0x65c6ed01 (stale) Mirror 16: 0x65c6ed01 (stale) Mirror 17: 0x65c6ed01 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x120c3): Mirror 12: 0x65c6ed01 (stale) Mirror 16: 0x65c6ed01 (stale) Mirror 17: 0x65c6ed01 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x120c3: skipped 000550 1784093612.799894 mapread 0x009ce4 thru 0x0120c2 (0x83df bytes) 000554 1784093612.815055 read 0x00b50f thru 0x00cd93 (0x1885 bytes) 000556 1784093612.816519 trunc from 0x0120c3 to 0x0279ae (0x158ec bytes) 000557 1784093612.830876 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x279af): Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x279af): Mirror 12: 0xb5d531a9 (stale) Mirror 16: 0xb5d531a9 (stale) Mirror 17: 0xb5d531a9 (stale) CRC-32 checksum value for chunk [0, 0x279af): Mirror 1: 0x2a160e29 Mirror 18: 0x2a160e29 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x279af: skipped 000560 1784093613.259731 mapread 0x01a39b thru 0x026036 (0xbc9c bytes) 000561 1784093613.286891 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x279af): Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x279af): Mirror 12: 0xb5d531a9 (stale) Mirror 16: 0xb5d531a9 (stale) Mirror 17: 0xb5d531a9 (stale) CRC-32 checksum value for chunk [0, 0x279af): Mirror 1: 0x2a160e29 Mirror 18: 0x2a160e29 Mirror 19: 0x2a160e29 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x279af: skipped 000563 1784093613.734420 mapwrite 0x00087d thru 0x00c81d (0xbfa1 bytes) 000565 1784093613.837222 mapread 0x01cd68 thru 0x021801 (0x4a9a bytes) 000566 1784093613.844885 mapwrite 0x03b4b4 thru 0x03d162 (0x1caf bytes) 000567 1784093613.896065 trunc from 0x03d163 to 0x01b446 (0x21d1c bytes) 000568 1784093613.910857 trunc from 0x01b447 to 0x036165 (0x1ad1f bytes) 000569 1784093613.922158 mapwrite 0x01ff15 thru 0x02e4e3 (0xe5cf bytes) 000570 1784093613.966110 read 0x0235df thru 0x029120 (0x5b42 bytes) 000572 1784093613.967474 read 0x011e6c thru 0x01c58f (0xa724 bytes) 000573 1784093613.973797 read 0x01f51a thru 0x027f32 (0x8a19 bytes) 000574 1784093613.979121 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x36166): Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36166): Mirror 12: 0x7c2101f8 (stale) Mirror 16: 0x7c2101f8 (stale) Mirror 17: 0x7c2101f8 (stale) Mirror 18: 0xa3bc25c4 (stale) Mirror 19: 0xa3bc25c4 (stale) CRC-32 checksum value for chunk [0, 0x36166): Mirror 1: 0x566a8209 Mirror 20: 0x566a8209 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36166: skipped 000575 1784093614.458808 mapread 0x00a34b thru 0x0171ac (0xce62 bytes) 000577 1784093614.507873 write 0x01a9de thru 0x024eaf (0xa4d2 bytes) 000578 1784093614.574572 mapread 0x006c7c thru 0x0113fa (0xa77f bytes) 000582 1784093614.587808 write 0x015c85 thru 0x024642 (0xe9be bytes) 000584 1784093614.589778 trunc from 0x036166 to 0x01ac3c (0x1b529 bytes) 000585 1784093614.607215 write 0x005cf6 thru 0x00fd7f (0xa08a bytes) 000586 1784093614.609668 write 0x02b438 thru 0x0328ec (0x74b5 bytes) 000588 1784093614.612753 mapwrite 0x000840 thru 0x007d87 (0x7548 bytes) 000589 1784093614.665518 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x328ed): Mirror 16: 0x83557452 (stale) Mirror 17: 0x83557452 (stale) Mirror 18: 0x5ef8199e (stale) Mirror 19: 0x5ef8199e (stale) Mirror 20: 0xfe2898b2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x328ed): Mirror 16: 0x83557452 (stale) Mirror 17: 0x83557452 (stale) Mirror 18: 0x5ef8199e (stale) Mirror 19: 0x5ef8199e (stale) Mirror 20: 0xfe2898b2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x328ed: skipped 000590 1784093614.893310 mapread 0x004168 thru 0x012dcf (0xec68 bytes) 000592 1784093614.922876 mapwrite 0x037813 thru 0x03ffff (0x87ed bytes) 000595 1784093614.972093 trunc from 0x040000 to 0x039216 (0x6de9 bytes) 000596 1784093614.985571 read 0x0138c4 thru 0x016fe9 (0x3726 bytes) 000598 1784093614.986782 mapread 0x0362c7 thru 0x039216 (0x2f50 bytes) 000599 1784093614.987818 mapread 0x00930f thru 0x00ac8f (0x1981 bytes) 000601 1784093614.989037 read 0x0387ac thru 0x039216 (0xa6b bytes) 000603 1784093614.990100 trunc from 0x039217 to 0x0189bc (0x2085a bytes) 000604 1784093614.999462 read 0x007be7 thru 0x00d27b (0x5695 bytes) 000605 1784093615.000234 trunc from 0x0189bd to 0x03b946 (0x22f8a bytes) 000607 1784093615.011698 read 0x007f9f thru 0x014d9f (0xce01 bytes) 000608 1784093615.013203 read 0x008d5f thru 0x012614 (0x98b6 bytes) 000610 1784093615.014825 read 0x0233c8 thru 0x02bcf2 (0x892b bytes) 000613 1784093615.024351 mapwrite 0x03c9dc thru 0x03ffff (0x3624 bytes) 000615 1784093615.074554 write 0x0004aa thru 0x001a58 (0x15af bytes) 000616 1784093615.075620 trunc from 0x040000 to 0x02b079 (0x14f86 bytes) 000618 1784093615.091497 MIRROR_RESYNC 6 mirrors 000622 1784093615.615535 trunc from 0x02b07a to 0x0247d5 (0x68a4 bytes) 000624 1784093615.678646 write 0x0198d8 thru 0x029675 (0xfd9e bytes) 000625 1784093615.683920 read 0x01dd34 thru 0x025e03 (0x80d0 bytes) 000628 1784093615.685800 mapwrite 0x032ff8 thru 0x03f892 (0xc89b bytes) 000632 1784093615.760477 mapread 0x0203ac thru 0x0282c6 (0x7f1b bytes) 000635 1784093615.763826 trunc from 0x03f893 to 0x00e570 (0x31322 bytes) 000636 1784093615.779035 mapwrite 0x0171f5 thru 0x02454d (0xd359 bytes) 000637 1784093615.839748 trunc from 0x02454e to 0x0114ad (0x130a0 bytes) 000639 1784093615.848846 mapread 0x00b93c thru 0x00efb3 (0x3678 bytes) 000641 1784093615.859282 MIRROR_RESYNC 6 mirrors 000642 1784093616.327081 mapwrite 0x006b2c thru 0x0108db (0x9db0 bytes) 000643 1784093616.440640 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x114ae): Mirror 17: 0x8740585d (stale) Mirror 18: 0x8740585d (stale) Mirror 19: 0x8740585d (stale) Mirror 20: 0x8740585d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x114ae): Mirror 17: 0x8740585d (stale) Mirror 18: 0x8740585d (stale) Mirror 19: 0x8740585d (stale) Mirror 20: 0x8740585d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x114ae: skipped 000645 1784093616.646202 trunc from 0x0114ae to 0x032e74 (0x219c7 bytes) 000646 1784093616.656648 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32e75): Mirror 18: 0x1c8d6d54 (stale) Mirror 19: 0x1c8d6d54 (stale) Mirror 20: 0x1c8d6d54 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32e75): Mirror 18: 0x1c8d6d54 (stale) Mirror 19: 0x1c8d6d54 (stale) Mirror 20: 0x1c8d6d54 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32e75: skipped 000647 1784093616.827659 trunc from 0x032e75 to 0x027eaa (0xafca bytes) 000648 1784093616.842328 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x27eab): Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27eab): Mirror 18: 0x8e8f2f77 (stale) Mirror 19: 0x8e8f2f77 (stale) Mirror 20: 0x8e8f2f77 (stale) CRC-32 checksum value for chunk [0, 0x27eab): Mirror 1: 0x8f52b93a Mirror 21: 0x8f52b93a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27eab: skipped 000649 1784093617.334613 mapwrite 0x0145f5 thru 0x01c386 (0x7d92 bytes) 000651 1784093617.436170 mapread 0x017935 thru 0x025b30 (0xe1fc bytes) 000652 1784093617.443815 trunc from 0x027eab to 0x00a076 (0x1de34 bytes) 000653 1784093617.460299 trunc from 0x00a077 to 0x004014 (0x6062 bytes) 000656 1784093617.476877 MIRROR_RESYNC 5 mirrors 000657 1784093617.981945 MIRROR_RESYNC 5 mirrors 000660 1784093618.214834 MIRROR_RESYNC 5 mirrors 000661 1784093618.412224 mapwrite 0x00fa32 thru 0x017ed9 (0x84a8 bytes) 000664 1784093618.478599 trunc from 0x017eda to 0x03a2a4 (0x223cb bytes) 000665 1784093618.487602 read 0x035b6b thru 0x03a2a4 (0x473a bytes) 000667 1784093618.492429 trunc from 0x03a2a5 to 0x016384 (0x23f20 bytes) 000669 1784093618.500573 write 0x03a927 thru 0x03bed5 (0x15af bytes) 000670 1784093618.501573 write 0x021c0f thru 0x026957 (0x4d49 bytes) 000671 1784093618.513113 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bed6): Mirror 19: 0x8503a7a8 (stale) Mirror 20: 0x8503a7a8 (stale) Mirror 21: 0x8503a7a8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bed6): Mirror 19: 0x8503a7a8 (stale) Mirror 20: 0x8503a7a8 (stale) Mirror 21: 0x8503a7a8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bed6: skipped 000672 1784093618.709764 mapread 0x00b441 thru 0x00eefc (0x3abc bytes) 000673 1784093618.719516 write 0x02e07e thru 0x039167 (0xb0ea bytes) 000675 1784093618.727923 MIRROR_RESYNC 4 mirrors 000676 1784093619.086516 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ee:0x0]: Invalid argument (22) 000677 1784093619.268535 mapwrite 0x024c3d thru 0x02d8fb (0x8cbf bytes) 000678 1784093619.350520 MIRROR_RESYNC 3 mirrors 000679 1784093619.657328 mapwrite 0x03445e thru 0x03ffff (0xbba2 bytes) 000680 1784093619.732697 MIRROR_RESYNC 3 mirrors 000682 1784093620.015635 read 0x0225bb thru 0x02375c (0x11a2 bytes) 000683 1784093620.019848 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ee:0x0]: Invalid argument (22) 000687 1784093620.152503 read 0x036f8e thru 0x03d6b7 (0x672a bytes) 000690 1784093620.159269 mapwrite 0x000f6f thru 0x00dbd7 (0xcc69 bytes) 000692 1784093620.253765 mapread 0x030a67 thru 0x03ffff (0xf599 bytes) 000693 1784093620.263447 write 0x036285 thru 0x03ffff (0x9d7b bytes) 000695 1784093620.265180 write 0x03c01d thru 0x03d8d8 (0x18bc bytes) 000696 1784093620.266151 mapread 0x01ea12 thru 0x022d80 (0x436f bytes) 000698 1784093620.272312 mapwrite 0x02693e thru 0x02fff2 (0x96b5 bytes) 000699 1784093620.300343 write 0x01f550 thru 0x01fdb6 (0x867 bytes) 000700 1784093620.301490 read 0x03ffb3 thru 0x03ffff (0x4d bytes) 000701 1784093620.302092 read 0x005b4e thru 0x012d17 (0xd1ca bytes) 000702 1784093620.307439 read 0x004c42 thru 0x0076b0 (0x2a6f bytes) 000703 1784093620.308011 mapwrite 0x02410a thru 0x024a58 (0x94f bytes) 000706 1784093620.329230 mapread 0x03772c thru 0x03e729 (0x6ffe bytes) 000709 1784093620.331714 write 0x024ca3 thru 0x03048c (0xb7ea bytes) 000711 1784093620.333666 write 0x03570a thru 0x039394 (0x3c8b bytes) 000712 1784093620.334689 read 0x030e05 thru 0x0325c2 (0x17be bytes) 000713 1784093620.335324 mapread 0x033587 thru 0x037940 (0x43ba bytes) 000714 1784093620.336210 write 0x03b1fe thru 0x03ffff (0x4e02 bytes) 000715 1784093620.337080 trunc from 0x040000 to 0x03e10f (0x1ef0 bytes) 000717 1784093620.360172 MIRROR_RESYNC 2 mirrors 000720 1784093620.593752 mapread 0x034633 thru 0x03e10f (0x9add bytes) 000721 1784093620.607637 MIRROR_RESYNC 2 mirrors 000723 1784093620.770359 mapwrite 0x00a147 thru 0x014695 (0xa54f bytes) 000724 1784093620.844512 mapread 0x0345a1 thru 0x03e10f (0x9b6f bytes) 000725 1784093620.854050 read 0x030396 thru 0x03a088 (0x9cf3 bytes) 000726 1784093620.857341 MIRROR_RESYNC 2 mirrors 000727 1784093621.161953 mapwrite 0x035157 thru 0x03f4a8 (0xa352 bytes) 000729 1784093621.237826 mapwrite 0x0051e9 thru 0x006f3a (0x1d52 bytes) 000730 1784093621.265998 MIRROR_RESYNC 2 mirrors 000731 1784093621.525166 MIRROR_EXTEND to 3 mirrors 000732 1784093621.817814 MIRROR_RESYNC 3 mirrors 000733 1784093622.005060 read 0x012c1f thru 0x018f2a (0x630c bytes) 000735 1784093622.010795 read 0x00e3d9 thru 0x018d8a (0xa9b2 bytes) 000736 1784093622.017556 read 0x00208a thru 0x004e76 (0x2ded bytes) 000738 1784093622.018359 write 0x03b149 thru 0x03fa73 (0x492b bytes) 000739 1784093622.063618 mapread 0x03998c thru 0x03fa73 (0x60e8 bytes) 000740 1784093622.067890 MIRROR_RESYNC 3 mirrors 000741 1784093622.391956 trunc from 0x03fa74 to 0x018f78 (0x26afb bytes) 000742 1784093622.443195 trunc from 0x018f79 to 0x03c16a (0x231f2 bytes) 000748 1784093622.457720 write 0x008f1f thru 0x011f67 (0x9049 bytes) 000749 1784093622.466153 MIRROR_RESYNC 3 mirrors 000752 1784093622.806472 MIRROR_EXTEND to 4 mirrors 000753 1784093623.076941 mapread 0x01b4dc thru 0x025b1a (0xa63f bytes) 000754 1784093623.088345 read 0x01d940 thru 0x02af89 (0xd64a bytes) 000755 1784093623.092960 trunc from 0x03c16b to 0x0339a7 (0x87c3 bytes) 000756 1784093623.134517 mapread 0x0259b5 thru 0x0339a7 (0xdff3 bytes) 000758 1784093623.147704 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x339a8): Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x339a8): Mirror 20: 0x6e000ac5 (stale) Mirror 21: 0x6e000ac5 (stale) Mirror 22: 0x6e000ac5 (stale) CRC-32 checksum value for chunk [0, 0x339a8): Mirror 1: 0x6e000ac5 Mirror 23: 0x6e000ac5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x339a8: skipped 000759 1784093623.497908 write 0x00e7f3 thru 0x015c9d (0x74ab bytes) 000761 1784093623.543112 trunc from 0x0339a8 to 0x019acf (0x19ed8 bytes) 000767 1784093623.566079 mapwrite 0x01330a thru 0x01d3a1 (0xa098 bytes) 000768 1784093623.606014 write 0x010529 thru 0x0127d5 (0x22ad bytes) 000771 1784093623.610103 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d3a2): Mirror 20: 0x7c741d30 (stale) Mirror 22: 0x7c741d30 (stale) Mirror 23: 0x7c741d30 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d3a2): Mirror 20: 0x7c741d30 (stale) Mirror 22: 0x7c741d30 (stale) Mirror 23: 0x7c741d30 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d3a2: skipped 000772 1784093623.769784 mapread 0x012f60 thru 0x01d3a1 (0xa442 bytes) 000773 1784093623.782400 mapwrite 0x00eece thru 0x01b379 (0xc4ac bytes) 000774 1784093623.802936 mapread 0x01131e thru 0x01d3a1 (0xc084 bytes) 000778 1784093623.805756 write 0x023218 thru 0x027ea5 (0x4c8e bytes) 000780 1784093623.807602 trunc from 0x027ea6 to 0x00bffa (0x1beab bytes) 000782 1784093623.816670 trunc from 0x00bffb to 0x012752 (0x6758 bytes) 000783 1784093623.824575 mapread 0x0082be thru 0x012752 (0xa495 bytes) 000787 1784093623.836312 mapwrite 0x03d427 thru 0x03ffff (0x2bd9 bytes) 000788 1784093623.865849 mapwrite 0x01f242 thru 0x01f278 (0x37 bytes) 000789 1784093623.889455 mapread 0x027cf7 thru 0x02f7c1 (0x7acb bytes) 000790 1784093623.895295 mapread 0x03838e thru 0x03ffff (0x7c72 bytes) 000791 1784093623.896502 write 0x00d32b thru 0x013d37 (0x6a0d bytes) 000792 1784093623.901027 trunc from 0x040000 to 0x008973 (0x3768c bytes) 000795 1784093623.912062 write 0x012288 thru 0x01caa8 (0xa821 bytes) 000796 1784093623.913833 mapread 0x00e7b2 thru 0x01caa8 (0xe2f7 bytes) 000797 1784093623.919108 read 0x005d1a thru 0x00a8a8 (0x4b8f bytes) 000798 1784093623.919821 write 0x030ff5 thru 0x034ae5 (0x3af1 bytes) 000800 1784093623.921196 mapread 0x01fca3 thru 0x02e4a5 (0xe803 bytes) 000804 1784093623.931270 trunc from 0x034ae6 to 0x03722f (0x274a bytes) 000807 1784093623.946856 MIRROR_RESYNC 4 mirrors 000810 1784093624.327858 trunc from 0x037230 to 0x024b48 (0x126e7 bytes) 000812 1784093624.356764 write 0x0109bd thru 0x013090 (0x26d4 bytes) 000815 1784093624.368721 MIRROR_RESYNC 4 mirrors 000817 1784093624.698567 mapread 0x00dbd3 thru 0x0159d1 (0x7dff bytes) 000818 1784093624.711859 MIRROR_EXTEND to 5 mirrors 000823 1784093625.020407 trunc from 0x024b49 to 0x00dd2b (0x16e1d bytes) 000829 1784093625.055520 write 0x030dd1 thru 0x03ffff (0xf22f bytes) 000830 1784093625.057144 write 0x022869 thru 0x0267be (0x3f56 bytes) 000831 1784093625.063581 read 0x0281f2 thru 0x029520 (0x132f bytes) 000833 1784093625.071777 MIRROR_RESYNC 5 mirrors 000834 1784093625.472203 mapwrite 0x0183ce thru 0x019120 (0xd53 bytes) 000837 1784093625.553406 read 0x02b36b thru 0x03a63e (0xf2d4 bytes) 000839 1784093625.561678 mapread 0x017032 thru 0x018669 (0x1638 bytes) 000840 1784093625.562516 write 0x005f41 thru 0x0113f9 (0xb4b9 bytes) 000842 1784093625.573024 mapwrite 0x00bb77 thru 0x018d65 (0xd1ef bytes) 000843 1784093625.604764 mapwrite 0x014469 thru 0x019a35 (0x55cd bytes) 000844 1784093625.633923 mapread 0x01144a thru 0x014667 (0x321e bytes) 000845 1784093625.635054 mapwrite 0x0025b3 thru 0x00ed46 (0xc794 bytes) 000846 1784093625.665577 mapread 0x019c79 thru 0x021a57 (0x7ddf bytes) 000847 1784093625.670027 MIRROR_RESYNC 5 mirrors 000848 1784093626.066357 mapwrite 0x003bc7 thru 0x00a76c (0x6ba6 bytes) 000850 1784093626.156613 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0x3b19bc7 (stale) Mirror 22: 0x3b19bc7 (stale) Mirror 23: 0x3b19bc7 (stale) Mirror 24: 0x3b19bc7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x18ebe12a Mirror 25: 0x18ebe12a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000851 1784093626.588529 read 0x0229a0 thru 0x023f94 (0x15f5 bytes) 000855 1784093626.615284 MIRROR_RESYNC 6 mirrors 000857 1784093627.029831 write 0x03dbea thru 0x03ffff (0x2416 bytes) 000858 1784093627.065054 mapread 0x00591a thru 0x01277c (0xce63 bytes) 000862 1784093627.084662 trunc from 0x040000 to 0x008fab (0x37054 bytes) 000863 1784093627.097553 mapread 0x002c81 thru 0x008fab (0x632b bytes) 000867 1784093627.100338 mapwrite 0x001a8c thru 0x004e56 (0x33cb bytes) 000868 1784093627.119722 write 0x00119e thru 0x003e4a (0x2cad bytes) 000869 1784093627.120893 mapwrite 0x008779 thru 0x00b388 (0x2c10 bytes) 000870 1784093627.151739 mapwrite 0x01b016 thru 0x028dd1 (0xddbc bytes) 000871 1784093627.197433 trunc from 0x028dd2 to 0x00c003 (0x1cdce bytes) 000872 1784093627.207110 mapread 0x006431 thru 0x00c003 (0x5bd3 bytes) 000873 1784093627.212094 read 0x00b1d9 thru 0x00c003 (0xe2b bytes) 000874 1784093627.212746 mapwrite 0x029d9f thru 0x031e22 (0x8084 bytes) 000879 1784093627.258171 read 0x02de29 thru 0x031e22 (0x3ffa bytes) 000881 1784093627.259297 mapread 0x02456b thru 0x024c7b (0x711 bytes) 000882 1784093627.260198 read 0x0259ce thru 0x02abb4 (0x51e7 bytes) 000883 1784093627.263336 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x31e23): Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31e23): Mirror 20: 0xe558dd1a (stale) Mirror 22: 0xe558dd1a (stale) Mirror 23: 0xe558dd1a (stale) Mirror 24: 0xe558dd1a (stale) Mirror 25: 0xe558dd1a (stale) CRC-32 checksum value for chunk [0, 0x31e23): Mirror 1: 0x59a22f3d Mirror 26: 0x59a22f3d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31e23: skipped 000884 1784093627.646652 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31e23): Mirror 20: 0xe558dd1a (stale) Mirror 23: 0xe558dd1a (stale) Mirror 24: 0xe558dd1a (stale) Mirror 25: 0xe558dd1a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31e23): Mirror 20: 0xe558dd1a (stale) Mirror 23: 0xe558dd1a (stale) Mirror 24: 0xe558dd1a (stale) Mirror 25: 0xe558dd1a (stale) CRC-32 checksum value for chunk [0, 0x31e23): Mirror 1: 0x59a22f3d Mirror 26: 0x59a22f3d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31e23: skipped 000888 1784093628.045233 write 0x023a6f thru 0x02da28 (0x9fba bytes) 000890 1784093628.098499 mapread 0x028e80 thru 0x030620 (0x77a1 bytes) 000891 1784093628.114341 trunc from 0x031e23 to 0x0073ae (0x2aa74 bytes) 000892 1784093628.140832 read 0x0007c2 thru 0x00188e (0x10cd bytes) 000893 1784093628.146061 read 0x005f61 thru 0x0073ae (0x144e bytes) 000894 1784093628.156843 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x73af): Mirror 20: 0x65a2aba7 (stale) Mirror 23: 0x65a2aba7 (stale) Mirror 24: 0x65a2aba7 (stale) Mirror 25: 0x65a2aba7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x73af): Mirror 20: 0x65a2aba7 (stale) Mirror 23: 0x65a2aba7 (stale) Mirror 24: 0x65a2aba7 (stale) Mirror 25: 0x65a2aba7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x73af: skipped 000895 1784093628.350638 read 0x00494d thru 0x0073ae (0x2a62 bytes) 000897 1784093628.359797 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x73af): Mirror 23: 0x65a2aba7 (stale) Mirror 24: 0x65a2aba7 (stale) Mirror 25: 0x65a2aba7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x73af): Mirror 23: 0x65a2aba7 (stale) Mirror 24: 0x65a2aba7 (stale) Mirror 25: 0x65a2aba7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x73af: skipped 000899 1784093628.555427 write 0x0084a9 thru 0x00a873 (0x23cb bytes) 000900 1784093628.556656 trunc from 0x00a874 to 0x022567 (0x17cf4 bytes) 000902 1784093628.570553 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22568): Mirror 23: 0x9cb05941 (stale) Mirror 24: 0x9cb05941 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22568): Mirror 23: 0x9cb05941 (stale) Mirror 24: 0x9cb05941 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22568: skipped 000903 1784093628.744006 write 0x02bf68 thru 0x036283 (0xa31c bytes) 000904 1784093628.745602 write 0x02227b thru 0x02d010 (0xad96 bytes) 000905 1784093628.749970 trunc from 0x036284 to 0x00754c (0x2ed37 bytes) 000906 1784093628.762603 read 0x003d81 thru 0x00754c (0x37cc bytes) 000914 1784093628.773285 trunc from 0x00754d to 0x03cb28 (0x355dc bytes) 000917 1784093628.786783 mapread 0x003ab7 thru 0x013a78 (0xffc2 bytes) 000918 1784093628.803683 write 0x03c6ec thru 0x03e7ec (0x2101 bytes) 000921 1784093628.808508 write 0x00b1c5 thru 0x00ccdf (0x1b1b bytes) 000922 1784093628.809537 write 0x039fac thru 0x03ffff (0x6054 bytes) 000923 1784093628.813035 mapread 0x033297 thru 0x03b621 (0x838b bytes) 000924 1784093628.818816 mapread 0x03e555 thru 0x03ffff (0x1aab bytes) 000925 1784093628.819766 trunc from 0x040000 to 0x01d864 (0x2279b bytes) 000926 1784093628.831294 mapread 0x00872c thru 0x010f5e (0x8833 bytes) 000927 1784093628.832615 read 0x01d3ce thru 0x01d864 (0x497 bytes) 000928 1784093628.840581 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1d865): Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d865): Mirror 23: 0x6ff4c9a6 (stale) Mirror 24: 0x6ff4c9a6 (stale) CRC-32 checksum value for chunk [0, 0x1d865): Mirror 1: 0x6a868bce Mirror 25: 0x6a868bce lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d865: skipped 000930 1784093629.302209 read 0x00b82e thru 0x00eee5 (0x36b8 bytes) 000931 1784093629.313664 MIRROR_RESYNC 4 mirrors 000932 1784093629.672945 write 0x03c464 thru 0x03ffff (0x3b9c bytes) 000933 1784093629.706773 trunc from 0x040000 to 0x036eef (0x9110 bytes) 000934 1784093629.716929 trunc from 0x036ef0 to 0x025a56 (0x11499 bytes) 000935 1784093629.725470 read 0x002748 thru 0x00f415 (0xccce bytes) 000936 1784093629.730903 mapwrite 0x008537 thru 0x016d40 (0xe80a bytes) 000937 1784093629.773847 write 0x00a004 thru 0x0102fc (0x62f9 bytes) 000939 1784093629.775589 mapwrite 0x004f3e thru 0x00b6e7 (0x67aa bytes) 000943 1784093629.804829 mapread 0x0165d9 thru 0x01ab7f (0x45a7 bytes) 000944 1784093629.805873 trunc from 0x025a57 to 0x036183 (0x1072d bytes) 000945 1784093629.815897 write 0x03879b thru 0x03ffff (0x7865 bytes) 000946 1784093629.817492 read 0x02bf26 thru 0x031373 (0x544e bytes) 000949 1784093629.825174 trunc from 0x040000 to 0x02f6ec (0x10913 bytes) 000950 1784093629.836338 mapwrite 0x03b4ce thru 0x03ffff (0x4b32 bytes) 000951 1784093629.889674 trunc from 0x040000 to 0x037fdc (0x8023 bytes) 000952 1784093629.899797 write 0x0139fd thru 0x01eeba (0xb4be bytes) 000957 1784093629.903358 mapread 0x009a49 thru 0x00a5c0 (0xb78 bytes) 000958 1784093629.904719 trunc from 0x037fdd to 0x024097 (0x13f45 bytes) 000959 1784093629.914387 trunc from 0x024098 to 0x02eca5 (0xac0e bytes) 000961 1784093629.925248 mapread 0x02aa55 thru 0x02eca5 (0x4251 bytes) 000962 1784093629.940235 write 0x025b59 thru 0x0308ad (0xad55 bytes) 000963 1784093629.944207 trunc from 0x0308ae to 0x019164 (0x17749 bytes) 000964 1784093629.953214 read 0x0075e4 thru 0x00cfb1 (0x59ce bytes) 000966 1784093629.954602 mapread 0x0147db thru 0x019164 (0x498a bytes) 000968 1784093629.959142 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x19165): Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19165): Mirror 23: 0x4aca138a (stale) Mirror 24: 0x4aca138a (stale) Mirror 25: 0x4aca138a (stale) CRC-32 checksum value for chunk [0, 0x19165): Mirror 1: 0x233004a5 Mirror 26: 0x233004a5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19165: skipped 000969 1784093630.375544 mapwrite 0x01dca1 thru 0x024d3a (0x709a bytes) 000970 1784093630.466066 mapread 0x01ae2c thru 0x024d3a (0x9f0f bytes) 000972 1784093630.467816 trunc from 0x024d3b to 0x02ebd6 (0x9e9c bytes) 000974 1784093630.483587 write 0x026714 thru 0x02f535 (0x8e22 bytes) 000975 1784093630.488822 read 0x00fcdf thru 0x0104e6 (0x808 bytes) 000977 1784093630.494538 mapread 0x02ba51 thru 0x02f535 (0x3ae5 bytes) 000978 1784093630.495823 read 0x003bce thru 0x007937 (0x3d6a bytes) 000979 1784093630.505003 mapread 0x006f4e thru 0x016370 (0xf423 bytes) 000981 1784093630.506899 write 0x03bec2 thru 0x03ffff (0x413e bytes) 000982 1784093630.510324 mapwrite 0x03bfbc thru 0x03ffff (0x4044 bytes) 000986 1784093630.545302 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: 0x1f504d5 (stale) Mirror 24: 0x1f504d5 (stale) Mirror 25: 0x1f504d5 (stale) Mirror 26: 0x277df689 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6641946d Mirror 27: 0x6641946d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000989 1784093630.972267 read 0x02f755 thru 0x03c21a (0xcac6 bytes) 000992 1784093630.985303 write 0x02f7c1 thru 0x03cc90 (0xd4d0 bytes) 000993 1784093631.029762 write 0x018358 thru 0x0280c0 (0xfd69 bytes) 000996 1784093631.038199 read 0x001254 thru 0x002e8d (0x1c3a bytes) 000999 1784093631.046244 trunc from 0x040000 to 0x031668 (0xe997 bytes) 001000 1784093631.065738 mapread 0x01d8f7 thru 0x02cacf (0xf1d9 bytes) All operations completed A-OK! PASS 70a (45s) == sanity-flr test 71: check mirror extend parent fid ==== 01:33:53 (1784093633) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 72: check append on FLR file ========== 01:33:54 (1784093634) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x639:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x61d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.118006 s, 17.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x639:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x63a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x61d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075481 s, 13.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x639:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x63a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x61e:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x61d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x639:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x63a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x61e:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x61d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x63b:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (4s) == sanity-flr test 200a: stress test ===================== 01:33:58 (1784093638) Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre2 Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8540704 .. Extending file size to 4805536 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4904960 .. Extending file size to 8689696 .. Extending file size to 5353312 .. failed Extending file size to 7747264 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8434016 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8849920 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8808768 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9001888 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9088864 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9060448 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58594 58595 58597 58599 58601 done 192.168.204.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg419-client.virtnet /mnt/lustre2 (opts:) 192.168.204.119@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg419-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (69s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 01:35:07 (1784093707) Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre2 Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7266880 .. Extending file size to 1384384 .. failed Extending file size to 4711008 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8045600 .. Extending file size to 8704672 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8166752 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8527776 .. failed Extending file size to 9058656 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8770432 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8929152 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 60774 60775 60777 60778 60781 192.168.204.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg419-client.virtnet /mnt/lustre2 (opts:) 192.168.204.119@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg419-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (66s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 01:36:13 (1784093773) Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3328 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3328 2205056 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0461518 s, 45.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.118599 s, 17.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xc9a20051 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg419-client.virtnet /mnt/lustre2 (opts:) PASS 200c (7s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 01:36:21 (1784093781) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0442522 s, 47.4 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 01:36:24 (1784093784) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x43a:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x634:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x634:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x635:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x650:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 01:36:27 (1784093787) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0521538 s, 40.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x636:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x651:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x652:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 01:36:30 (1784093790) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0502632 s, 41.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x637:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x638:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x653:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x637:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x638:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x653:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x654:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 01:36:33 (1784093793) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.050422 s, 41.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x639:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x655:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x639:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x655:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x656:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 01:36:57 (1784093817) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg419-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033067 s, 31.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x658:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x657:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x658:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x657:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63b:0x0] } PASS 204d (47s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 01:37:44 (1784093864) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg419-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x659:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x65a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0439085 s, 47.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x659:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x65a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x659:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x65a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 01:38:30 (1784093910) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg419-server mds-ost sync done. Creating new pool test_204f oleg419-server: Pool lustre.test_204f created Adding targets to pool oleg419-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg419-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0530848 s, 39.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x65b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x65b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg419-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg419-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg419-server: Pool lustre.test_204f destroyed PASS 204f (59s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 01:39:29 (1784093969) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 01:39:31 (1784093971) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 01:39:34 (1784093974) oleg419-server: Pool lustre.test_206 created Adding targets to pool oleg419-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg419-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg419-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg419-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg419-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (12s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 01:39:46 (1784093986) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289042 s, 36.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109518 s, 95.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xc4cb6d06 Mirror 3: 0xc4cb6d06 PASS 207 (4s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 01:39:50 (1784093990) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 01:39:51 (1784093991) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 01:39:52 (1784093992) Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0195806 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xee0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2180000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2180000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2180000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1110000 Read this (4 bytes): BBBB 192.168.204.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg419-client.virtnet /mnt/lustre2 (opts:) PASS 209a (16s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 01:40:08 (1784094008) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00377703 s, 2.2 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 01:40:10 (1784094010) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159651 s, 65.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x45f lmm_fid: [0x200000404:0x45f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x45f lmm_fid: [0x200000404:0x45f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x647:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835039 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-15 01:40:11.000000000 -0400 Modify: 2026-07-15 01:40:11.000000000 -0400 Change: 2026-07-15 01:40:11.000000000 -0400 Birth: 2026-07-15 01:40:11.000000000 -0400 PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 01:40:13 (1784094013) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 01:40:14 (1784094014) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.11805 s, 347 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0143594 s, 285 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 15 01:40 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x461:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 15 01:40 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 01:40:17 (1784094017) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (2s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 01:40:19 (1784094019) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (3s) == sanity-flr test complete, duration 992 sec ============ 01:40:22 (1784094022) === sanity-flr: start cleanup 01:40:23 (1784094023) === === sanity-flr: finish cleanup 01:40:25 (1784094025) ===