-----============= acceptance-small: sanity-flr ============----- Fri Sep 4 21:32:04 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg451-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 21:32:23 (1788571943) === oleg451-client.virtnet: executing check_config_client /mnt/lustre oleg451-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg451-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff940d05238800.idle_timeout=debug osc.lustre-OST0001-osc-ffff940d05238800.idle_timeout=debug disable quota as required oleg451-server: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 21:33:02 (1788571982) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7620] == sanity-flr test 0a: lfs mirror create with -N option == 21:33:05 (1788571985) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 21:33:23 (1788572003) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 21:33:26 (1788572006) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 21:33:30 (1788572010) PASS 0d (27s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 21:33:57 (1788572037) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 21:34:01 (1788572041) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 21:34:05 (1788572045) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff940d05238800.stats=clear osc.lustre-OST0001-osc-ffff940d05238800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 21:34:14 (1788572054) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--not-pool POOL_NAME] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pin] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 21:34:26 (1788572066) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 21:34:34 (1788572074) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 21:34:38 (1788572078) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 21:34:46 (1788572086) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 21:34:56 (1788572096) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 21:35:03 (1788572103) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 21:35:06 (1788572106) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 21:35:13 (1788572113) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 21:35:24 (1788572124) PASS 7 (9s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 21:35:33 (1788572133) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 1.76227 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119349 s, 8.8 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 21:35:46 (1788572146) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.362347 s, 34.7 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 21:35:55 (1788572155) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 21:36:07 (1788572167) Stopping /mnt/lustre-ost1 (opts:) on oleg451-server oleg451-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff940d05238800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff940d05238800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg451-server: oleg451-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg451-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg451-server oleg451-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff940d05238800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff940d05238800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg451-server: oleg451-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg451-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (67s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 21:37:14 (1788572234) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.443672 s, 14.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg451-server oleg451-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff940d05238800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff940d05238800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg451-server: oleg451-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg451-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (39s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 21:37:53 (1788572273) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg451-server oleg451-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff940d05238800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff940d05238800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg451-server: oleg451-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg451-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg451-server oleg451-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff940d05238800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff940d05238800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg451-server: oleg451-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg451-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (74s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 21:39:07 (1788572347) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.78339 s, 9.1 MB/s -rw-r--r-- 1 root root 52428800 Sep 4 21:39 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.026s user 0m0.000s sys 0m0.182s Stopping /mnt/lustre-ost1 (opts:) on oleg451-server oleg451-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff940d05238800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff940d05238800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.013s user 0m0.001s sys 0m0.207s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg451-server: oleg451-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg451-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg451-server oleg451-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff940d05238800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff940d05238800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.610s user 0m0.001s sys 0m0.187s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg451-server: oleg451-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg451-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg451-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff940d05238800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff940d05238800.ost_server_uuid in FULL state after 0 sec PASS 33b (98s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 21:40:45 (1788572445) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 21:40:48 (1788572448) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 21:40:52 (1788572452) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 21:40:55 (1788572455) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 21:41:02 (1788572462) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.647993 s, 19.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 21:41:14 (1788572474) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338486 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231118 s, 45.4 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 21:41:20 (1788572480) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 21:41:30 (1788572490) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000752907 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00172662 s, 593 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00172922 s, 592 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000871898 s, 1.2 MB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 21:41:40 (1788572500) create test files with size 15750690 .. 32a1759c8382263b0d80b39101125ece - 55618fbc507c9c3caf9889e237bc984f - 805a7ff314a2046b79126c21e29584d2 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg451-client.virtnet /mnt/lustre (opts:) Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 21:42:03 (1788572523) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 21:42:18 (1788572538) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 21:42:25 (1788572545) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124588 s, 8.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218879 s, 47.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 21:42:32 (1788572552) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.12711 s, 16.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.402955 s, 10.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.182058 s, 11.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (13s) == sanity-flr test 42: lfs mirror verify ================= 21:42:46 (1788572566) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 21:42:48 (1788572568) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 21:42:51 (1788572571) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326787 s, 32.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 21:43:00 (1788572580) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 21:43:04 (1788572584) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 21:43:14 (1788572594) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.370271 s, 28.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg451-client.virtnet /mnt/lustre (opts:) Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 21:43:26 (1788572606) llite.lustre-ffff940d07726000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.282673 s, 37.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg451-client.virtnet /mnt/lustre (opts:) Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff940d1a729800.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 21:43:41 (1788572621) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (36s) == sanity-flr test 46: Verify setstripe --copy option ==== 21:44:17 (1788572657) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (37s) == sanity-flr test 47: Verify mirror obj alloc =========== 21:44:55 (1788572695) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 21:44:58 (1788572698) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.31295 s, 10.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.243294 s, 12.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x25b293d5 (nosync,stale) Mirror 2: 0x25b293d5 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 6e061ac4b63dda319966aa88922a7d34 - mirror 1 checksum: c4ea520681d1a7a38ad5c5854a5fd734 - mirror 2 checksum: 6e061ac4b63dda319966aa88922a7d34 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-04T21:45:01-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-04T21:45:01-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 6e061ac4b63dda319966aa88922a7d34 - mirror 1 checksum: c4ea520681d1a7a38ad5c5854a5fd734 - mirror 2 checksum: c4ea520681d1a7a38ad5c5854a5fd734 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-04T21:45:01-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-04T21:45:01-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 21:45:11 (1788572711) Failing mds1 on oleg451-server Stopping /mnt/lustre-mds1 (opts:) on oleg451-server 21:45:20 (1788572720) shut down facet: mds1 facet_host: oleg451-server facet_failover_host: oleg451-server Failover mds1 to oleg451-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-MDT0000 21:45:53 (1788572753) targets are mounted 21:45:53 (1788572753) facet_failover done oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677575 s, 15.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xee:0x0] } PASS 50A (61s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 21:46:12 (1788572772) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (5s) == sanity-flr test 50b: mirror rsync handles sparseness == 21:46:17 (1788572777) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 21:46:21 (1788572781) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.015774 s, 1.0 MB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 21:46:31 (1788572791) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 21:46:35 (1788572795) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0182017 s, 225 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.151804 s, 27.0 kB/s PASS 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 21:46:45 (1788572805) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (29s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 21:47:13 (1788572833) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 21:47:44 (1788572864) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 21:48:08 (1788572888) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 21:48:34 (1788572914) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 21:48:43 (1788572923) mirror create pid 54319 mirror split pid 54320 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7845: 54319 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54320 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 21:49:51 (1788572991) Seed set to 2993 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000004 1788572993.813762 mapwrite 0x01399d thru 0x01ea9c (0xb100 bytes) 000005 1788572994.087769 mapwrite 0x00c9a9 thru 0x0126ca (0x5d22 bytes) 000006 1788572994.209036 mapwrite 0x02da0f thru 0x033772 (0x5d64 bytes) 000007 1788572994.335610 read 0x01d0d1 thru 0x01f84d (0x277d bytes) 000008 1788572994.346282 read 0x022941 thru 0x02b31e (0x89de bytes) 000011 1788572994.375897 mapread 0x023e04 thru 0x0317d9 (0xd9d6 bytes) 000012 1788572994.381670 punch from 0x023fbe to 0x02524f (0x1292 bytes) 000013 1788572994.402396 mapread 0x010320 thru 0x01ae5f (0xab40 bytes) truncating to largest ever: 0x1b148 000015 1788572994.403929 trunc from 0x033773 to 0x01b147 (0x1862b bytes) 000017 1788572994.435718 read 0x01333a thru 0x014809 (0x14d0 bytes) 000019 1788572994.438431 read 0x018e11 thru 0x01b147 (0x2337 bytes) 000022 1788572994.440422 mapwrite 0x03e1ef thru 0x03ffff (0x1e11 bytes) 000024 1788572994.577382 write 0x03e36a thru 0x03ffff (0x1c96 bytes) truncating to largest ever: 0x3e3cb 000026 1788572994.580398 trunc from 0x040000 to 0x03e3ca (0x1c35 bytes) 000028 1788572994.606239 trunc from 0x03e3cb to 0x015bc4 (0x28806 bytes) 000029 1788572994.628113 write 0x03fdf1 thru 0x03ffff (0x20f bytes) 000030 1788572994.630725 mapwrite 0x03e3b0 thru 0x03ffff (0x1c50 bytes) 000033 1788572994.694399 trunc from 0x040000 to 0x004b38 (0x3b4c7 bytes) 000034 1788572994.723578 trunc from 0x004b39 to 0x003baf (0xf89 bytes) 000035 1788572994.751371 punch from 0x001991 to 0x003baf (0x221f bytes) 000036 1788572994.757997 trunc from 0x003bb0 to 0x00c4aa (0x88fb bytes) 000037 1788572994.784696 trunc from 0x00c4ab to 0x014675 (0x81cb bytes) 000038 1788572994.806143 trunc from 0x014676 to 0x00235c (0x12319 bytes) 000039 1788572994.831535 MIRROR_EXTEND to 2 mirrors 000040 1788572995.995051 trunc from 0x00235d to 0x011da8 (0xfa4c bytes) 000041 1788572996.149602 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) 000042 1788572996.350042 mapwrite 0x024fac thru 0x02e3e6 (0x943b bytes) 000043 1788572996.476027 mapwrite 0x002815 thru 0x00919d (0x6989 bytes) 000047 1788572996.556990 read 0x022bad thru 0x02677f (0x3bd3 bytes) 000048 1788572996.557969 mapread 0x01ece4 thru 0x02bb63 (0xce80 bytes) 000049 1788572996.574422 write 0x03837c thru 0x03c064 (0x3ce9 bytes) 000051 1788572996.584509 MIRROR_EXTEND to 2 mirrors 000052 1788572997.857583 punch from 0x00fabc to 0x01cfae (0xd4f3 bytes) 000053 1788572997.937511 read 0x0156ef thru 0x01ad58 (0x566a bytes) 000054 1788572997.947978 mapwrite 0x0374e4 thru 0x03ffff (0x8b1c bytes) 000056 1788572998.174549 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) 000058 1788572998.370710 mapread 0x0050f0 thru 0x0096e9 (0x45fa bytes) 000060 1788572998.406002 write 0x008af6 thru 0x01418a (0xb695 bytes) 000061 1788572998.412662 read 0x010fcd thru 0x014a60 (0x3a94 bytes) 000064 1788572998.415377 read 0x00bad0 thru 0x01345a (0x798b bytes) 000066 1788572998.417107 write 0x01cd95 thru 0x01e4ce (0x173a bytes) 000068 1788572998.432674 punch from 0x028de1 to 0x02c760 (0x3980 bytes) 000069 1788572998.469279 read 0x003159 thru 0x0080dd (0x4f85 bytes) 000070 1788572998.469944 mapread 0x022010 thru 0x02af52 (0x8f43 bytes) 000071 1788572998.480969 mapwrite 0x0217a1 thru 0x025fe1 (0x4841 bytes) 000072 1788572998.627929 MIRROR_EXTEND to 2 mirrors 000073 1788573000.360781 write 0x005156 thru 0x00f9a0 (0xa84b bytes) 000074 1788573000.600383 trunc from 0x040000 to 0x01968d (0x26972 bytes) 000075 1788573000.658967 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) 000076 1788573000.945032 trunc from 0x01968e to 0x02c531 (0x12ea4 bytes) 000080 1788573000.967646 mapread 0x00dd6e thru 0x01cbe4 (0xee77 bytes) 000087 1788573001.020271 mapwrite 0x035baf thru 0x03ffff (0xa451 bytes) 000088 1788573001.140729 read 0x03b701 thru 0x03ffff (0x48ff bytes) 000089 1788573001.142396 read 0x0048be thru 0x014462 (0xfba5 bytes) 000090 1788573001.154915 punch from 0x01c1c1 to 0x01fb07 (0x3947 bytes) 000091 1788573001.163125 punch from 0x0137a3 to 0x017dfc (0x465a bytes) 000092 1788573001.169807 mapwrite 0x03b2f0 thru 0x03ffff (0x4d10 bytes) 000093 1788573001.266477 punch from 0x02aec8 to 0x02e2da (0x3413 bytes) 000095 1788573001.277926 trunc from 0x040000 to 0x00422e (0x3bdd1 bytes) 000096 1788573001.320164 punch from 0x002da1 to 0x00422e (0x148e bytes) 000097 1788573001.325446 mapread 0x0040dd thru 0x00422e (0x152 bytes) 000099 1788573001.335409 read 0x001fae thru 0x00422e (0x2281 bytes) 000101 1788573001.357581 mapread 0x002ad8 thru 0x00422e (0x1757 bytes) 000102 1788573001.366899 mapread 0x00017c thru 0x00422e (0x40b3 bytes) 000103 1788573001.367990 read 0x002848 thru 0x00422e (0x19e7 bytes) 000105 1788573001.372374 mapread 0x00309b thru 0x00422e (0x1194 bytes) 000106 1788573001.373741 mapwrite 0x02cc46 thru 0x034a53 (0x7e0e bytes) 000107 1788573001.491568 read 0x003b63 thru 0x00eb0d (0xafab bytes) 000109 1788573001.501656 mapwrite 0x00cccc thru 0x01aeb1 (0xe1e6 bytes) 000112 1788573001.637070 punch from 0x016cc8 to 0x01785c (0xb95 bytes) 000113 1788573001.659617 read 0x02b8ca thru 0x034a53 (0x918a bytes) 000114 1788573001.665682 punch from 0x01fabc to 0x02a977 (0xaebc bytes) 000115 1788573001.679678 mapwrite 0x027507 thru 0x032528 (0xb022 bytes) 000116 1788573001.814898 write 0x03eaa2 thru 0x03ffff (0x155e bytes) 000117 1788573001.817199 read 0x01e681 thru 0x02b01e (0xc99e bytes) 000118 1788573001.828342 mapread 0x031369 thru 0x038b2f (0x77c7 bytes) 000121 1788573001.857493 read 0x0298d3 thru 0x029c53 (0x381 bytes) 000123 1788573001.862179 read 0x01e9b4 thru 0x027d7e (0x93cb bytes) 000124 1788573001.863633 punch from 0x022bc6 to 0x023247 (0x682 bytes) 000125 1788573001.877985 write 0x014052 thru 0x015c99 (0x1c48 bytes) 000126 1788573001.879126 read 0x01a02a thru 0x026323 (0xc2fa bytes) 000128 1788573001.900554 trunc from 0x040000 to 0x0342f0 (0xbd0f bytes) 000129 1788573001.920476 mapwrite 0x007071 thru 0x00d7e2 (0x6772 bytes) 000131 1788573002.020067 punch from 0x008057 to 0x008d60 (0xd0a bytes) 000132 1788573002.031932 read 0x020830 thru 0x026042 (0x5813 bytes) 000134 1788573002.042822 MIRROR_EXTEND to 2 mirrors 000135 1788573003.305143 punch from 0x00175e to 0x00e326 (0xcbc9 bytes) 000137 1788573003.416266 write 0x03db0a thru 0x03ffff (0x24f6 bytes) 000138 1788573003.418728 mapwrite 0x03ba41 thru 0x03f17a (0x373a bytes) 000142 1788573003.599905 mapread 0x02867e thru 0x0286f0 (0x73 bytes) 000143 1788573003.638796 mapwrite 0x00f7ff thru 0x01eaf2 (0xf2f4 bytes) 000144 1788573003.835470 write 0x01e43d thru 0x02b36d (0xcf31 bytes) 000145 1788573003.843360 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) 000146 1788573004.117963 read 0x0039b0 thru 0x003be3 (0x234 bytes) 000147 1788573004.128497 mapwrite 0x02b12f thru 0x032b1c (0x79ee bytes) 000148 1788573004.233535 read 0x02c60b thru 0x02cbc3 (0x5b9 bytes) 000150 1788573004.240242 mapwrite 0x01be80 thru 0x01bedd (0x5e bytes) 000151 1788573004.356360 trunc from 0x040000 to 0x02668d (0x19972 bytes) 000152 1788573004.374527 trunc from 0x02668e to 0x025f71 (0x71c bytes) 000155 1788573004.404169 mapwrite 0x00ff61 thru 0x01ee00 (0xeea0 bytes) 000156 1788573004.523067 MIRROR_EXTEND to 2 mirrors 000158 1788573005.542525 read 0x01f443 thru 0x01f823 (0x3e1 bytes) 000160 1788573005.571083 write 0x005afb thru 0x009678 (0x3b7e bytes) 000161 1788573005.677965 trunc from 0x025f72 to 0x016513 (0xfa5e bytes) 000163 1788573005.710200 mapwrite 0x03ba0e thru 0x03ffff (0x45f2 bytes) 000165 1788573005.923564 punch from 0x0388d6 to 0x03a72b (0x1e56 bytes) 000166 1788573005.938325 mapread 0x00c13b thru 0x0115d8 (0x549e bytes) 000167 1788573005.988158 write 0x033272 thru 0x03ffff (0xcd8e bytes) 000168 1788573006.003394 mapread 0x005160 thru 0x01380b (0xe6ac bytes) 000170 1788573006.029901 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3fbd5213 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc0a700f9 Mirror 3: 0xc0a700f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000171 1788573008.350677 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3fbd5213 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc0a700f9 Mirror 3: 0xc0a700f9 Mirror 4: 0xc0a700f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000172 1788573010.235469 read 0x012e5e thru 0x021f01 (0xf0a4 bytes) 000175 1788573010.279731 trunc from 0x040000 to 0x010125 (0x2feda bytes) 000176 1788573010.361692 mapwrite 0x017ede thru 0x0214e1 (0x9604 bytes) 000177 1788573010.467111 MIRROR_RESYNC 4 mirrors 000178 1788573012.721010 write 0x001a23 thru 0x010496 (0xea74 bytes) 000183 1788573012.972617 write 0x021268 thru 0x0244e8 (0x3281 bytes) 000184 1788573012.989139 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x244e9): Mirror 3: 0xb9305675 (stale) Mirror 4: 0xb9305675 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x244e9): Mirror 3: 0xb9305675 (stale) Mirror 4: 0xb9305675 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x244e9: skipped 000185 1788573013.424490 write 0x016ad6 thru 0x022b28 (0xc053 bytes) 000188 1788573013.442603 punch from 0x01a490 to 0x0244e8 (0xa059 bytes) 000189 1788573013.457404 read 0x011b5c thru 0x020398 (0xe83d bytes) 000190 1788573013.466846 mapread 0x013787 thru 0x01af20 (0x779a bytes) 000192 1788573013.469916 mapread 0x000c91 thru 0x007518 (0x6888 bytes) 000194 1788573013.497840 read 0x01bff8 thru 0x0244e8 (0x84f1 bytes) 000195 1788573013.510400 mapwrite 0x009c3c thru 0x0111f8 (0x75bd bytes) 000196 1788573013.580101 read 0x00bf3c thru 0x00e43c (0x2501 bytes) 000197 1788573013.582463 mapwrite 0x00f76f thru 0x011d66 (0x25f8 bytes) 000198 1788573013.643329 mapread 0x02224c thru 0x0244e8 (0x229d bytes) 000199 1788573013.647757 punch from 0x0136ad to 0x01a33b (0x6c8f bytes) 000203 1788573013.665703 punch from 0x01c8fb to 0x01e776 (0x1e7c bytes) 000204 1788573013.681910 MIRROR_RESYNC 3 mirrors 000206 1788573015.292727 write 0x030789 thru 0x032df6 (0x266e bytes) 000207 1788573015.363279 mapwrite 0x03e029 thru 0x03ffff (0x1fd7 bytes) 000208 1788573015.481882 punch from 0x019631 to 0x01ec05 (0x55d5 bytes) 000210 1788573015.492421 mapwrite 0x009910 thru 0x00b87c (0x1f6d bytes) 000212 1788573015.553154 mapread 0x02b049 thru 0x0301ca (0x5182 bytes) 000213 1788573015.564038 read 0x013b4c thru 0x017ab7 (0x3f6c bytes) 000214 1788573015.571754 write 0x02f27c thru 0x03489d (0x5622 bytes) 000215 1788573015.578891 trunc from 0x040000 to 0x030b3d (0xf4c2 bytes) 000217 1788573015.600115 trunc from 0x030b3e to 0x02ff00 (0xc3d bytes) 000218 1788573015.617976 mapread 0x02a2e7 thru 0x02ff00 (0x5c1a bytes) 000219 1788573015.619983 mapread 0x0139fe thru 0x0212b5 (0xd8b8 bytes) 000222 1788573015.637213 trunc from 0x02ff01 to 0x006a3f (0x294c1 bytes) 000223 1788573015.656325 mapread 0x006971 thru 0x006a3f (0xcf bytes) 000227 1788573015.661259 mapread 0x002829 thru 0x0053a8 (0x2b80 bytes) 000228 1788573015.663151 punch from 0x003cfb to 0x006a3f (0x2d45 bytes) 000229 1788573015.669266 mapread 0x00162f thru 0x006a3f (0x5411 bytes) 000230 1788573015.679223 mapwrite 0x03c792 thru 0x03ffff (0x386e bytes) 000231 1788573015.748681 write 0x035b3c thru 0x03a1c1 (0x4686 bytes) 000232 1788573015.750372 write 0x0197df thru 0x023e26 (0xa648 bytes) 000233 1788573015.761463 mapwrite 0x02a8a1 thru 0x02b2e2 (0xa42 bytes) 000236 1788573015.823157 punch from 0x024b91 to 0x02e27d (0x96ed bytes) 000237 1788573015.829860 write 0x016861 thru 0x02376e (0xcf0e bytes) 000238 1788573015.837036 write 0x0278dd thru 0x03690a (0xf02e bytes) 000239 1788573015.844789 mapread 0x03dab2 thru 0x03ffff (0x254e bytes) 000240 1788573015.845820 write 0x039d6f thru 0x03dcfc (0x3f8e bytes) 000242 1788573015.849034 trunc from 0x040000 to 0x004b73 (0x3b48c bytes) 000243 1788573015.870183 trunc from 0x004b74 to 0x039cf1 (0x3517e bytes) 000244 1788573015.910082 mapwrite 0x0280c6 thru 0x03795d (0xf898 bytes) 000247 1788573016.031008 mapwrite 0x008edf thru 0x012e39 (0x9f5b bytes) 000248 1788573016.164003 MIRROR_RESYNC 3 mirrors 000249 1788573018.417708 punch from 0x007139 to 0x00a783 (0x364b bytes) 000250 1788573018.495263 write 0x034b2b thru 0x03ad01 (0x61d7 bytes) 000251 1788573018.503486 mapwrite 0x0132ea thru 0x01bfe4 (0x8cfb bytes) 000254 1788573018.589204 write 0x00ab10 thru 0x019413 (0xe904 bytes) 000255 1788573018.600115 read 0x031b33 thru 0x0393c6 (0x7894 bytes) 000258 1788573018.610346 write 0x03cf6c thru 0x03ffff (0x3094 bytes) 000259 1788573018.617676 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x310e3b63 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x310e3b63 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000262 1788573019.108767 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x310e3b63 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf9436501 Mirror 4: 0xf9436501 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000263 1788573020.591064 punch from 0x0378ff to 0x03ffff (0x8701 bytes) 000265 1788573020.672330 read 0x0283f5 thru 0x036e04 (0xea10 bytes) 000266 1788573020.681251 mapwrite 0x027a49 thru 0x028edc (0x1494 bytes) 000268 1788573020.746237 mapwrite 0x01e162 thru 0x026d67 (0x8c06 bytes) 000271 1788573020.855577 write 0x03326b thru 0x0346c9 (0x145f bytes) 000272 1788573020.857893 read 0x037b3e thru 0x03ffff (0x84c2 bytes) 000273 1788573020.872078 mapread 0x00656b thru 0x007043 (0xad9 bytes) 000276 1788573020.875627 write 0x02f223 thru 0x034d9b (0x5b79 bytes) 000278 1788573020.878011 read 0x00eea2 thru 0x018ae0 (0x9c3f bytes) 000279 1788573020.879200 mapwrite 0x0077b3 thru 0x009fb4 (0x2802 bytes) 000280 1788573021.023758 punch from 0x002255 to 0x007fda (0x5d86 bytes) 000282 1788573021.033822 mapread 0x032ba3 thru 0x03ff91 (0xd3ef bytes) 000283 1788573021.036520 read 0x0261c0 thru 0x02d9d9 (0x781a bytes) 000285 1788573021.042672 punch from 0x02e6a6 to 0x02e9b9 (0x314 bytes) 000287 1788573021.062126 mapwrite 0x01a921 thru 0x02589c (0xaf7c bytes) 000288 1788573021.171740 write 0x0307e2 thru 0x038fdf (0x87fe bytes) 000289 1788573021.186087 read 0x000372 thru 0x00d562 (0xd1f1 bytes) 000290 1788573021.203346 mapread 0x01655a thru 0x018aee (0x2595 bytes) 000291 1788573021.221270 MIRROR_RESYNC 3 mirrors 000292 1788573022.786199 mapread 0x030929 thru 0x03d6af (0xcd87 bytes) 000294 1788573022.871814 MIRROR_RESYNC 3 mirrors 000295 1788573023.502546 trunc from 0x040000 to 0x00e9a0 (0x3165f bytes) 000297 1788573023.583257 mapread 0x006db1 thru 0x00a164 (0x33b4 bytes) 000299 1788573023.605767 write 0x00b9cc thru 0x0192cd (0xd902 bytes) 000300 1788573023.607830 mapread 0x00199d thru 0x00954a (0x7bae bytes) 000301 1788573023.609199 write 0x0245ea thru 0x02982e (0x5245 bytes) 000303 1788573023.611398 write 0x024c4f thru 0x02f305 (0xa6b7 bytes) 000304 1788573023.614255 mapread 0x029196 thru 0x02b3d7 (0x2242 bytes) 000306 1788573023.616543 read 0x014ef4 thru 0x0247cf (0xf8dc bytes) 000307 1788573023.624154 mapread 0x009b75 thru 0x012f0f (0x939b bytes) 000308 1788573023.625161 trunc from 0x02f306 to 0x0098c4 (0x25a41 bytes) 000310 1788573023.649262 mapwrite 0x03fbb5 thru 0x03ffff (0x44b bytes) 000311 1788573023.724959 mapread 0x03fa2c thru 0x03ffff (0x5d4 bytes) 000313 1788573023.726476 read 0x03a8f6 thru 0x03ffff (0x570a bytes) 000314 1788573023.728053 write 0x03c5e3 thru 0x03ffff (0x3a1d bytes) 000315 1788573023.730127 mapread 0x00f864 thru 0x01cb3d (0xd2da bytes) 000316 1788573023.743829 write 0x02da60 thru 0x036dd5 (0x9376 bytes) 000317 1788573023.750526 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xeee4cf01 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xeee4cf01 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000319 1788573024.074726 write 0x03b877 thru 0x03ffff (0x4789 bytes) 000321 1788573024.089681 read 0x0162f6 thru 0x025de0 (0xfaeb bytes) 000322 1788573024.112421 write 0x019878 thru 0x02528a (0xba13 bytes) 000324 1788573024.116709 mapwrite 0x01446a thru 0x01be94 (0x7a2b bytes) 000327 1788573024.209393 trunc from 0x040000 to 0x034428 (0xbbd7 bytes) 000328 1788573024.236820 mapread 0x025a94 thru 0x025aa1 (0xe bytes) 000329 1788573024.239281 mapwrite 0x005380 thru 0x00e652 (0x92d3 bytes) 000331 1788573024.311080 trunc from 0x034429 to 0x0172a0 (0x1d188 bytes) 000332 1788573024.337134 trunc from 0x0172a1 to 0x027b85 (0x108e5 bytes) 000333 1788573024.356066 punch from 0x0253c5 to 0x027b85 (0x27c1 bytes) 000334 1788573024.362971 mapread 0x00152e thru 0x00d514 (0xbfe7 bytes) 000335 1788573024.366238 write 0x02bfb8 thru 0x02e166 (0x21af bytes) 000336 1788573024.368901 trunc from 0x02e167 to 0x0300aa (0x1f44 bytes) 000337 1788573024.394108 mapwrite 0x006b37 thru 0x015b20 (0xefea bytes) 000338 1788573024.471414 punch from 0x02e1b8 to 0x0300aa (0x1ef3 bytes) 000339 1788573024.477127 punch from 0x02b208 to 0x0300aa (0x4ea3 bytes) 000340 1788573024.483099 punch from 0x015504 to 0x0225dd (0xd0da bytes) 000343 1788573024.496788 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) 000344 1788573024.654047 mapread 0x000407 thru 0x00eb13 (0xe70d bytes) 000345 1788573024.683896 trunc from 0x0300ab to 0x02eeda (0x11d0 bytes) 000348 1788573024.699559 mapread 0x01fb19 thru 0x027248 (0x7730 bytes) 000350 1788573024.714537 MIRROR_EXTEND to 2 mirrors 000355 1788573026.287408 write 0x006e1d thru 0x0141d7 (0xd3bb bytes) 000356 1788573026.529383 trunc from 0x02eedb to 0x0335c1 (0x46e7 bytes) 000357 1788573026.558442 punch from 0x01ecc9 to 0x02234d (0x3685 bytes) 000358 1788573026.579906 write 0x00cb21 thru 0x0106b7 (0x3b97 bytes) 000360 1788573026.597828 mapread 0x007879 thru 0x00a168 (0x28f0 bytes) 000362 1788573026.619180 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x335c2): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x335c2): Mirror 2: 0xc9b65d31 (stale) CRC-32 checksum value for chunk [0, 0x335c2): Mirror 1: 0xddb58c59 Mirror 3: 0xddb58c59 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x335c2: skipped 000363 1788573028.234844 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) 000364 1788573028.727690 trunc from 0x0335c2 to 0x009261 (0x2a360 bytes) 000365 1788573028.827392 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x9262): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9262): Mirror 3: 0x370e1ea8 (stale) CRC-32 checksum value for chunk [0, 0x9262): Mirror 1: 0x370e1ea8 Mirror 4: 0x370e1ea8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9262: skipped 000366 1788573030.189059 mapread 0x004365 thru 0x009261 (0x4efd bytes) 000367 1788573030.209633 mapwrite 0x0060d1 thru 0x00cebc (0x6dec bytes) 000371 1788573030.564430 punch from 0x000d0e to 0x005200 (0x44f3 bytes) 000374 1788573030.575838 write 0x00568a thru 0x00d809 (0x8180 bytes) 000377 1788573030.585798 read 0x0031df thru 0x0087fa (0x561c bytes) 000378 1788573030.602073 write 0x012053 thru 0x01c288 (0xa236 bytes) 000379 1788573030.611680 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1c289): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c289): Mirror 3: 0xb2350d96 (stale) Mirror 4: 0xa8384e98 (stale) CRC-32 checksum value for chunk [0, 0x1c289): Mirror 1: 0x2ec9014d Mirror 5: 0x2ec9014d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c289: skipped 000380 1788573032.253871 punch from 0x011b69 to 0x01b598 (0x9a30 bytes) 000382 1788573032.413850 punch from 0x019b83 to 0x01c288 (0x2706 bytes) 000383 1788573032.430927 punch from 0x0146e4 to 0x01c288 (0x7ba5 bytes) 000384 1788573032.437422 mapwrite 0x02d2c5 thru 0x0308ec (0x3628 bytes) 000385 1788573032.562874 punch from 0x01c6ff to 0x023846 (0x7148 bytes) 000387 1788573032.570373 punch from 0x0220cc to 0x0308ec (0xe821 bytes) 000388 1788573032.579118 mapread 0x025709 thru 0x0267bb (0x10b3 bytes) 000389 1788573032.594693 write 0x013652 thru 0x01ed34 (0xb6e3 bytes) 000391 1788573032.606860 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x308ed): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x308ed): Mirror 3: 0xf237cdb5 (stale) Mirror 4: 0xf0e030d7 (stale) Mirror 5: 0x61ee7801 (stale) CRC-32 checksum value for chunk [0, 0x308ed): Mirror 1: 0x25bba4c8 Mirror 6: 0x25bba4c8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x308ed: skipped 000392 1788573034.590644 read 0x004864 thru 0x00505e (0x7fb bytes) 000393 1788573034.614830 mapwrite 0x026060 thru 0x028499 (0x243a bytes) 000394 1788573034.926262 punch from 0x0216e5 to 0x025e26 (0x4742 bytes) 000395 1788573034.937941 mapwrite 0x007968 thru 0x00e434 (0x6acd bytes) 000396 1788573035.086188 write 0x007506 thru 0x015367 (0xde62 bytes) 000397 1788573035.108168 read 0x01dae4 thru 0x02643b (0x8958 bytes) 000400 1788573035.141792 read 0x020efb thru 0x0245ff (0x3705 bytes) 000401 1788573035.158990 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x308ed): Mirror 3: 0xf237cdb5 (stale) Mirror 4: 0xf0e030d7 (stale) Mirror 6: 0x25bba4c8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x308ed): Mirror 3: 0xf237cdb5 (stale) Mirror 4: 0xf0e030d7 (stale) Mirror 6: 0x25bba4c8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x308ed: skipped 000402 1788573035.570964 read 0x000907 thru 0x00b1fe (0xa8f8 bytes) 000404 1788573035.594396 mapread 0x023e7f thru 0x026031 (0x21b3 bytes) 000405 1788573035.640058 trunc from 0x0308ed to 0x00b24a (0x256a2 bytes) 000406 1788573035.685548 mapread 0x006f5a thru 0x009e6c (0x2f13 bytes) 000408 1788573035.692741 mapwrite 0x02c52d thru 0x03a0bb (0xdb8f bytes) 000409 1788573035.798510 mapwrite 0x014883 thru 0x019be0 (0x535e bytes) 000411 1788573035.856346 read 0x003f5d thru 0x0067ac (0x2850 bytes) 000413 1788573035.858233 trunc from 0x03a0bc to 0x0253d5 (0x14ce6 bytes) 000414 1788573035.888665 write 0x014176 thru 0x023bf1 (0xfa7c bytes) 000415 1788573035.900032 mapread 0x007728 thru 0x010568 (0x8e41 bytes) 000416 1788573035.907500 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x253d6): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x253d6): Mirror 3: 0x6ec4c829 (stale) Mirror 4: 0x5bf4f019 (stale) Mirror 6: 0xd82f5efc (stale) CRC-32 checksum value for chunk [0, 0x253d6): Mirror 1: 0xec7ec9a8 Mirror 7: 0xec7ec9a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x253d6: skipped 000417 1788573037.312279 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x253d6): Mirror 4: 0x5bf4f019 (stale) Mirror 6: 0xd82f5efc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x253d6): Mirror 4: 0x5bf4f019 (stale) Mirror 6: 0xd82f5efc (stale) CRC-32 checksum value for chunk [0, 0x253d6): Mirror 1: 0xec7ec9a8 Mirror 7: 0xec7ec9a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x253d6: skipped 000420 1788573038.075810 mapwrite 0x02d746 thru 0x02d88f (0x14a bytes) 000422 1788573038.377643 read 0x0014ec thru 0x00a599 (0x90ae bytes) 000425 1788573038.396962 read 0x018e9b thru 0x0280e0 (0xf246 bytes) 000426 1788573038.421203 punch from 0x0255ac to 0x028e7c (0x38d1 bytes) 000427 1788573038.429705 trunc from 0x02d890 to 0x0233da (0xa4b5 bytes) 000428 1788573038.451169 mapread 0x018b7c thru 0x02024f (0x76d4 bytes) 000429 1788573038.457567 MIRROR_RESYNC 4 mirrors 000430 1788573040.270352 punch from 0x00729f to 0x00d527 (0x6289 bytes) 000432 1788573040.374617 trunc from 0x0233db to 0x00a7b7 (0x18c23 bytes) 000433 1788573040.426175 write 0x038890 thru 0x03ffff (0x7770 bytes) 000434 1788573040.429143 trunc from 0x040000 to 0x026e9f (0x19160 bytes) 000436 1788573040.481516 MIRROR_RESYNC 4 mirrors 000437 1788573042.311431 mapwrite 0x02d080 thru 0x039e1b (0xcd9c bytes) 000438 1788573042.673618 mapwrite 0x023fcc thru 0x02abc5 (0x6bfa bytes) 000439 1788573042.822278 punch from 0x02908d to 0x0297b0 (0x724 bytes) 000440 1788573042.835942 punch from 0x022903 to 0x02f6be (0xcdbc bytes) 000441 1788573042.854893 trunc from 0x039e1c to 0x003c24 (0x361f7 bytes) 000443 1788573042.916128 MIRROR_RESYNC 4 mirrors 000444 1788573044.019874 punch from 0x0013a7 to 0x003c24 (0x287e bytes) 000445 1788573044.188871 punch from 0x0001e5 to 0x003c24 (0x3a40 bytes) 000446 1788573044.201320 mapread 0x0001ed thru 0x003c24 (0x3a38 bytes) 000447 1788573044.256340 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c25): Mirror 6: 0x317af903 (stale) Mirror 7: 0x317af903 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c25): Mirror 6: 0x317af903 (stale) Mirror 7: 0x317af903 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c25: skipped 000448 1788573044.691997 mapread 0x001c5c thru 0x003c24 (0x1fc9 bytes) 000453 1788573044.723084 read 0x00001d thru 0x003c24 (0x3c08 bytes) 000454 1788573044.732689 mapwrite 0x02b25f thru 0x02da43 (0x27e5 bytes) 000455 1788573044.841043 punch from 0x02ce53 to 0x02da43 (0xbf1 bytes) 000456 1788573044.847103 write 0x03ff8b thru 0x03ffff (0x75 bytes) 000458 1788573044.852016 trunc from 0x040000 to 0x037967 (0x8698 bytes) 000459 1788573044.877066 punch from 0x0185fc to 0x02736d (0xed72 bytes) 000460 1788573044.885167 mapwrite 0x0042eb thru 0x0107d8 (0xc4ee bytes) 000461 1788573044.966719 punch from 0x031f9d to 0x037967 (0x59cb bytes) 000462 1788573044.972687 mapwrite 0x020809 thru 0x02c69a (0xbe92 bytes) 000463 1788573045.050849 punch from 0x021a0b to 0x0290d0 (0x76c6 bytes) 000466 1788573045.066681 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37968): Mirror 7: 0x4c0838b3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37968): Mirror 7: 0x4c0838b3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37968: skipped 000467 1788573045.468937 read 0x026c97 thru 0x02d07d (0x63e7 bytes) 000468 1788573045.482120 write 0x0353b4 thru 0x0377ca (0x2417 bytes) 000469 1788573045.513023 mapwrite 0x03c5dc thru 0x03ffff (0x3a24 bytes) 000470 1788573045.765482 mapread 0x02c576 thru 0x035d00 (0x978b bytes) 000471 1788573045.791695 mapwrite 0x0309b5 thru 0x036baa (0x61f6 bytes) 000472 1788573045.894078 trunc from 0x040000 to 0x00aa5b (0x355a4 bytes) 000473 1788573045.919038 mapread 0x0062f7 thru 0x00aa5b (0x4765 bytes) 000476 1788573045.951196 read 0x000acc thru 0x00909a (0x85cf bytes) 000477 1788573045.960760 mapread 0x006bb8 thru 0x007bf8 (0x1041 bytes) 000478 1788573045.961673 mapwrite 0x00cf82 thru 0x0110ac (0x412b bytes) 000481 1788573046.113818 read 0x0082a4 thru 0x00a6cd (0x242a bytes) 000482 1788573046.114429 read 0x0071a0 thru 0x0110ac (0x9f0d bytes) 000483 1788573046.114916 mapread 0x00062b thru 0x0031c2 (0x2b98 bytes) 000484 1788573046.117514 read 0x00c9d6 thru 0x00ca56 (0x81 bytes) 000486 1788573046.124712 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) 000491 1788573046.339703 mapwrite 0x03bb30 thru 0x03ffff (0x44d0 bytes) 000492 1788573046.467529 mapread 0x03c2d4 thru 0x03ffff (0x3d2c bytes) 000493 1788573046.470486 read 0x0210a4 thru 0x02309e (0x1ffb bytes) 000498 1788573046.479617 write 0x036ccc thru 0x03ffff (0x9334 bytes) 000499 1788573046.482220 trunc from 0x040000 to 0x033de8 (0xc217 bytes) 000500 1788573046.501086 read 0x018ffb thru 0x01e530 (0x5536 bytes) 000501 1788573046.522009 read 0x0157b9 thru 0x01fc0a (0xa452 bytes) 000503 1788573046.523261 trunc from 0x033de9 to 0x031cd9 (0x210f bytes) 000504 1788573046.543980 mapwrite 0x008d2d thru 0x01895d (0xfc31 bytes) 000506 1788573046.709127 mapwrite 0x014235 thru 0x0181f4 (0x3fc0 bytes) 000507 1788573046.810497 punch from 0x0044bb to 0x00e950 (0xa496 bytes) 000508 1788573046.819408 read 0x006910 thru 0x015d29 (0xf41a bytes) 000510 1788573046.834504 mapread 0x022282 thru 0x03141d (0xf19c bytes) 000511 1788573046.838553 read 0x021e95 thru 0x031573 (0xf6df bytes) 000513 1788573046.840942 read 0x0179b8 thru 0x0218bf (0x9f08 bytes) 000514 1788573046.854601 read 0x00d48c thru 0x018a52 (0xb5c7 bytes) 000515 1788573046.855852 punch from 0x0266d9 to 0x031cd9 (0xb601 bytes) 000516 1788573046.874610 mapwrite 0x01e4c6 thru 0x028205 (0x9d40 bytes) 000517 1788573047.001721 read 0x0026e3 thru 0x00d651 (0xaf6f bytes) 000519 1788573047.005795 write 0x017d39 thru 0x020c55 (0x8f1d bytes) 000520 1788573047.013803 mapread 0x00bd37 thru 0x01ba9a (0xfd64 bytes) 000521 1788573047.016814 punch from 0x0058fe to 0x007459 (0x1b5c bytes) 000523 1788573047.033446 write 0x03f045 thru 0x03ffff (0xfbb bytes) 000524 1788573047.035380 trunc from 0x040000 to 0x014e7c (0x2b183 bytes) 000526 1788573047.068840 write 0x0326f6 thru 0x03e818 (0xc123 bytes) 000528 1788573047.078361 mapread 0x03a127 thru 0x03e818 (0x46f2 bytes) 000529 1788573047.080394 punch from 0x023d4a to 0x031ee4 (0xe19b bytes) 000531 1788573047.104755 mapread 0x024131 thru 0x02de7a (0x9d4a bytes) 000532 1788573047.121161 mapread 0x0247c7 thru 0x032b9f (0xe3d9 bytes) truncating to largest ever: 0x3efee 000533 1788573047.126995 trunc from 0x03e819 to 0x03efed (0x7d5 bytes) 000534 1788573047.177786 mapwrite 0x006c8b thru 0x0123b3 (0xb729 bytes) 000535 1788573047.313924 mapread 0x024caa thru 0x029fa4 (0x52fb bytes) 000536 1788573047.315515 trunc from 0x03efee to 0x006289 (0x38d64 bytes) 000537 1788573047.335463 trunc from 0x00628a to 0x00bb5b (0x58d2 bytes) 000539 1788573047.360661 mapread 0x004732 thru 0x009234 (0x4b03 bytes) 000540 1788573047.382969 trunc from 0x00bb5c to 0x03030e (0x247b3 bytes) 000542 1788573047.404811 write 0x01438a thru 0x016a93 (0x270a bytes) 000543 1788573047.419016 trunc from 0x03030f to 0x00298e (0x2d980 bytes) 000545 1788573047.446121 mapwrite 0x020162 thru 0x021374 (0x1213 bytes) 000547 1788573047.536129 mapread 0x0157a0 thru 0x01d2ff (0x7b60 bytes) 000549 1788573047.602108 trunc from 0x021375 to 0x01961c (0x7d58 bytes) 000550 1788573047.630092 mapwrite 0x00e6d9 thru 0x016aa6 (0x83ce bytes) 000551 1788573047.814716 mapwrite 0x001dd3 thru 0x00c542 (0xa770 bytes) 000552 1788573048.078122 mapread 0x00b7cc thru 0x01682c (0xb061 bytes) 000554 1788573048.089346 trunc from 0x01961d to 0x00cd87 (0xc895 bytes) 000555 1788573048.131142 punch from 0x001019 to 0x00cd87 (0xbd6f bytes) 000556 1788573048.139415 read 0x007a04 thru 0x00c556 (0x4b53 bytes) 000557 1788573048.158588 punch from 0x00b02a to 0x00b7ce (0x7a5 bytes) 000558 1788573048.189776 MIRROR_EXTEND to 2 mirrors 000559 1788573049.275766 trunc from 0x00cd88 to 0x03a61f (0x2d898 bytes) 000561 1788573049.557553 punch from 0x00a266 to 0x00d24e (0x2fe9 bytes) 000562 1788573049.581053 mapwrite 0x01956b thru 0x022df3 (0x9889 bytes) 000564 1788573049.760246 mapwrite 0x023dac thru 0x02602a (0x227f bytes) 000565 1788573049.883541 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3a620): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a620): Mirror 2: 0xe0df3c7 (stale) CRC-32 checksum value for chunk [0, 0x3a620): Mirror 1: 0xc8dd3858 Mirror 3: 0xc8dd3858 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a620: skipped 000566 1788573051.520154 mapread 0x02d506 thru 0x033777 (0x6272 bytes) 000569 1788573051.557043 punch from 0x014ecb to 0x021f1f (0xd055 bytes) 000570 1788573051.798634 mapwrite 0x0076ea thru 0x00b6e6 (0x3ffd bytes) 000571 1788573051.959685 mapread 0x0093ec thru 0x00ea10 (0x5625 bytes) 000572 1788573051.961160 trunc from 0x03a620 to 0x0261db (0x14444 bytes) 000573 1788573052.004275 write 0x03724d thru 0x03ffff (0x8db3 bytes) 000574 1788573052.008133 trunc from 0x040000 to 0x030a19 (0xf5e6 bytes) 000575 1788573052.037639 trunc from 0x030a1a to 0x033da1 (0x3388 bytes) 000578 1788573052.085323 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33da2): Mirror 3: 0x3e3599da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33da2): Mirror 3: 0x3e3599da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33da2: skipped 000579 1788573052.432242 write 0x002ed4 thru 0x00884b (0x5978 bytes) 000583 1788573052.456083 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) 000585 1788573052.613654 read 0x022927 thru 0x02a6b1 (0x7d8b bytes) 000588 1788573052.625194 mapread 0x00ffa5 thru 0x016b95 (0x6bf1 bytes) 000590 1788573052.649397 mapread 0x02a560 thru 0x02ac1a (0x6bb bytes) 000591 1788573052.650240 trunc from 0x033da2 to 0x03a22a (0x6489 bytes) 000593 1788573052.674264 write 0x02acca thru 0x0336e4 (0x8a1b bytes) 000594 1788573052.682194 trunc from 0x03a22b to 0x000880 (0x399aa bytes) 000598 1788573052.705707 write 0x02b4de thru 0x03985c (0xe37f bytes) 000599 1788573052.709064 mapwrite 0x02bdec thru 0x0380ff (0xc314 bytes) 000600 1788573052.766445 punch from 0x023a65 to 0x02c2e7 (0x8883 bytes) 000601 1788573052.772709 read 0x01e618 thru 0x02e2c0 (0xfca9 bytes) 000602 1788573052.783275 read 0x021bf8 thru 0x02975a (0x7b63 bytes) 000603 1788573052.784270 punch from 0x019c29 to 0x027295 (0xd66d bytes) 000604 1788573052.794406 write 0x014ffb thru 0x015347 (0x34d bytes) 000606 1788573052.798657 punch from 0x0126c1 to 0x01baed (0x942d bytes) 000607 1788573052.809584 read 0x009871 thru 0x00c38f (0x2b1f bytes) 000609 1788573052.811153 mapread 0x026e25 thru 0x0284e4 (0x16c0 bytes) 000610 1788573052.822501 mapwrite 0x00983c thru 0x015ffe (0xc7c3 bytes) 000613 1788573052.890517 write 0x00011a thru 0x00376f (0x3656 bytes) 000614 1788573052.891501 trunc from 0x03985d to 0x032753 (0x7109 bytes) 000615 1788573052.973036 write 0x02c2b4 thru 0x02e849 (0x2596 bytes) 000616 1788573052.974907 trunc from 0x032754 to 0x009f32 (0x28821 bytes) 000620 1788573053.005059 trunc from 0x009f33 to 0x0185fe (0xe6cc bytes) 000621 1788573053.038346 write 0x015d2b thru 0x02527d (0xf553 bytes) 000623 1788573053.054391 punch from 0x022daf to 0x024463 (0x16b5 bytes) 000625 1788573053.081657 write 0x0161ca thru 0x0219cd (0xb804 bytes) 000627 1788573053.087840 mapread 0x020dd9 thru 0x02527d (0x44a5 bytes) 000628 1788573053.116363 write 0x01d8da thru 0x01ec3f (0x1366 bytes) 000629 1788573053.117263 punch from 0x01fb30 to 0x02527d (0x574e bytes) 000630 1788573053.140672 write 0x036022 thru 0x03f6b9 (0x9698 bytes) 000631 1788573053.152668 write 0x010d09 thru 0x013af6 (0x2dee bytes) 000633 1788573053.187038 trunc from 0x03f6ba to 0x02bf6e (0x1374b bytes) 000634 1788573053.207974 punch from 0x000c15 to 0x004fe7 (0x43d3 bytes) 000635 1788573053.221338 mapread 0x000d94 thru 0x00fed7 (0xf144 bytes) 000636 1788573053.273843 write 0x033aa9 thru 0x0396f2 (0x5c4a bytes) 000638 1788573053.276038 punch from 0x028750 to 0x02db10 (0x53c1 bytes) 000639 1788573053.288426 read 0x034e1c thru 0x038a87 (0x3c6c bytes) 000640 1788573053.289179 mapwrite 0x012baa thru 0x0135a2 (0x9f9 bytes) 000641 1788573053.355121 trunc from 0x0396f3 to 0x039516 (0x1dc bytes) 000642 1788573053.401571 trunc from 0x039517 to 0x01c463 (0x1d0b3 bytes) 000643 1788573053.419569 trunc from 0x01c464 to 0x00a8f9 (0x11b6a bytes) 000644 1788573053.450069 mapread 0x004a29 thru 0x00a8f9 (0x5ed1 bytes) 000646 1788573053.458813 punch from 0x008e81 to 0x00a8f9 (0x1a79 bytes) 000648 1788573053.473747 read 0x0051bf thru 0x00a8f9 (0x573b bytes) 000649 1788573053.482842 trunc from 0x00a8fa to 0x037ac7 (0x2d1ce bytes) 000650 1788573053.516873 mapwrite 0x03105b thru 0x031a98 (0xa3e bytes) 000653 1788573053.626879 trunc from 0x037ac8 to 0x038488 (0x9c1 bytes) 000654 1788573053.648957 mapwrite 0x0026f7 thru 0x011fc5 (0xf8cf bytes) 000655 1788573053.725443 mapread 0x032276 thru 0x03678a (0x4515 bytes) 000656 1788573053.726314 read 0x037ac8 thru 0x038488 (0x9c1 bytes) 000658 1788573053.733891 mapread 0x01790e thru 0x025e5c (0xe54f bytes) 000660 1788573053.737926 trunc from 0x038489 to 0x019275 (0x1f213 bytes) 000664 1788573053.767035 mapread 0x00b037 thru 0x010cb0 (0x5c7a bytes) 000667 1788573053.772905 read 0x00a405 thru 0x00a883 (0x47f bytes) 000668 1788573053.774804 trunc from 0x019276 to 0x00f1ca (0xa0ab bytes) 000669 1788573053.809183 read 0x007ddb thru 0x00a0a2 (0x22c8 bytes) 000670 1788573053.813589 write 0x03744a thru 0x03980e (0x23c5 bytes) 000671 1788573053.817512 trunc from 0x03980f to 0x00c521 (0x2d2ed bytes) 000673 1788573053.853507 mapwrite 0x0147b0 thru 0x018a47 (0x4298 bytes) 000674 1788573054.059792 trunc from 0x018a48 to 0x0256a6 (0xcc5f bytes) 000677 1788573054.078166 read 0x011921 thru 0x015fd4 (0x46b4 bytes) 000678 1788573054.079027 read 0x00ff90 thru 0x01a9c1 (0xaa32 bytes) 000679 1788573054.087955 write 0x031de6 thru 0x0374eb (0x5706 bytes) 000680 1788573054.089951 read 0x016910 thru 0x01f0a1 (0x8792 bytes) 000682 1788573054.091660 trunc from 0x0374ec to 0x027088 (0x10463 bytes) 000683 1788573054.127450 MIRROR_EXTEND to 2 mirrors 000684 1788573055.201019 trunc from 0x027089 to 0x025e39 (0x124f bytes) 000685 1788573055.400973 trunc from 0x025e3a to 0x017bd3 (0xe266 bytes) 000686 1788573055.424452 mapread 0x0124ed thru 0x0130a6 (0xbba bytes) 000687 1788573055.436652 trunc from 0x017bd4 to 0x0281be (0x105eb bytes) 000690 1788573055.464131 MIRROR_RESYNC 2 mirrors 000691 1788573056.712377 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) 000693 1788573057.010091 read 0x022b35 thru 0x023c8f (0x115b bytes) 000696 1788573057.042058 MIRROR_EXTEND to 2 mirrors 000698 1788573058.373799 write 0x02e111 thru 0x037614 (0x9504 bytes) 000699 1788573058.529774 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37615): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37615): Mirror 2: 0xb2545ec5 (stale) CRC-32 checksum value for chunk [0, 0x37615): Mirror 1: 0xd3357581 Mirror 3: 0xd3357581 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37615: skipped 000700 1788573059.639888 read 0x002d10 thru 0x00db87 (0xae78 bytes) 000701 1788573059.648787 punch from 0x0126cb to 0x017261 (0x4b97 bytes) 000702 1788573059.731471 mapwrite 0x013400 thru 0x0228e8 (0xf4e9 bytes) 000703 1788573059.840143 mapwrite 0x0191ed thru 0x0279d2 (0xe7e6 bytes) 000704 1788573059.957330 write 0x00b4b4 thru 0x0179af (0xc4fc bytes) 000705 1788573059.968510 mapwrite 0x037fd2 thru 0x03f65f (0x768e bytes) 000708 1788573060.061822 write 0x011dbb thru 0x0213ba (0xf600 bytes) 000709 1788573060.063880 read 0x019c1d thru 0x029b2e (0xff12 bytes) 000711 1788573060.066118 read 0x028155 thru 0x038027 (0xfed3 bytes) 000713 1788573060.067793 mapread 0x03bdc8 thru 0x03db2b (0x1d64 bytes) 000715 1788573060.069343 mapwrite 0x02fea6 thru 0x03f848 (0xf9a3 bytes) 000717 1788573060.156917 MIRROR_RESYNC 3 mirrors 000718 1788573061.717123 write 0x026088 thru 0x028a4b (0x29c4 bytes) 000719 1788573061.813535 trunc from 0x03f849 to 0x035445 (0xa403 bytes) 000720 1788573061.836863 mapwrite 0x0088c7 thru 0x00c6a1 (0x3ddb bytes) 000722 1788573061.922744 write 0x023774 thru 0x0334e5 (0xfd72 bytes) 000723 1788573061.935790 read 0x0230fd thru 0x0240a1 (0xfa5 bytes) 000724 1788573061.937335 punch from 0x00ce4f to 0x0194be (0xc670 bytes) 000727 1788573061.957649 mapread 0x01b644 thru 0x024b36 (0x94f3 bytes) 000728 1788573061.969414 write 0x0175e4 thru 0x01f650 (0x806d bytes) 000730 1788573061.971980 read 0x03398f thru 0x035445 (0x1ab7 bytes) 000732 1788573061.984395 trunc from 0x035446 to 0x020d53 (0x146f2 bytes) 000733 1788573062.004251 punch from 0x00ce84 to 0x00dc08 (0xd85 bytes) 000734 1788573062.018396 mapread 0x01a855 thru 0x01db8d (0x3339 bytes) 000735 1788573062.020433 read 0x01d812 thru 0x020d53 (0x3542 bytes) 000736 1788573062.021530 read 0x01e64b thru 0x020d53 (0x2709 bytes) 000737 1788573062.023056 mapwrite 0x03d419 thru 0x03ffff (0x2be7 bytes) 000738 1788573062.123648 write 0x0154b8 thru 0x0208f9 (0xb442 bytes) 000739 1788573062.124962 read 0x031718 thru 0x03ba6e (0xa357 bytes) 000740 1788573062.126531 mapread 0x03d17b thru 0x03ffff (0x2e85 bytes) 000742 1788573062.128506 punch from 0x0280fa to 0x02c2ae (0x41b5 bytes) 000743 1788573062.144785 read 0x01e792 thru 0x026aaf (0x831e bytes) 000744 1788573062.145592 punch from 0x00f04b to 0x01387b (0x4831 bytes) 000745 1788573062.151055 write 0x027209 thru 0x02a6b1 (0x34a9 bytes) 000747 1788573062.161687 mapread 0x0318b8 thru 0x033b07 (0x2250 bytes) 000748 1788573062.162613 punch from 0x013ef5 to 0x015531 (0x163d bytes) 000749 1788573062.175882 mapread 0x013695 thru 0x02057c (0xcee8 bytes) 000750 1788573062.201941 read 0x02e255 thru 0x02ed67 (0xb13 bytes) 000751 1788573062.203199 trunc from 0x040000 to 0x00157b (0x3ea84 bytes) 000752 1788573062.226848 write 0x018b52 thru 0x01c5cb (0x3a7a bytes) 000754 1788573062.235125 punch from 0x00647f to 0x00f900 (0x9482 bytes) 000755 1788573062.250528 trunc from 0x01c5cc to 0x012e56 (0x9775 bytes) 000756 1788573062.275724 punch from 0x005c17 to 0x010b86 (0xaf70 bytes) 000757 1788573062.282096 mapwrite 0x0206a4 thru 0x027b73 (0x74d0 bytes) 000759 1788573062.387099 trunc from 0x027b74 to 0x006188 (0x219eb bytes) 000760 1788573062.416847 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6189): Mirror 3: 0x41a5752b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6189): Mirror 3: 0x41a5752b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6189: skipped 000761 1788573062.689864 write 0x00308d thru 0x006bfd (0x3b71 bytes) 000762 1788573062.699257 write 0x01724f thru 0x01938c (0x213e bytes) 000763 1788573062.700314 trunc from 0x01938d to 0x03e4cb (0x2513f bytes) 000764 1788573062.714131 trunc from 0x03e4cc to 0x02ff72 (0xe559 bytes) 000766 1788573062.734250 punch from 0x02a457 to 0x02ff72 (0x5b1c bytes) 000767 1788573062.748156 mapread 0x001916 thru 0x00ba8f (0xa17a bytes) 000768 1788573062.767782 mapwrite 0x003b94 thru 0x011773 (0xdbe0 bytes) 000772 1788573062.837826 punch from 0x0155f2 to 0x0169a3 (0x13b2 bytes) 000773 1788573062.848380 mapwrite 0x035c54 thru 0x03a77d (0x4b2a bytes) 000774 1788573062.954624 mapread 0x00d691 thru 0x00df41 (0x8b1 bytes) 000775 1788573062.955939 mapread 0x00112f thru 0x005624 (0x44f6 bytes) 000776 1788573062.958957 read 0x02f901 thru 0x03a77d (0xae7d bytes) 000777 1788573062.960204 write 0x03dad5 thru 0x03fa74 (0x1fa0 bytes) 000778 1788573062.962133 read 0x00821f thru 0x00d6aa (0x548c bytes) 000779 1788573062.964089 punch from 0x01f3e5 to 0x02323b (0x3e57 bytes) 000780 1788573062.980973 write 0x026245 thru 0x0325a3 (0xc35f bytes) 000781 1788573062.995604 mapwrite 0x027c02 thru 0x02e132 (0x6531 bytes) 000783 1788573063.068428 punch from 0x03a9c4 to 0x03fa74 (0x50b1 bytes) 000784 1788573063.074646 trunc from 0x03fa75 to 0x034067 (0xba0d bytes) 000785 1788573063.095865 mapread 0x00eb3d thru 0x0161c3 (0x7687 bytes) 000787 1788573063.099068 mapwrite 0x01c826 thru 0x0224a7 (0x5c82 bytes) 000788 1788573063.176905 punch from 0x00ab00 to 0x016c59 (0xc15a bytes) 000790 1788573063.192137 mapwrite 0x021f61 thru 0x02eced (0xcd8d bytes) 000791 1788573063.318313 mapread 0x005aca thru 0x008285 (0x27bc bytes) 000792 1788573063.320026 write 0x03774b thru 0x03dc6d (0x6523 bytes) 000794 1788573063.323338 trunc from 0x03dc6e to 0x00d03e (0x30c2f bytes) 000795 1788573063.366264 punch from 0x0035c9 to 0x007f5b (0x4993 bytes) 000796 1788573063.379427 MIRROR_RESYNC 2 mirrors 000797 1788573064.560280 mapwrite 0x013f6a thru 0x01416e (0x205 bytes) 000798 1788573064.699461 mapread 0x002eed thru 0x006761 (0x3875 bytes) 000799 1788573064.711823 write 0x0305f0 thru 0x0344c8 (0x3ed9 bytes) 000800 1788573064.716766 read 0x02f069 thru 0x0344c8 (0x5460 bytes) 000801 1788573064.722595 punch from 0x032387 to 0x0344c8 (0x2142 bytes) 000804 1788573064.738823 mapread 0x020331 thru 0x026a6c (0x673c bytes) 000805 1788573064.754987 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x344c9): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x344c9): Mirror 3: 0x7c89e9be (stale) CRC-32 checksum value for chunk [0, 0x344c9): Mirror 1: 0xd31c8069 Mirror 4: 0xd31c8069 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x344c9: skipped 000807 1788573066.359115 trunc from 0x0344c9 to 0x033b99 (0x92f bytes) 000808 1788573066.477246 mapread 0x0045bb thru 0x0045e3 (0x29 bytes) 000811 1788573066.485757 read 0x018455 thru 0x0197cc (0x1378 bytes) 000813 1788573066.491143 write 0x0233d9 thru 0x025cff (0x2927 bytes) 000814 1788573066.508281 MIRROR_RESYNC 3 mirrors 000815 1788573068.580077 read 0x00691b thru 0x00c615 (0x5cfb bytes) 000816 1788573068.589200 mapwrite 0x03ca46 thru 0x03ffff (0x35ba bytes) 000817 1788573068.786191 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x2ebb4226 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x2ebb4226 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000818 1788573069.100209 trunc from 0x040000 to 0x02a2a2 (0x15d5d bytes) 000820 1788573069.118735 read 0x013ef5 thru 0x01eec4 (0xafd0 bytes) 000823 1788573069.129376 write 0x034a84 thru 0x03f7ed (0xad6a bytes) 000824 1788573069.131979 trunc from 0x03f7ee to 0x00bd9d (0x33a50 bytes) 000825 1788573069.148733 read 0x0018aa thru 0x00bd9d (0xa4f4 bytes) 000826 1788573069.157910 write 0x00d211 thru 0x017ec9 (0xacb9 bytes) 000830 1788573069.165623 write 0x00371d thru 0x00be45 (0x8729 bytes) 000832 1788573069.168155 trunc from 0x017eca to 0x0161cd (0x1cfc bytes) 000836 1788573069.198401 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) 000837 1788573069.352263 mapwrite 0x025f07 thru 0x0280b6 (0x21b0 bytes) 000838 1788573069.435610 trunc from 0x0280b7 to 0x018b89 (0xf52d bytes) 000839 1788573069.457429 trunc from 0x018b8a to 0x013163 (0x5a26 bytes) 000843 1788573069.489130 mapread 0x00229d thru 0x00c202 (0x9f66 bytes) 000844 1788573069.508593 trunc from 0x013164 to 0x002895 (0x108ce bytes) 000845 1788573069.534632 write 0x006845 thru 0x00933d (0x2af9 bytes) 000846 1788573069.537726 trunc from 0x00933e to 0x030f52 (0x27c15 bytes) 000847 1788573069.566712 trunc from 0x030f53 to 0x03a76f (0x981d bytes) 000848 1788573069.591687 mapwrite 0x0194c3 thru 0x023991 (0xa4cf bytes) 000850 1788573069.693332 punch from 0x00a1bd to 0x00e771 (0x45b5 bytes) 000851 1788573069.705675 write 0x026236 thru 0x029a96 (0x3861 bytes) 000852 1788573069.707791 trunc from 0x03a770 to 0x031e45 (0x892a bytes) 000853 1788573069.749272 mapread 0x026c66 thru 0x031e45 (0xb1e0 bytes) 000854 1788573069.773964 read 0x00340a thru 0x00d6a4 (0xa29b bytes) 000858 1788573069.791533 mapread 0x007a7c thru 0x00b654 (0x3bd9 bytes) 000859 1788573069.792646 punch from 0x01a79e to 0x01b39f (0xc02 bytes) 000861 1788573069.802950 punch from 0x01704e to 0x018fc0 (0x1f73 bytes) 000862 1788573069.808201 read 0x020fa2 thru 0x0234af (0x250e bytes) 000863 1788573069.809609 mapread 0x00dc98 thru 0x01650b (0x8874 bytes) 000866 1788573069.825689 read 0x011f3a thru 0x01a833 (0x88fa bytes) 000867 1788573069.827221 mapwrite 0x020e15 thru 0x02522f (0x441b bytes) 000870 1788573069.899381 write 0x00ea71 thru 0x00fdd5 (0x1365 bytes) 000871 1788573069.901812 mapread 0x001043 thru 0x00c39f (0xb35d bytes) 000875 1788573069.905962 mapwrite 0x033104 thru 0x035137 (0x2034 bytes) 000878 1788573070.002612 mapread 0x0128f2 thru 0x0195ba (0x6cc9 bytes) 000879 1788573070.004581 mapread 0x00dd82 thru 0x01d9ee (0xfc6d bytes) 000880 1788573070.006097 mapwrite 0x03a033 thru 0x03f94d (0x591b bytes) 000882 1788573070.093104 mapread 0x003885 thru 0x00e013 (0xa78f bytes) 000884 1788573070.095802 mapwrite 0x022222 thru 0x02d809 (0xb5e8 bytes) 000885 1788573070.152363 trunc from 0x03f94e to 0x00b1fd (0x34750 bytes) 000887 1788573070.183896 write 0x028188 thru 0x03045c (0x82d5 bytes) 000888 1788573070.186145 mapwrite 0x018634 thru 0x01d38e (0x4d5b bytes) 000889 1788573070.323160 trunc from 0x03045d to 0x0133be (0x1d09e bytes) 000891 1788573070.355956 mapread 0x00cc87 thru 0x0128fb (0x5c75 bytes) 000893 1788573070.394856 mapwrite 0x037e6a thru 0x03bb96 (0x3d2d bytes) 000894 1788573070.500071 punch from 0x00fef9 to 0x01f2ed (0xf3f5 bytes) 000895 1788573070.514224 write 0x01b914 thru 0x027252 (0xb93f bytes) 000896 1788573070.545733 write 0x03917c thru 0x03ffff (0x6e84 bytes) 000898 1788573070.550643 trunc from 0x040000 to 0x017dad (0x28252 bytes) 000899 1788573070.580262 mapwrite 0x028247 thru 0x036a7f (0xe839 bytes) 000900 1788573070.742163 read 0x0148b8 thru 0x0212d9 (0xca22 bytes) 000901 1788573070.753358 punch from 0x02a3eb to 0x032cde (0x88f4 bytes) 000906 1788573070.765522 MIRROR_EXTEND to 2 mirrors 000907 1788573072.030054 read 0x02804f thru 0x033d56 (0xbd08 bytes) 000908 1788573072.051277 MIRROR_RESYNC 2 mirrors 000912 1788573072.593853 read 0x00f8f2 thru 0x01ae87 (0xb596 bytes) 000913 1788573072.610093 mapwrite 0x03ffc4 thru 0x03ffff (0x3c bytes) 000914 1788573072.798150 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) 000916 1788573073.019818 punch from 0x004428 to 0x00aa05 (0x65de bytes) 000917 1788573073.027545 punch from 0x016100 to 0x0222e0 (0xc1e1 bytes) 000918 1788573073.032998 trunc from 0x040000 to 0x015b37 (0x2a4c8 bytes) truncating to largest ever: 0x3f12b 000920 1788573073.050904 trunc from 0x015b38 to 0x03f12a (0x295f3 bytes) 000921 1788573073.069792 mapwrite 0x032d7c thru 0x036be8 (0x3e6d bytes) 000924 1788573073.132727 write 0x02f4b6 thru 0x038cbb (0x9806 bytes) 000927 1788573073.136138 punch from 0x006ac4 to 0x013aaf (0xcfec bytes) 000928 1788573073.148693 trunc from 0x03f12b to 0x022a6d (0x1c6bd bytes) 000930 1788573073.168462 read 0x00dfbf thru 0x0104b5 (0x24f7 bytes) 000931 1788573073.176214 read 0x00796a thru 0x00d4a2 (0x5b39 bytes) 000932 1788573073.186754 mapread 0x01b038 thru 0x01b948 (0x911 bytes) 000935 1788573073.189924 read 0x002671 thru 0x00d34a (0xacda bytes) 000936 1788573073.190956 read 0x0056d4 thru 0x00de4d (0x877a bytes) 000937 1788573073.191628 read 0x0063e8 thru 0x007821 (0x143a bytes) 000938 1788573073.192611 punch from 0x00aad3 to 0x0152b0 (0xa7de bytes) 000939 1788573073.197599 write 0x01a068 thru 0x0253ec (0xb385 bytes) 000941 1788573073.200098 write 0x03cf78 thru 0x03ffff (0x3088 bytes) 000946 1788573073.204777 punch from 0x006b10 to 0x00c5a5 (0x5a96 bytes) 000950 1788573073.226003 trunc from 0x040000 to 0x02ca57 (0x135a8 bytes) 000953 1788573073.239051 mapwrite 0x039143 thru 0x03cca6 (0x3b64 bytes) 000954 1788573073.312093 mapwrite 0x011736 thru 0x018485 (0x6d50 bytes) 000957 1788573073.372447 MIRROR_EXTEND to 2 mirrors 000958 1788573074.575585 punch from 0x01bb9d to 0x0286f3 (0xcb57 bytes) 000959 1788573074.709975 mapread 0x03540f thru 0x03cca6 (0x7898 bytes) 000960 1788573074.733083 mapread 0x00a7e8 thru 0x00df7f (0x3798 bytes) 000961 1788573074.744213 write 0x025b22 thru 0x034ab3 (0xef92 bytes) 000962 1788573074.752190 mapwrite 0x01e53e thru 0x0278c3 (0x9386 bytes) 000963 1788573074.818445 mapwrite 0x0163a1 thru 0x020d9a (0xa9fa bytes) 000964 1788573074.893718 write 0x029ed6 thru 0x0324f1 (0x861c bytes) 000965 1788573074.896467 punch from 0x0236f6 to 0x02aa02 (0x730d bytes) 000968 1788573074.920563 mapread 0x027378 thru 0x02843a (0x10c3 bytes) 000970 1788573074.958713 read 0x029079 thru 0x02d1c3 (0x414b bytes) 000971 1788573074.961692 mapwrite 0x0217e5 thru 0x02362a (0x1e46 bytes) 000972 1788573075.025175 trunc from 0x03cca7 to 0x031c96 (0xb010 bytes) 000975 1788573075.066350 read 0x02b699 thru 0x031c96 (0x65fe bytes) 000977 1788573075.069246 mapread 0x0067ff thru 0x01166c (0xae6e bytes) 000979 1788573075.077909 trunc from 0x031c97 to 0x017581 (0x1a715 bytes) 000980 1788573075.099251 trunc from 0x017582 to 0x038586 (0x21005 bytes) 000981 1788573075.126097 read 0x032f85 thru 0x038586 (0x5602 bytes) 000982 1788573075.156242 MIRROR_RESYNC 2 mirrors 000984 1788573076.353566 mapread 0x012764 thru 0x0156b6 (0x2f53 bytes) 000985 1788573076.391274 read 0x015716 thru 0x0199e8 (0x42d3 bytes) 000986 1788573076.392572 mapread 0x029fcf thru 0x034da9 (0xaddb bytes) 000987 1788573076.438119 punch from 0x01f1d3 to 0x02c386 (0xd1b4 bytes) 000989 1788573076.537389 write 0x00b37f thru 0x0119fc (0x667e bytes) 000990 1788573076.549170 trunc from 0x038587 to 0x029468 (0xf11e bytes) 000991 1788573076.601148 trunc from 0x029469 to 0x000a77 (0x289f1 bytes) 000992 1788573076.625181 MIRROR_RESYNC 2 mirrors 000993 1788573077.238460 read 0x0009fc thru 0x000a77 (0x7c bytes) 000994 1788573077.245087 read 0x000832 thru 0x000a77 (0x246 bytes) 000995 1788573077.245533 mapread 0x0008b8 thru 0x000a77 (0x1c0 bytes) 000996 1788573077.246457 punch from 0x000a3a to 0x000a77 (0x3e bytes) 000997 1788573077.311869 mapread 0x00074e thru 0x000a77 (0x32a bytes) 000998 1788573077.317752 read 0x00016c thru 0x000a77 (0x90c bytes) All operations completed A-OK! PASS 70a (92s) == sanity-flr test 71: check mirror extend parent fid ==== 21:51:23 (1788573083) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 21:51:26 (1788573086) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.310092 s, 6.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21075 s, 5.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28b:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28b:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x290:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 21:51:36 (1788573096) Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre2 Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 1101056 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7828352 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6597280 .. Extending file size to 8524992 .. Extending file size to 8377344 .. Extending file size to 8894560 .. Extending file size to 8537056 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9148736 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57222 57223 57225 57228 57230 done 192.168.204.151@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg451-client.virtnet /mnt/lustre2 (opts:) 192.168.204.151@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg451-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (86s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 21:53:02 (1788573182) Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre2 Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5214496 .. Extending file size to 7510304 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8172032 .. Extending file size to 7699488 .. Extending file size to 8279680 .. Extending file size to 8921088 .. Extending file size to 8287136 .. Extending file size to 9068224 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8411200 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 58915 58917 58918 58921 58922 192.168.204.151@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg451-client.virtnet /mnt/lustre2 (opts:) 192.168.204.151@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg451-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (85s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 21:54:27 (1788573267) Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.213855 s, 9.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.639446 s, 3.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x47b29d5d (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.151@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg451-client.virtnet /mnt/lustre2 (opts:) PASS 200c (23s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 21:54:53 (1788573293) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.120192 s, 17.4 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 21:55:02 (1788573302) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b2:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a1:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a1:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a2:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2a5:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 21:55:11 (1788573311) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.388427 s, 5.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 21:55:21 (1788573321) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.312036 s, 6.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a9:0x0] } PASS 204b (11s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 21:55:32 (1788573332) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.3385 s, 6.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2aa:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2aa:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } PASS 204c (37s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 21:56:10 (1788573370) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg451-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968465 s, 10.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a8:0x0] } PASS 204d (63s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 21:57:13 (1788573433) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg451-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2aa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.295515 s, 7.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2aa:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2aa:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (63s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 21:58:16 (1788573496) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg451-server mds-ost sync done. Creating new pool test_204f oleg451-server: Pool lustre.test_204f created Adding targets to pool oleg451-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg451-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.351098 s, 6.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2af:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2af:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg451-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg451-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg451-server: Pool lustre.test_204f destroyed PASS 204f (84s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 21:59:40 (1788573580) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 21:59:49 (1788573589) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 21:59:58 (1788573598) oleg451-server: Pool lustre.test_206 created Adding targets to pool oleg451-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg451-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg451-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg451-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg451-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 22:00:25 (1788573625) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102479 s, 10.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630665 s, 16.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xcb14ede8 Mirror 3: 0xcb14ede8 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 22:00:38 (1788573638) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 22:00:42 (1788573642) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 22:00:46 (1788573646) Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.140884 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xea0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1f90000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1f90000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1f90000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1810000 Read this (4 bytes): BBBB 192.168.204.151@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg451-client.virtnet /mnt/lustre2 (opts:) PASS 209a (40s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 22:01:26 (1788573686) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0439483 s, 186 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 22:01:34 (1788573694) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741578 s, 14.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1d7 lmm_fid: [0x200000404:0x1d7:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1d7 lmm_fid: [0x200000404:0x1d7:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834391 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 22:01:36.000000000 -0400 Modify: 2026-09-04 22:01:36.000000000 -0400 Change: 2026-09-04 22:01:38.000000000 -0400 Birth: 2026-09-04 22:01:36.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 22:01:43 (1788573703) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 22:01:48 (1788573708) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.815631 s, 50.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.164838 s, 24.8 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 4 22:01 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d9:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 4 22:01 /mnt/lustre/f211.sanity-flr PASS 211 (12s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 22:02:00 (1788573720) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (12s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 22:02:12 (1788573732) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1816 sec =========== 22:02:22 (1788573742) === sanity-flr: start cleanup 22:02:24 (1788573744) === === sanity-flr: finish cleanup 22:02:31 (1788573751) ===