-----============= acceptance-small: sanity-flr ============----- Tue Sep 8 08:10:22 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg332-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 08:10:41 (1788869441) === oleg332-client.virtnet: executing check_config_client /mnt/lustre oleg332-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg332-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9e9190ac6800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e9190ac6800.idle_timeout=debug disable quota as required oleg332-server: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 08:11:20 (1788869480) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7636] == sanity-flr test 0a: lfs mirror create with -N option == 08:11:23 (1788869483) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 08:11:42 (1788869502) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 08:11:46 (1788869506) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 08:11:50 (1788869510) PASS 0d (29s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 08:12:19 (1788869539) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 08:12:23 (1788869543) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 08:12:27 (1788869547) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9e9190ac6800.stats=clear osc.lustre-OST0001-osc-ffff9e9190ac6800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 08:12:35 (1788869555) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 08:12:47 (1788869567) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 08:12:57 (1788869577) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 08:13:01 (1788869581) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 08:13:10 (1788869590) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 08:13:19 (1788869599) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 08:13:28 (1788869608) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 08:13:32 (1788869612) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 08:13:39 (1788869619) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 08:13:50 (1788869630) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 08:13:58 (1788869638) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.794375 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114085 s, 9.2 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 08:14:09 (1788869649) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 0.398734 s, 36.8 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 08:14:18 (1788869658) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 08:14:30 (1788869670) Stopping /mnt/lustre-ost1 (opts:) on oleg332-server oleg332-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e9190ac6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e9190ac6800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg332-server: oleg332-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg332-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg332-server oleg332-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9e9190ac6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e9190ac6800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg332-server: oleg332-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg332-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (81s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 08:15:51 (1788869751) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.387218 s, 16.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg332-server oleg332-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e9190ac6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e9190ac6800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg332-server: oleg332-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg332-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (52s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 08:16:43 (1788869803) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg332-server oleg332-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e9190ac6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e9190ac6800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg332-server: oleg332-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg332-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg332-server oleg332-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9e9190ac6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e9190ac6800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg332-server: oleg332-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg332-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (87s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 08:18:10 (1788869890) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.58074 s, 9.4 MB/s -rw-r--r-- 1 root root 52428800 Sep 8 08:18 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.200s user 0m0.000s sys 0m0.153s Stopping /mnt/lustre-ost1 (opts:) on oleg332-server oleg332-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e9190ac6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e9190ac6800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.248s user 0m0.001s sys 0m0.165s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg332-server: oleg332-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg332-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg332-server oleg332-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9e9190ac6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e9190ac6800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.463s user 0m0.001s sys 0m0.188s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg332-server: oleg332-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg332-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg332-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e9190ac6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e9190ac6800.ost_server_uuid in FULL state after 0 sec PASS 33b (98s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 08:19:49 (1788869989) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 08:19:52 (1788869992) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 08:19:56 (1788869996) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 08:19:59 (1788869999) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 08:20:06 (1788870006) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.573496 s, 21.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 08:20:20 (1788870020) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446899 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222547 s, 47.1 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 08:20:27 (1788870027) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 08:20:35 (1788870035) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000782198 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0025146 s, 407 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00215226 s, 476 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00248186 s, 413 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 08:20:45 (1788870045) create test files with size 15747094 .. cda74ac8cd8ed30224c6be1341f20869 - ee936f4daffa7ce10fc2623cc914c47b - 7a1a3286482ce0320eb870af4488a58d - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg332-client.virtnet /mnt/lustre (opts:) Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 08:21:07 (1788870067) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 08:21:23 (1788870083) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 08:21:30 (1788870090) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130741 s, 8.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576277 s, 18.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 08:21:39 (1788870099) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.172913 s, 12.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.271507 s, 15.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.178836 s, 11.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 08:21:55 (1788870115) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 08:21:58 (1788870118) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 08:22:01 (1788870121) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343776 s, 30.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 08:22:10 (1788870130) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 08:22:13 (1788870133) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 08:22:23 (1788870143) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.385197 s, 27.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg332-client.virtnet /mnt/lustre (opts:) Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 08:22:34 (1788870154) llite.lustre-ffff9e9190ac3800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.303334 s, 34.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg332-client.virtnet /mnt/lustre (opts:) Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9e9191a3a800.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 08:22:47 (1788870167) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (33s) == sanity-flr test 46: Verify setstripe --copy option ==== 08:23:20 (1788870200) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (38s) == sanity-flr test 47: Verify mirror obj alloc =========== 08:23:58 (1788870238) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 08:24:01 (1788870241) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.244293 s, 12.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.200654 s, 15.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x49e33873 (nosync,stale) Mirror 2: 0x49e33873 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 768f77c7e6718c51b89ed54552d26abe - mirror 1 checksum: 892911cc43b2c5d8ae0639841392a857 - mirror 2 checksum: 768f77c7e6718c51b89ed54552d26abe - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-08T08:24:04-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-08T08:24:04-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 768f77c7e6718c51b89ed54552d26abe - mirror 1 checksum: 892911cc43b2c5d8ae0639841392a857 - mirror 2 checksum: 892911cc43b2c5d8ae0639841392a857 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-08T08:24:04-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-08T08:24:04-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 08:24:14 (1788870254) Failing mds1 on oleg332-server Stopping /mnt/lustre-mds1 (opts:) on oleg332-server 08:24:21 (1788870261) shut down facet: mds1 facet_host: oleg332-server facet_failover_host: oleg332-server Failover mds1 to oleg332-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-MDT0000 08:24:45 (1788870285) targets are mounted 08:24:45 (1788870285) facet_failover done oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336961 s, 31.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xee:0x0] } PASS 50A (48s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 08:25:02 (1788870302) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 08:25:06 (1788870306) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 08:25:09 (1788870309) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00946483 s, 1.7 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 08:25:17 (1788870317) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 08:25:20 (1788870320) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0173537 s, 236 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0583401 s, 70.2 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 08:25:29 (1788870329) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 08:25:50 (1788870350) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 08:26:18 (1788870378) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 08:26:43 (1788870403) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 08:27:08 (1788870428) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 08:27:15 (1788870435) mirror create pid 54264 mirror split pid 54265 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 54264 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 54265 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 08:28:22 (1788870502) Seed set to 504 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1788870504.378797 mapwrite 0x025d6e thru 0x02ecb7 (0x8f4a bytes) 000003 1788870504.483357 write 0x027a31 thru 0x02c92c (0x4efc bytes) 000007 1788870504.487716 write 0x0349e0 thru 0x03b26e (0x688f bytes) 000009 1788870504.491391 read 0x01c810 thru 0x01d336 (0xb27 bytes) 000011 1788870504.497321 write 0x0236cc thru 0x02faa7 (0xc3dc bytes) truncating to largest ever: 0x32d06 000012 1788870504.505530 trunc from 0x03b26f to 0x032d05 (0x8569 bytes) 000014 1788870504.530943 mapread 0x01eed4 thru 0x02b2de (0xc40b bytes) 000015 1788870504.542196 mapread 0x007b1a thru 0x0094e9 (0x19d0 bytes) 000018 1788870504.551753 trunc from 0x032d06 to 0x02ee58 (0x3ead bytes) 000019 1788870504.572242 trunc from 0x02ee59 to 0x02df3d (0xf1b bytes) 000020 1788870504.587511 mapwrite 0x03a2f3 thru 0x03ffff (0x5d0d bytes) 000022 1788870504.671462 mapread 0x038690 thru 0x03952d (0xe9e bytes) 000027 1788870504.680925 trunc from 0x040000 to 0x01cc78 (0x23387 bytes) 000028 1788870504.707396 mapread 0x017010 thru 0x01c70d (0x56fe bytes) 000031 1788870504.713861 mapread 0x008502 thru 0x00c4cf (0x3fce bytes) 000033 1788870504.719309 mapread 0x0029f7 thru 0x01249b (0xfaa5 bytes) 000034 1788870504.732283 read 0x0047fc thru 0x00b878 (0x707d bytes) 000035 1788870504.733305 write 0x00b7da thru 0x0182ab (0xcad2 bytes) 000038 1788870504.739538 read 0x00a78b thru 0x010e98 (0x670e bytes) 000039 1788870504.741938 trunc from 0x01cc79 to 0x01aa72 (0x2206 bytes) 000040 1788870504.782220 write 0x028d28 thru 0x0366b9 (0xd992 bytes) 000042 1788870504.790931 mapread 0x033579 thru 0x035a6d (0x24f5 bytes) 000043 1788870504.801735 MIRROR_EXTEND to 2 mirrors 000047 1788870505.813832 read 0x019a3a thru 0x027267 (0xd82e bytes) 000056 1788870505.827114 write 0x025ab6 thru 0x02896d (0x2eb8 bytes) 000057 1788870505.908534 mapwrite 0x024444 thru 0x02f9fd (0xb5ba bytes) 000059 1788870505.994906 mapwrite 0x02553e thru 0x0298f7 (0x43ba bytes) 000060 1788870506.082755 MIRROR_RESYNC 2 mirrors 000062 1788870506.990767 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b8:0x0]: Invalid argument (22) truncating to largest ever: 0x359ec 000065 1788870507.300706 trunc from 0x0366ba to 0x0359eb (0xcce bytes) 000066 1788870507.352086 mapread 0x0086bd thru 0x015448 (0xcd8c bytes) 000067 1788870507.398831 mapread 0x015e1f thru 0x024326 (0xe508 bytes) 000068 1788870507.424404 trunc from 0x0359ec to 0x013629 (0x223c2 bytes) 000069 1788870507.440104 write 0x03ab80 thru 0x03e68b (0x3b0c bytes) 000070 1788870507.445746 write 0x002a2a thru 0x0036c7 (0xc9e bytes) 000071 1788870507.451658 write 0x004d9f thru 0x012229 (0xd48b bytes) 000072 1788870507.459514 read 0x03b0e4 thru 0x03e68b (0x35a8 bytes) 000073 1788870507.460660 mapread 0x0067a8 thru 0x00ca26 (0x627f bytes) 000074 1788870507.463438 write 0x002e8b thru 0x00a801 (0x7977 bytes) 000076 1788870507.467097 write 0x02443c thru 0x0271d3 (0x2d98 bytes) 000077 1788870507.475445 mapread 0x00d96c thru 0x017ab4 (0xa149 bytes) 000078 1788870507.483391 mapwrite 0x030fe9 thru 0x036f22 (0x5f3a bytes) 000080 1788870507.542303 read 0x001f2f thru 0x00f389 (0xd45b bytes) 000081 1788870507.557725 trunc from 0x03e68c to 0x004fcc (0x396bf bytes) 000083 1788870507.576987 write 0x026d39 thru 0x028ad3 (0x1d9b bytes) 000090 1788870507.586484 read 0x00a705 thru 0x012e8c (0x8788 bytes) 000091 1788870507.595729 trunc from 0x028ad4 to 0x01f530 (0x95a3 bytes) 000094 1788870507.618757 mapread 0x01826d thru 0x01c6b6 (0x444a bytes) 000098 1788870507.624094 read 0x01c4d6 thru 0x01f530 (0x305b bytes) 000100 1788870507.625430 write 0x03bb39 thru 0x03ffff (0x44c7 bytes) 000101 1788870507.632234 MIRROR_EXTEND to 2 mirrors 000102 1788870509.011161 mapwrite 0x024026 thru 0x025b8b (0x1b66 bytes) 000104 1788870509.265031 read 0x006862 thru 0x013746 (0xcee5 bytes) 000106 1788870509.281468 read 0x014ca8 thru 0x022f8f (0xe2e8 bytes) 000108 1788870509.294648 write 0x03a3e0 thru 0x03ffff (0x5c20 bytes) 000109 1788870509.295769 write 0x0186fb thru 0x019940 (0x1246 bytes) 000110 1788870509.302927 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b8:0x0]: Invalid argument (22) truncating to largest ever: 0x37a1d 000111 1788870509.491185 trunc from 0x040000 to 0x037a1c (0x85e3 bytes) 000112 1788870509.520901 mapwrite 0x0145dc thru 0x0208f8 (0xc31d bytes) 000113 1788870509.656591 mapread 0x002caf thru 0x00e8fe (0xbc50 bytes) 000115 1788870509.672992 MIRROR_EXTEND to 2 mirrors 000117 1788870510.370167 write 0x02dae9 thru 0x03751b (0x9a33 bytes) 000120 1788870510.493089 mapwrite 0x00b3c8 thru 0x00e497 (0x30d0 bytes) 000121 1788870510.572547 mapwrite 0x032c0d thru 0x035f6d (0x3361 bytes) 000122 1788870510.606341 read 0x02c455 thru 0x02de70 (0x1a1c bytes) 000125 1788870510.613124 mapwrite 0x0045f3 thru 0x005d49 (0x1757 bytes) 000128 1788870510.652497 trunc from 0x037a1d to 0x00705a (0x309c2 bytes) 000129 1788870510.673261 mapwrite 0x001446 thru 0x006a5b (0x5616 bytes) 000130 1788870510.718952 MIRROR_RESYNC 2 mirrors 000132 1788870511.594630 mapread 0x0033bc thru 0x00705a (0x3c9f bytes) 000133 1788870511.611484 mapwrite 0x013c62 thru 0x01553d (0x18dc bytes) 000134 1788870511.862082 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1553e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1553e): Mirror 2: 0xcab179e2 (stale) CRC-32 checksum value for chunk [0, 0x1553e): Mirror 1: 0xfba207d Mirror 3: 0xfba207d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1553e: skipped 000138 1788870513.013330 mapwrite 0x02e07c thru 0x03a42b (0xc3b0 bytes) 000139 1788870513.196984 write 0x02546c thru 0x026231 (0xdc6 bytes) 000140 1788870513.206847 mapread 0x00b072 thru 0x00bbdd (0xb6c bytes) 000141 1788870513.216229 mapread 0x00c84f thru 0x00e093 (0x1845 bytes) 000143 1788870513.219162 write 0x03302b thru 0x03e5fa (0xb5d0 bytes) 000145 1788870513.226012 mapwrite 0x01e177 thru 0x01e522 (0x3ac bytes) 000146 1788870513.291695 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3e5fb): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e5fb): Mirror 2: 0x6d148f85 (stale) Mirror 3: 0x81ecf442 (stale) CRC-32 checksum value for chunk [0, 0x3e5fb): Mirror 1: 0xbb2cec99 Mirror 4: 0xbb2cec99 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e5fb: skipped 000152 1788870514.208349 write 0x03f7bf thru 0x03ffff (0x841 bytes) 000154 1788870514.373846 write 0x02987d thru 0x037a19 (0xe19d bytes) 000155 1788870514.413539 read 0x027034 thru 0x036dde (0xfdab bytes) 000156 1788870514.426985 mapread 0x0397cf thru 0x03ffff (0x6831 bytes) 000158 1788870514.477049 trunc from 0x040000 to 0x00e873 (0x3178c bytes) 000159 1788870514.542686 read 0x00e442 thru 0x00e873 (0x432 bytes) 000162 1788870514.563172 trunc from 0x00e874 to 0x014a52 (0x61df bytes) 000163 1788870514.596409 trunc from 0x014a53 to 0x017519 (0x2ac7 bytes) 000165 1788870514.621390 mapread 0x007925 thru 0x00aae3 (0x31bf bytes) 000166 1788870514.627024 mapwrite 0x01ecbd thru 0x021034 (0x2378 bytes) 000168 1788870514.810000 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21035): Mirror 3: 0xd7d92163 (stale) Mirror 4: 0xd077e9d2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21035): Mirror 3: 0xd7d92163 (stale) Mirror 4: 0xd077e9d2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21035: skipped 000172 1788870515.169929 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21035): Mirror 3: 0xd7d92163 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21035): Mirror 3: 0xd7d92163 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21035: skipped 000173 1788870515.477636 write 0x0131b4 thru 0x01d8e5 (0xa732 bytes) 000174 1788870515.491738 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b8:0x0]: Invalid argument (22) 000176 1788870515.604864 read 0x00f896 thru 0x0126f3 (0x2e5e bytes) 000177 1788870515.611609 mapread 0x00d4ee thru 0x018a7e (0xb591 bytes) 000180 1788870515.644086 mapwrite 0x00b104 thru 0x00b684 (0x581 bytes) 000183 1788870515.694456 mapwrite 0x015f93 thru 0x022339 (0xc3a7 bytes) 000184 1788870515.771855 write 0x03c946 thru 0x03ffc3 (0x367e bytes) 000185 1788870515.773076 mapread 0x01d5a5 thru 0x02d0c4 (0xfb20 bytes) 000186 1788870515.785425 trunc from 0x03ffc4 to 0x0127f8 (0x2d7cb bytes) 000187 1788870515.801636 write 0x0321ca thru 0x03b04a (0x8e81 bytes) 000188 1788870515.805439 mapread 0x034931 thru 0x03b04a (0x671a bytes) 000191 1788870515.813093 MIRROR_EXTEND to 2 mirrors 000192 1788870516.654904 mapread 0x025f81 thru 0x035de1 (0xfe61 bytes) 000194 1788870516.690762 read 0x0251fc thru 0x034c7e (0xfa83 bytes) 000196 1788870516.691537 mapread 0x01b9ae thru 0x025ee5 (0xa538 bytes) 000200 1788870516.701737 trunc from 0x03b04b to 0x00f3ee (0x2bc5c bytes) 000201 1788870516.789870 trunc from 0x00f3ef to 0x00b410 (0x3fde bytes) 000203 1788870516.820474 write 0x025c6c thru 0x03165d (0xb9f2 bytes) 000205 1788870516.834168 mapwrite 0x00f546 thru 0x01c728 (0xd1e3 bytes) 000206 1788870516.950545 mapread 0x02b399 thru 0x03165d (0x62c5 bytes) 000207 1788870516.952878 mapwrite 0x021b7f thru 0x02c10b (0xa58d bytes) 000208 1788870517.046628 mapwrite 0x012eb4 thru 0x01547a (0x25c7 bytes) 000211 1788870517.084627 mapwrite 0x030a3b thru 0x03b45b (0xaa21 bytes) 000213 1788870517.178199 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b45c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b45c): Mirror 2: 0x2250063c (stale) CRC-32 checksum value for chunk [0, 0x3b45c): Mirror 1: 0x5127f06b Mirror 3: 0x5127f06b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b45c: skipped 000216 1788870518.020189 write 0x01873d thru 0x01b597 (0x2e5b bytes) 000217 1788870518.122087 read 0x03361d thru 0x03a046 (0x6a2a bytes) 000218 1788870518.155474 write 0x02b41b thru 0x02f1ba (0x3da0 bytes) 000219 1788870518.171052 mapwrite 0x02a597 thru 0x02eeca (0x4934 bytes) 000221 1788870518.284096 write 0x036541 thru 0x03fdb2 (0x9872 bytes) 000222 1788870518.295849 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3fdb3): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fdb3): Mirror 2: 0x5070db87 (stale) Mirror 3: 0xd027c6b0 (stale) CRC-32 checksum value for chunk [0, 0x3fdb3): Mirror 1: 0xa10d7800 Mirror 4: 0xa10d7800 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fdb3: skipped 000223 1788870519.207075 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fdb3): Mirror 2: 0x5070db87 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fdb3): Mirror 2: 0x5070db87 (stale) CRC-32 checksum value for chunk [0, 0x3fdb3): Mirror 1: 0xa10d7800 Mirror 4: 0xa10d7800 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fdb3: skipped 000224 1788870519.832991 mapread 0x001c3b thru 0x01195b (0xfd21 bytes) 000225 1788870519.916650 write 0x0381ba thru 0x03bb9c (0x39e3 bytes) 000226 1788870520.023946 mapread 0x01066e thru 0x01a25c (0x9bef bytes) 000227 1788870520.047284 MIRROR_RESYNC 3 mirrors 000228 1788870520.907075 write 0x0335c3 thru 0x03ffff (0xca3d bytes) 000230 1788870521.018412 mapread 0x030167 thru 0x03acb7 (0xab51 bytes) 000231 1788870521.046538 mapread 0x00c287 thru 0x011ae0 (0x585a bytes) 000232 1788870521.062633 write 0x00049e thru 0x00d155 (0xccb8 bytes) 000234 1788870521.069212 mapread 0x013685 thru 0x019a83 (0x63ff bytes) 000236 1788870521.082296 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3b04cfbe (stale) Mirror 4: 0x3b04cfbe (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x23118473 Mirror 5: 0x23118473 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000237 1788870521.774658 read 0x011178 thru 0x01d257 (0xc0e0 bytes) 000238 1788870521.800493 write 0x02a0db thru 0x038e10 (0xed36 bytes) 000239 1788870521.904289 mapwrite 0x029423 thru 0x03245d (0x903b bytes) 000240 1788870521.980542 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3b04cfbe (stale) Mirror 4: 0x3b04cfbe (stale) Mirror 5: 0x23118473 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1ff2c5cb Mirror 6: 0x1ff2c5cb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000241 1788870522.838436 mapread 0x02c339 thru 0x02e687 (0x234f bytes) 000243 1788870522.882453 mapread 0x0248ee thru 0x0316bd (0xcdd0 bytes) 000244 1788870522.910062 read 0x02d113 thru 0x03952b (0xc419 bytes) 000245 1788870522.926158 mapwrite 0x022ca1 thru 0x02a44f (0x77af bytes) truncating to largest ever: 0x39c60 000246 1788870523.122441 trunc from 0x040000 to 0x039c5f (0x63a0 bytes) 000248 1788870523.167438 mapread 0x01132a thru 0x01e95f (0xd636 bytes) 000249 1788870523.187528 mapwrite 0x001892 thru 0x00cf72 (0xb6e1 bytes) 000250 1788870523.245528 write 0x03d7f0 thru 0x03ffff (0x2810 bytes) 000253 1788870523.249533 read 0x0366ad thru 0x03ffff (0x9953 bytes) 000254 1788870523.257358 mapwrite 0x028828 thru 0x036ea1 (0xe67a bytes) 000255 1788870523.301095 mapread 0x028fc5 thru 0x037da7 (0xede3 bytes) 000256 1788870523.302395 mapread 0x00b967 thru 0x0180ae (0xc748 bytes) 000257 1788870523.304353 write 0x03f399 thru 0x03ffff (0xc67 bytes) 000258 1788870523.306151 mapwrite 0x0325f2 thru 0x038d9d (0x67ac bytes) 000260 1788870523.347397 trunc from 0x040000 to 0x00d193 (0x32e6c bytes) 000261 1788870523.375408 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd194): Mirror 4: 0x965e93b1 (stale) Mirror 5: 0x33ebe1f1 (stale) Mirror 6: 0x33ebe1f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd194): Mirror 4: 0x965e93b1 (stale) Mirror 5: 0x33ebe1f1 (stale) Mirror 6: 0x33ebe1f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd194: skipped 000265 1788870523.682720 mapread 0x0053f8 thru 0x00d193 (0x7d9c bytes) 000266 1788870523.699155 write 0x00cba2 thru 0x0130ff (0x655e bytes) 000268 1788870523.701837 trunc from 0x013100 to 0x015d1c (0x2c1d bytes) 000273 1788870523.723485 trunc from 0x015d1d to 0x009360 (0xc9bc bytes) 000275 1788870523.753364 mapread 0x000b53 thru 0x009360 (0x880e bytes) 000276 1788870523.766539 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9361): Mirror 4: 0xecb7f21d (stale) Mirror 6: 0xa2bf8344 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9361): Mirror 4: 0xecb7f21d (stale) Mirror 6: 0xa2bf8344 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9361: skipped 000278 1788870524.104649 mapread 0x00928b thru 0x009360 (0xd6 bytes) truncating to largest ever: 0x3a470 000281 1788870524.116792 trunc from 0x009361 to 0x03a46f (0x3110f bytes) 000283 1788870524.148243 mapread 0x037901 thru 0x03a46f (0x2b6f bytes) 000288 1788870524.181072 trunc from 0x03a470 to 0x019a08 (0x20a67 bytes) 000289 1788870524.220267 mapwrite 0x037cfd thru 0x037f65 (0x269 bytes) truncating to largest ever: 0x3ba11 000292 1788870524.334112 trunc from 0x037f66 to 0x03ba10 (0x3aab bytes) 000293 1788870524.358939 trunc from 0x03ba11 to 0x025d2d (0x15ce3 bytes) 000294 1788870524.387033 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25d2e): Mirror 6: 0xc07725e9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25d2e): Mirror 6: 0xc07725e9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25d2e: skipped 000295 1788870524.782489 write 0x0049e4 thru 0x00f3a2 (0xa9bf bytes) 000296 1788870524.793834 trunc from 0x025d2e to 0x00a22d (0x1bb00 bytes) 000297 1788870524.811357 write 0x01fdf6 thru 0x0226ff (0x290a bytes) 000298 1788870524.815146 write 0x023055 thru 0x02d925 (0xa8d1 bytes) 000299 1788870524.820462 mapread 0x027070 thru 0x0292e5 (0x2276 bytes) 000302 1788870524.823734 mapwrite 0x003d55 thru 0x011c0b (0xdeb7 bytes) 000304 1788870524.893610 mapwrite 0x0384dc thru 0x03a6fc (0x2221 bytes) 000305 1788870524.949882 write 0x010dea thru 0x01e1b8 (0xd3cf bytes) 000306 1788870524.956426 trunc from 0x03a6fd to 0x029864 (0x10e98 bytes) 000307 1788870524.975989 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b8:0x0]: Invalid argument (22) 000308 1788870525.099844 mapwrite 0x02c930 thru 0x0361f1 (0x98c2 bytes) 000309 1788870525.186405 read 0x00de0e thru 0x013068 (0x525b bytes) 000310 1788870525.202441 MIRROR_EXTEND to 2 mirrors 000311 1788870525.773499 mapread 0x013070 thru 0x01455f (0x14f0 bytes) 000312 1788870525.786160 trunc from 0x0361f2 to 0x00a4a4 (0x2bd4d bytes) 000313 1788870525.920210 trunc from 0x00a4a5 to 0x03803b (0x2db97 bytes) 000314 1788870525.958507 read 0x00585e thru 0x00a96b (0x510e bytes) 000315 1788870525.976819 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3803c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3803c): Mirror 2: 0xe13ff335 (stale) CRC-32 checksum value for chunk [0, 0x3803c): Mirror 1: 0x8d0546a3 Mirror 3: 0x8d0546a3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3803c: skipped 000317 1788870527.196419 write 0x01c25e thru 0x023cce (0x7a71 bytes) 000320 1788870527.281384 write 0x015592 thru 0x017d3b (0x27aa bytes) 000322 1788870527.297913 MIRROR_RESYNC 3 mirrors 000323 1788870528.246808 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b8:0x0]: Invalid argument (22) 000324 1788870528.768676 read 0x01b47e thru 0x022af6 (0x7679 bytes) 000325 1788870528.786037 trunc from 0x03803c to 0x03175b (0x68e0 bytes) 000326 1788870528.883939 mapread 0x02c1a1 thru 0x02ec59 (0x2ab9 bytes) 000327 1788870528.911106 trunc from 0x03175c to 0x0094fc (0x2825f bytes) 000328 1788870528.934844 read 0x007479 thru 0x0094fc (0x2084 bytes) 000329 1788870528.949009 mapread 0x004b5b thru 0x007e96 (0x333c bytes) 000332 1788870528.951271 read 0x006832 thru 0x0094fc (0x2ccb bytes) 000333 1788870528.959314 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b8:0x0]: Invalid argument (22) 000338 1788870529.112181 mapwrite 0x03a4aa thru 0x03ffff (0x5b56 bytes) 000339 1788870529.248017 mapwrite 0x03ff3c thru 0x03ffff (0xc4 bytes) 000340 1788870529.289766 write 0x02cf84 thru 0x02e9ee (0x1a6b bytes) 000341 1788870529.300344 mapread 0x008e9f thru 0x008f32 (0x94 bytes) 000343 1788870529.314424 mapread 0x01c360 thru 0x0259c4 (0x9665 bytes) 000346 1788870529.339214 read 0x02c146 thru 0x0347ae (0x8669 bytes) 000351 1788870529.350966 mapread 0x00470b thru 0x00c4bd (0x7db3 bytes) 000352 1788870529.352379 mapwrite 0x00c4ba thru 0x010637 (0x417e bytes) 000354 1788870529.406500 read 0x02f032 thru 0x0364a8 (0x7477 bytes) 000357 1788870529.410021 trunc from 0x040000 to 0x00bcca (0x34335 bytes) 000359 1788870529.428196 mapwrite 0x039531 thru 0x03fe1b (0x68eb bytes) 000361 1788870529.589023 write 0x004e2b thru 0x006ce1 (0x1eb7 bytes) 000362 1788870529.591236 read 0x01eda2 thru 0x0295e7 (0xa846 bytes) 000363 1788870529.594119 read 0x03dda0 thru 0x03f5fc (0x185d bytes) 000366 1788870529.604314 MIRROR_EXTEND to 2 mirrors 000367 1788870530.571622 mapwrite 0x001c6f thru 0x010de8 (0xf17a bytes) 000369 1788870530.818252 trunc from 0x03fe1c to 0x004715 (0x3b706 bytes) 000371 1788870530.838270 mapwrite 0x012b79 thru 0x0217fb (0xec83 bytes) 000374 1788870530.917799 read 0x01307c thru 0x0217fb (0xe780 bytes) 000375 1788870530.922014 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b8:0x0]: Invalid argument (22) 000377 1788870531.054181 read 0x01aefd thru 0x0217fb (0x68ff bytes) 000378 1788870531.066791 write 0x03227f thru 0x039960 (0x76e2 bytes) 000380 1788870531.070112 mapread 0x024f54 thru 0x034a75 (0xfb22 bytes) 000381 1788870531.091100 read 0x0207c0 thru 0x02fa47 (0xf288 bytes) 000383 1788870531.092670 mapwrite 0x0314ba thru 0x033f96 (0x2add bytes) 000385 1788870531.212646 mapread 0x007fc3 thru 0x008324 (0x362 bytes) 000387 1788870531.224729 mapwrite 0x031104 thru 0x033122 (0x201f bytes) 000388 1788870531.287455 trunc from 0x039961 to 0x0352ef (0x4671 bytes) 000389 1788870531.323867 trunc from 0x0352f0 to 0x01bead (0x19442 bytes) 000391 1788870531.346247 trunc from 0x01beae to 0x01bb80 (0x32d bytes) 000393 1788870531.382357 write 0x02e46b thru 0x031f00 (0x3a96 bytes) 000394 1788870531.386276 mapwrite 0x02d56a thru 0x0341ae (0x6c45 bytes) 000396 1788870531.519543 mapwrite 0x0187c4 thru 0x0241f6 (0xba33 bytes) 000399 1788870531.635124 read 0x0278d9 thru 0x02e683 (0x6dab bytes) 000402 1788870531.651536 mapwrite 0x01f860 thru 0x028f9d (0x973e bytes) 000403 1788870531.709933 mapread 0x01834b thru 0x026a13 (0xe6c9 bytes) 000404 1788870531.711172 mapwrite 0x003c3d thru 0x005734 (0x1af8 bytes) 000406 1788870531.789800 mapwrite 0x024390 thru 0x0271df (0x2e50 bytes) 000407 1788870531.861333 mapwrite 0x03feb1 thru 0x03ffff (0x14f bytes) 000409 1788870531.934793 write 0x03d0c6 thru 0x03e8d1 (0x180c bytes) 000410 1788870531.935804 read 0x0359a3 thru 0x03ffff (0xa65d bytes) 000412 1788870531.937971 trunc from 0x040000 to 0x00077d (0x3f882 bytes) 000417 1788870531.955142 mapwrite 0x016ab1 thru 0x01c278 (0x57c8 bytes) 000418 1788870532.019128 mapwrite 0x02cc21 thru 0x036df1 (0xa1d1 bytes) 000419 1788870532.104171 mapwrite 0x01f6f4 thru 0x0243f4 (0x4d01 bytes) 000422 1788870532.211610 mapwrite 0x00a999 thru 0x00fd7e (0x53e6 bytes) 000424 1788870532.291016 mapread 0x035e4b thru 0x036df1 (0xfa7 bytes) 000425 1788870532.292019 mapread 0x0253e9 thru 0x03488b (0xf4a3 bytes) 000426 1788870532.294224 trunc from 0x036df2 to 0x035283 (0x1b6e bytes) 000429 1788870532.309440 write 0x0328f1 thru 0x03ffff (0xd70f bytes) 000430 1788870532.311433 mapread 0x011870 thru 0x012597 (0xd28 bytes) 000431 1788870532.313347 write 0x03f477 thru 0x03ffff (0xb89 bytes) 000432 1788870532.314302 trunc from 0x040000 to 0x012c80 (0x2d37f bytes) 000435 1788870532.334790 MIRROR_EXTEND to 2 mirrors 000436 1788870533.056124 mapread 0x012144 thru 0x012c80 (0xb3d bytes) 000438 1788870533.061532 write 0x01076e thru 0x011e22 (0x16b5 bytes) 000442 1788870533.184840 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x12c81): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12c81): Mirror 2: 0xf54fd0ba (stale) CRC-32 checksum value for chunk [0, 0x12c81): Mirror 1: 0x671b30bf Mirror 3: 0x671b30bf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12c81: skipped 000445 1788870534.259955 read 0x00c4dc thru 0x012c80 (0x67a5 bytes) 000446 1788870534.296306 mapread 0x0027bc thru 0x0107ff (0xe044 bytes) 000447 1788870534.337136 trunc from 0x012c81 to 0x0397ab (0x26b2b bytes) 000448 1788870534.459463 read 0x009e9f thru 0x014320 (0xa482 bytes) 000449 1788870534.487826 mapwrite 0x00f148 thru 0x00fd26 (0xbdf bytes) 000450 1788870534.554799 write 0x00a493 thru 0x016ab1 (0xc61f bytes) 000453 1788870534.560805 mapread 0x022bfb thru 0x025f3e (0x3344 bytes) 000454 1788870534.563327 read 0x00f0d6 thru 0x018535 (0x9460 bytes) 000455 1788870534.571132 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x397ac): Mirror 3: 0xd3c19b43 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x397ac): Mirror 3: 0xd3c19b43 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x397ac: skipped 000456 1788870535.086212 mapwrite 0x01d982 thru 0x02be0e (0xe48d bytes) 000458 1788870535.216808 mapwrite 0x00f95d thru 0x01dc42 (0xe2e6 bytes) 000460 1788870535.331126 mapread 0x005a51 thru 0x00ea79 (0x9029 bytes) 000461 1788870535.359311 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b8:0x0]: Invalid argument (22) 000462 1788870535.483028 write 0x035887 thru 0x03ffff (0xa779 bytes) 000464 1788870535.493994 mapwrite 0x0268fb thru 0x034bd1 (0xe2d7 bytes) 000465 1788870535.636069 read 0x018cce thru 0x01abd8 (0x1f0b bytes) 000468 1788870535.664188 MIRROR_EXTEND to 2 mirrors 000469 1788870536.388032 trunc from 0x040000 to 0x00cecb (0x33134 bytes) 000470 1788870536.513412 read 0x00ad7c thru 0x00cecb (0x2150 bytes) 000473 1788870536.521779 read 0x006766 thru 0x00cecb (0x6766 bytes) 000474 1788870536.531683 read 0x000eaf thru 0x00a40b (0x955d bytes) 000475 1788870536.533614 write 0x014eeb thru 0x021f20 (0xd036 bytes) 000478 1788870536.549093 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b8:0x0]: Invalid argument (22) 000482 1788870536.714274 mapread 0x00bdfe thru 0x0108be (0x4ac1 bytes) 000484 1788870536.734326 mapwrite 0x00cae9 thru 0x00e992 (0x1eaa bytes) 000485 1788870536.773786 read 0x004512 thru 0x0117f3 (0xd2e2 bytes) 000487 1788870536.779496 mapwrite 0x014d8e thru 0x021bfd (0xce70 bytes) 000489 1788870536.853007 mapwrite 0x00dac5 thru 0x01138d (0x38c9 bytes) 000490 1788870536.961165 read 0x01c342 thru 0x021f20 (0x5bdf bytes) 000491 1788870536.963969 write 0x01b323 thru 0x0235ca (0x82a8 bytes) 000494 1788870536.975783 write 0x021bbc thru 0x025658 (0x3a9d bytes) 000495 1788870536.980810 mapwrite 0x009f14 thru 0x018d55 (0xee42 bytes) 000497 1788870537.054223 read 0x013fcf thru 0x021df8 (0xde2a bytes) 000498 1788870537.054937 mapread 0x01d745 thru 0x023ce2 (0x659e bytes) 000499 1788870537.057567 mapwrite 0x02ba9c thru 0x034190 (0x86f5 bytes) 000502 1788870537.235897 trunc from 0x034191 to 0x02da5a (0x6736 bytes) 000503 1788870537.278904 read 0x0275f7 thru 0x02da5a (0x6464 bytes) 000506 1788870537.284969 read 0x00e5ef thru 0x00fe30 (0x1842 bytes) 000507 1788870537.286889 mapwrite 0x00d7b5 thru 0x017c24 (0xa470 bytes) 000510 1788870537.376085 read 0x004f05 thru 0x00fb1c (0xac18 bytes) 000511 1788870537.376815 write 0x0038a1 thru 0x00e0f1 (0xa851 bytes) 000514 1788870537.380369 read 0x00fe69 thru 0x019306 (0x949e bytes) 000516 1788870537.383613 write 0x024132 thru 0x02cf3f (0x8e0e bytes) 000517 1788870537.384854 mapread 0x02a56d thru 0x02d16f (0x2c03 bytes) 000529 1788870537.407542 write 0x02ba24 thru 0x036bba (0xb197 bytes) 000530 1788870537.420893 MIRROR_EXTEND to 2 mirrors 000531 1788870538.135941 MIRROR_EXTEND to 3 mirrors 000533 1788870538.853596 read 0x00b452 thru 0x00f8f0 (0x449f bytes) 000534 1788870538.868360 mapwrite 0x00fb93 thru 0x01f157 (0xf5c5 bytes) 000535 1788870539.045137 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36bbb): Mirror 3: 0x7a22e30 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36bbb): Mirror 3: 0x7a22e30 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36bbb: skipped 000536 1788870539.348936 mapwrite 0x036a10 thru 0x03f530 (0x8b21 bytes) 000537 1788870539.428527 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f531): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f531): Mirror 3: 0x6baf0937 (stale) CRC-32 checksum value for chunk [0, 0x3f531): Mirror 1: 0x20ebc3a4 Mirror 4: 0x20ebc3a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f531: skipped 000538 1788870540.257244 mapread 0x016f92 thru 0x01ef8a (0x7ff9 bytes) 000539 1788870540.293866 read 0x03cf8b thru 0x03f530 (0x25a6 bytes) 000541 1788870540.307520 mapwrite 0x02f379 thru 0x037885 (0x850d bytes) 000543 1788870540.528157 trunc from 0x03f531 to 0x0171af (0x28381 bytes) 000546 1788870540.590820 mapwrite 0x01f4be thru 0x027a8d (0x85d0 bytes) truncating to largest ever: 0x3f5b0 000550 1788870540.682542 trunc from 0x027a8e to 0x03f5af (0x17b22 bytes) 000552 1788870540.705196 write 0x03ed84 thru 0x03ffff (0x127c bytes) 000554 1788870540.713438 trunc from 0x040000 to 0x02f483 (0x10b7c bytes) 000555 1788870540.754150 write 0x00bbbe thru 0x00cdf8 (0x123b bytes) 000557 1788870540.787484 MIRROR_RESYNC 3 mirrors 000559 1788870541.533798 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b8:0x0]: Invalid argument (22) 000560 1788870541.896665 mapread 0x012da1 thru 0x0131b9 (0x419 bytes) 000561 1788870541.907113 mapwrite 0x009aac thru 0x00e5fd (0x4b52 bytes) 000562 1788870542.199903 mapwrite 0x03ea84 thru 0x03ffff (0x157c bytes) 000564 1788870542.334797 read 0x004a86 thru 0x008cce (0x4249 bytes) 000565 1788870542.340945 mapread 0x01ba33 thru 0x021049 (0x5617 bytes) 000566 1788870542.356720 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b8:0x0]: Invalid argument (22) 000567 1788870542.568859 write 0x013433 thru 0x01598a (0x2558 bytes) 000568 1788870542.579879 trunc from 0x040000 to 0x02a3de (0x15c21 bytes) 000569 1788870542.593522 mapread 0x0176f3 thru 0x024c16 (0xd524 bytes) 000570 1788870542.622962 read 0x00fc86 thru 0x018bc2 (0x8f3d bytes) 000573 1788870542.628611 mapwrite 0x02cf31 thru 0x037979 (0xaa49 bytes) 000575 1788870542.700080 trunc from 0x03797a to 0x01c149 (0x1b830 bytes) 000578 1788870542.724089 mapread 0x00216c thru 0x01214a (0xffdf bytes) 000582 1788870542.740392 trunc from 0x01c14a to 0x02eb01 (0x129b8 bytes) 000583 1788870542.757930 trunc from 0x02eb02 to 0x034275 (0x5774 bytes) 000584 1788870542.776645 read 0x020bfc thru 0x028497 (0x789c bytes) 000587 1788870542.801398 read 0x01e16a thru 0x029634 (0xb4cb bytes) 000588 1788870542.802763 trunc from 0x034276 to 0x0052e9 (0x2ef8c bytes) 000589 1788870542.823747 mapread 0x001bf4 thru 0x0052e9 (0x36f6 bytes) 000593 1788870542.844176 MIRROR_EXTEND to 2 mirrors 000594 1788870543.612497 mapread 0x002f92 thru 0x0052e9 (0x2358 bytes) 000595 1788870543.656913 trunc from 0x0052ea to 0x01b1dc (0x15ef3 bytes) 000596 1788870543.781164 trunc from 0x01b1dd to 0x012dc5 (0x8417 bytes) 000597 1788870543.818543 mapwrite 0x0245d5 thru 0x026e7c (0x28a8 bytes) 000598 1788870543.908794 write 0x01af2e thru 0x02abc9 (0xfc9c bytes) 000600 1788870543.920857 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b8:0x0]: Invalid argument (22) 000602 1788870544.130351 MIRROR_EXTEND to 2 mirrors 000603 1788870544.744753 mapwrite 0x01bc2b thru 0x02bbbe (0xff94 bytes) 000604 1788870544.974078 read 0x009c0f thru 0x00acd6 (0x10c8 bytes) 000605 1788870544.981135 trunc from 0x02bbbf to 0x03eb5d (0x12f9f bytes) 000607 1788870545.030108 MIRROR_RESYNC 2 mirrors 000608 1788870545.871119 write 0x03d781 thru 0x03ffff (0x287f bytes) 000610 1788870545.940609 mapread 0x02f347 thru 0x03c5ea (0xd2a4 bytes) 000611 1788870545.964616 mapread 0x024bcb thru 0x02b47e (0x68b4 bytes) 000612 1788870545.978022 trunc from 0x040000 to 0x02e974 (0x1168b bytes) 000613 1788870546.006186 write 0x00497b thru 0x014390 (0xfa16 bytes) 000615 1788870546.019210 mapwrite 0x002f1c thru 0x003b60 (0xc45 bytes) 000616 1788870546.073937 mapwrite 0x012c38 thru 0x01331f (0x6e8 bytes) 000617 1788870546.117317 write 0x018898 thru 0x02136d (0x8ad6 bytes) 000618 1788870546.121805 read 0x022d0c thru 0x02a274 (0x7569 bytes) 000619 1788870546.122314 trunc from 0x02e975 to 0x021e4c (0xcb28 bytes) 000620 1788870546.137947 read 0x01f611 thru 0x021e4c (0x283c bytes) 000621 1788870546.143038 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x21e4d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21e4d): Mirror 2: 0x85f38446 (stale) CRC-32 checksum value for chunk [0, 0x21e4d): Mirror 1: 0x77782d79 Mirror 3: 0x77782d79 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21e4d: skipped 000622 1788870547.067320 mapwrite 0x001d33 thru 0x00ddd7 (0xc0a5 bytes) 000624 1788870547.346866 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21e4d): Mirror 2: 0x85f38446 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21e4d): Mirror 2: 0x85f38446 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21e4d: skipped 000627 1788870547.733882 mapread 0x017bbc thru 0x018166 (0x5ab bytes) 000628 1788870547.745332 mapread 0x020897 thru 0x021e4c (0x15b6 bytes) 000630 1788870547.757518 mapwrite 0x00afd3 thru 0x017dd2 (0xce00 bytes) 000632 1788870547.866029 write 0x001971 thru 0x00f5bd (0xdc4d bytes) 000634 1788870547.879990 trunc from 0x021e4d to 0x00a12e (0x17d1e bytes) 000635 1788870547.975034 write 0x00e475 thru 0x017a44 (0x95d0 bytes) 000637 1788870547.980496 trunc from 0x017a45 to 0x01784c (0x1f8 bytes) 000640 1788870548.011813 read 0x0104cc thru 0x0128fd (0x2432 bytes) 000641 1788870548.017872 mapread 0x00d323 thru 0x01784c (0xa52a bytes) 000643 1788870548.027934 mapread 0x007a23 thru 0x00f5c3 (0x7ba1 bytes) 000644 1788870548.029906 write 0x003fdf thru 0x0084ac (0x44ce bytes) 000649 1788870548.047770 trunc from 0x01784d to 0x034b37 (0x1d2eb bytes) 000650 1788870548.079046 trunc from 0x034b38 to 0x0233b4 (0x11783 bytes) 000651 1788870548.102991 mapread 0x016b7e thru 0x01a17a (0x35fd bytes) 000652 1788870548.122234 write 0x03db35 thru 0x03ffff (0x24cb bytes) 000653 1788870548.124678 read 0x00e8f6 thru 0x014c29 (0x6334 bytes) 000654 1788870548.126226 read 0x02d7e1 thru 0x0324b1 (0x4cd1 bytes) 000657 1788870548.137484 mapwrite 0x0177a2 thru 0x02717d (0xf9dc bytes) 000659 1788870548.237772 mapread 0x00632f thru 0x00fdbc (0x9a8e bytes) 000660 1788870548.240337 trunc from 0x040000 to 0x035f73 (0xa08c bytes) 000661 1788870548.259991 trunc from 0x035f74 to 0x028503 (0xda70 bytes) 000664 1788870548.296044 MIRROR_RESYNC 2 mirrors 000665 1788870549.049384 mapwrite 0x0143c4 thru 0x01fffd (0xbc3a bytes) 000666 1788870549.264167 MIRROR_RESYNC 2 mirrors 000668 1788870550.031974 mapwrite 0x03cba3 thru 0x03f872 (0x2cd0 bytes) 000670 1788870550.279715 trunc from 0x03f873 to 0x01d1ca (0x226a8 bytes) 000671 1788870550.323727 write 0x0133e9 thru 0x01e8ca (0xb4e2 bytes) 000674 1788870550.346193 MIRROR_RESYNC 2 mirrors 000677 1788870550.950389 mapread 0x00ffff thru 0x017c65 (0x7c67 bytes) 000679 1788870550.980470 trunc from 0x01e8cb to 0x013b64 (0xad66 bytes) 000681 1788870551.088262 read 0x00d438 thru 0x01219f (0x4d68 bytes) 000682 1788870551.095938 trunc from 0x013b65 to 0x0169c6 (0x2e62 bytes) 000683 1788870551.125290 mapwrite 0x0178a7 thru 0x0222f5 (0xaa4f bytes) 000684 1788870551.199112 mapread 0x00ddad thru 0x011305 (0x3559 bytes) 000685 1788870551.200436 trunc from 0x0222f6 to 0x0350f0 (0x12dfb bytes) 000687 1788870551.224997 mapread 0x0320ef thru 0x0350f0 (0x3002 bytes) 000688 1788870551.233995 write 0x00a09c thru 0x017266 (0xd1cb bytes) 000692 1788870551.241002 write 0x01fd16 thru 0x02095a (0xc45 bytes) 000697 1788870551.246472 mapread 0x02fbf6 thru 0x034f7f (0x538a bytes) 000699 1788870551.248574 mapread 0x025820 thru 0x0303a5 (0xab86 bytes) 000700 1788870551.257471 mapread 0x032140 thru 0x0350f0 (0x2fb1 bytes) 000703 1788870551.260479 trunc from 0x0350f1 to 0x0266a3 (0xea4d bytes) 000705 1788870551.278791 mapread 0x01fcd2 thru 0x0266a3 (0x69d2 bytes) 000708 1788870551.281973 read 0x00977d thru 0x00eec3 (0x5747 bytes) 000710 1788870551.289521 mapwrite 0x03d7e7 thru 0x03ffff (0x2819 bytes) 000711 1788870551.386448 trunc from 0x040000 to 0x03d773 (0x288c bytes) 000712 1788870551.422239 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b8:0x0]: Invalid argument (22) 000713 1788870551.692738 write 0x0345f4 thru 0x03605d (0x1a6a bytes) 000715 1788870551.721105 write 0x035576 thru 0x03ffff (0xaa8a bytes) 000716 1788870551.723822 write 0x03f734 thru 0x03ffff (0x8cc bytes) 000718 1788870551.728760 MIRROR_EXTEND to 2 mirrors 000720 1788870552.825990 read 0x030d10 thru 0x032296 (0x1587 bytes) 000722 1788870552.834621 read 0x032080 thru 0x03c335 (0xa2b6 bytes) 000724 1788870552.851135 mapread 0x0080af thru 0x00d70f (0x5661 bytes) 000725 1788870552.857933 MIRROR_RESYNC 2 mirrors 000726 1788870553.146010 mapread 0x038d36 thru 0x03ffff (0x72ca bytes) 000727 1788870553.147486 write 0x01cf4c thru 0x020b98 (0x3c4d bytes) 000729 1788870553.308788 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4bffd47 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd263ef5a Mirror 3: 0xd263ef5a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000730 1788870554.419326 trunc from 0x040000 to 0x016a2a (0x295d5 bytes) 000731 1788870554.589408 read 0x0136e5 thru 0x016a2a (0x3346 bytes) 000735 1788870554.599159 read 0x000945 thru 0x000f9a (0x656 bytes) 000736 1788870554.634943 write 0x001955 thru 0x0083af (0x6a5b bytes) 000738 1788870554.636694 write 0x010560 thru 0x01553d (0x4fde bytes) 000739 1788870554.642584 read 0x007daa thru 0x0160e9 (0xe340 bytes) 000742 1788870554.646734 mapread 0x00dc0f thru 0x016a2a (0x8e1c bytes) 000744 1788870554.650176 read 0x011df7 thru 0x016a2a (0x4c34 bytes) 000745 1788870554.651319 trunc from 0x016a2b to 0x0033fd (0x1362d bytes) 000746 1788870554.692956 trunc from 0x0033fe to 0x019384 (0x15f87 bytes) 000747 1788870554.719396 write 0x00cd2f thru 0x012c81 (0x5f53 bytes) 000749 1788870554.764107 MIRROR_RESYNC 3 mirrors 000751 1788870555.528209 read 0x00b6ef thru 0x0141d1 (0x8ae3 bytes) 000752 1788870555.534888 trunc from 0x019385 to 0x016a7e (0x2906 bytes) 000753 1788870555.612624 mapwrite 0x03f9e9 thru 0x03ffff (0x617 bytes) 000756 1788870555.834754 trunc from 0x040000 to 0x02a030 (0x15fcf bytes) 000757 1788870555.869600 mapread 0x01e814 thru 0x026ec6 (0x86b3 bytes) 000758 1788870555.884527 trunc from 0x02a031 to 0x004dbf (0x25271 bytes) 000761 1788870555.909887 MIRROR_RESYNC 3 mirrors 000762 1788870556.843866 write 0x03bcb4 thru 0x03ffff (0x434c bytes) 000764 1788870556.929397 read 0x00b4f6 thru 0x0123ed (0x6ef8 bytes) 000765 1788870556.959035 read 0x028317 thru 0x029354 (0x103e bytes) 000766 1788870556.968415 trunc from 0x040000 to 0x03645f (0x9ba0 bytes) 000768 1788870557.014170 write 0x015f60 thru 0x01bb58 (0x5bf9 bytes) 000769 1788870557.016223 mapwrite 0x02494b thru 0x027ca8 (0x335e bytes) 000772 1788870557.109794 trunc from 0x036460 to 0x00ba06 (0x2aa59 bytes) 000774 1788870557.154353 mapwrite 0x03260c thru 0x03cba5 (0xa59a bytes) 000775 1788870557.280198 mapread 0x003ca4 thru 0x0051a1 (0x14fe bytes) 000776 1788870557.283077 write 0x015f0d thru 0x0179f9 (0x1aed bytes) 000777 1788870557.299634 mapread 0x008ec2 thru 0x010578 (0x76b7 bytes) 000781 1788870557.329527 mapread 0x02da75 thru 0x034d4d (0x72d9 bytes) 000782 1788870557.357388 write 0x02cb92 thru 0x02f39e (0x280d bytes) 000783 1788870557.363327 write 0x026905 thru 0x029c4c (0x3348 bytes) 000785 1788870557.385383 write 0x030269 thru 0x035b4c (0x58e4 bytes) 000786 1788870557.386743 trunc from 0x03cba6 to 0x028956 (0x1424f bytes) 000787 1788870557.408534 mapwrite 0x02f7a4 thru 0x03746f (0x7ccc bytes) 000789 1788870557.522290 mapread 0x01fc45 thru 0x02bbfb (0xbfb7 bytes) 000790 1788870557.562729 trunc from 0x037470 to 0x0095ea (0x2de85 bytes) 000792 1788870557.579924 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x95eb): Mirror 2: 0xbfdd9d83 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x95eb): Mirror 2: 0xbfdd9d83 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x95eb: skipped 000794 1788870557.801735 mapwrite 0x02c474 thru 0x035fdd (0x9b6a bytes) 000795 1788870557.924511 mapread 0x028372 thru 0x02b5fd (0x328c bytes) 000799 1788870557.933713 read 0x004a18 thru 0x005100 (0x6e9 bytes) 000800 1788870557.950417 mapwrite 0x0377e2 thru 0x03cbb9 (0x53d8 bytes) 000801 1788870558.154021 write 0x016687 thru 0x019b03 (0x347d bytes) 000802 1788870558.193944 MIRROR_RESYNC 2 mirrors 000804 1788870559.117451 write 0x02f8b6 thru 0x03ee13 (0xf55e bytes) 000806 1788870559.211779 mapread 0x02a8b8 thru 0x02edfb (0x4544 bytes) 000807 1788870559.241132 trunc from 0x03ee14 to 0x024a6a (0x1a3a9 bytes) 000809 1788870559.281956 trunc from 0x024a6b to 0x02aac4 (0x605a bytes) 000810 1788870559.298206 write 0x01de67 thru 0x02690f (0x8aa9 bytes) 000811 1788870559.340195 trunc from 0x02aac5 to 0x00ea6b (0x1c059 bytes) 000812 1788870559.384632 mapwrite 0x0202be thru 0x02b073 (0xadb6 bytes) 000813 1788870559.522136 mapwrite 0x037c62 thru 0x03ffff (0x839e bytes) 000816 1788870559.613629 read 0x004aaa thru 0x010761 (0xbcb8 bytes) 000818 1788870559.625047 write 0x02004f thru 0x026c66 (0x6c18 bytes) 000819 1788870559.632531 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b8:0x0]: Invalid argument (22) 000821 1788870559.829293 mapread 0x02878c thru 0x02aece (0x2743 bytes) 000824 1788870559.854191 mapread 0x01b40c thru 0x026bdb (0xb7d0 bytes) 000825 1788870559.867480 read 0x027510 thru 0x02804e (0xb3f bytes) 000827 1788870559.869759 mapread 0x00900e thru 0x0140bf (0xb0b2 bytes) 000831 1788870559.889630 mapread 0x03fed4 thru 0x03ffff (0x12c bytes) 000832 1788870559.901954 trunc from 0x040000 to 0x007002 (0x38ffd bytes) 000836 1788870559.925980 MIRROR_EXTEND to 2 mirrors 000839 1788870560.762400 trunc from 0x007003 to 0x037c12 (0x30c10 bytes) 000840 1788870560.840179 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b8:0x0]: Invalid argument (22) 000843 1788870561.021511 mapread 0x0167e2 thru 0x01c583 (0x5da2 bytes) 000844 1788870561.052105 mapwrite 0x03b2e5 thru 0x03ffff (0x4d1b bytes) 000845 1788870561.197384 read 0x008b34 thru 0x012a39 (0x9f06 bytes) 000847 1788870561.216588 mapread 0x037bf7 thru 0x03d448 (0x5852 bytes) 000851 1788870561.226417 trunc from 0x040000 to 0x02f009 (0x10ff6 bytes) 000852 1788870561.283273 mapread 0x018c13 thru 0x0232c1 (0xa6af bytes) 000853 1788870561.293810 mapwrite 0x02f80d thru 0x03c5a7 (0xcd9b bytes) 000854 1788870561.392382 write 0x03e416 thru 0x03ffff (0x1bea bytes) 000855 1788870561.394348 read 0x038432 thru 0x03ffff (0x7bce bytes) 000856 1788870561.405358 trunc from 0x040000 to 0x017a5b (0x285a4 bytes) 000857 1788870561.426427 MIRROR_EXTEND to 2 mirrors 000859 1788870562.259908 mapread 0x011dc1 thru 0x017a5b (0x5c9b bytes) 000861 1788870562.292298 mapread 0x004aef thru 0x00e540 (0x9a52 bytes) 000863 1788870562.334220 trunc from 0x017a5c to 0x0326ca (0x1ac6f bytes) 000864 1788870562.487988 trunc from 0x0326cb to 0x00bfad (0x2671d bytes) 000865 1788870562.523408 trunc from 0x00bfae to 0x01fe1c (0x13e6f bytes) 000866 1788870562.539299 mapwrite 0x00f612 thru 0x0127d9 (0x31c8 bytes) 000869 1788870562.610538 MIRROR_RESYNC 2 mirrors 000870 1788870563.715003 mapread 0x0053d6 thru 0x010979 (0xb5a4 bytes) 000871 1788870563.731816 mapwrite 0x00afdf thru 0x00f3cf (0x43f1 bytes) 000873 1788870563.901814 mapwrite 0x03f607 thru 0x03ffff (0x9f9 bytes) 000875 1788870564.013833 trunc from 0x040000 to 0x013a2e (0x2c5d1 bytes) 000877 1788870564.043953 read 0x0127fd thru 0x013a2e (0x1232 bytes) 000883 1788870564.053402 read 0x00f8bf thru 0x013a2e (0x4170 bytes) 000884 1788870564.056274 read 0x00abc2 thru 0x013a2e (0x8e6d bytes) 000885 1788870564.057992 mapwrite 0x01d58b thru 0x021fb7 (0x4a2d bytes) 000886 1788870564.131782 mapwrite 0x030b98 thru 0x03a157 (0x95c0 bytes) 000888 1788870564.234954 mapwrite 0x00943e thru 0x00ed8e (0x5951 bytes) 000889 1788870564.293161 mapread 0x001762 thru 0x00b09b (0x993a bytes) 000891 1788870564.296159 mapread 0x038f95 thru 0x03a157 (0x11c3 bytes) 000892 1788870564.298019 trunc from 0x03a158 to 0x03bb08 (0x19b1 bytes) 000893 1788870564.319170 trunc from 0x03bb09 to 0x03eb0e (0x3006 bytes) 000894 1788870564.343026 read 0x027ea7 thru 0x035449 (0xd5a3 bytes) 000896 1788870564.356423 trunc from 0x03eb0f to 0x03b18d (0x3981 bytes) 000898 1788870564.412509 mapread 0x03268b thru 0x0359b1 (0x3327 bytes) 000899 1788870564.417968 write 0x02cefe thru 0x03c964 (0xfa67 bytes) 000901 1788870564.421893 mapwrite 0x00b199 thru 0x01556b (0xa3d3 bytes) 000903 1788870564.483979 mapread 0x012378 thru 0x0175e9 (0x5272 bytes) 000906 1788870564.491981 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3c965): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c965): Mirror 2: 0xb47951e8 (stale) CRC-32 checksum value for chunk [0, 0x3c965): Mirror 1: 0x77b10ddc Mirror 3: 0x77b10ddc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c965: skipped 000907 1788870565.349153 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3c965): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c965): Mirror 2: 0xb47951e8 (stale) CRC-32 checksum value for chunk [0, 0x3c965): Mirror 1: 0x77b10ddc Mirror 3: 0x77b10ddc Mirror 4: 0x77b10ddc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c965: skipped 000909 1788870566.179208 write 0x008bee thru 0x012534 (0x9947 bytes) 000910 1788870566.267364 write 0x03349b thru 0x036d7a (0x38e0 bytes) 000912 1788870566.280431 trunc from 0x03c965 to 0x002bda (0x39d8a bytes) 000913 1788870566.321510 mapread 0x002560 thru 0x002b6c (0x60d bytes) 000915 1788870566.329844 write 0x019caa thru 0x020e1c (0x7173 bytes) truncating to largest ever: 0x3fd79 000916 1788870566.332924 trunc from 0x020e1d to 0x03fd78 (0x1ef5c bytes) 000917 1788870566.350440 trunc from 0x03fd79 to 0x0265ef (0x19789 bytes) 000918 1788870566.366973 mapread 0x00fdf1 thru 0x01b434 (0xb644 bytes) 000920 1788870566.383596 read 0x018359 thru 0x0200cb (0x7d73 bytes) 000921 1788870566.384759 trunc from 0x0265f0 to 0x0053f8 (0x211f7 bytes) 000923 1788870566.400185 mapread 0x001b6e thru 0x0053f8 (0x388b bytes) 000924 1788870566.407726 read 0x004220 thru 0x0053f8 (0x11d9 bytes) 000925 1788870566.408973 mapwrite 0x021897 thru 0x02ad0d (0x9477 bytes) 000926 1788870566.513587 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ad0e): Mirror 3: 0xb98dc5b0 (stale) Mirror 4: 0xb98dc5b0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ad0e): Mirror 3: 0xb98dc5b0 (stale) Mirror 4: 0xb98dc5b0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ad0e: skipped 000929 1788870566.776442 write 0x03bfe8 thru 0x03ffff (0x4018 bytes) 000930 1788870566.778202 mapread 0x034e6c thru 0x03ffff (0xb194 bytes) 000931 1788870566.800200 trunc from 0x040000 to 0x03e848 (0x17b7 bytes) 000935 1788870566.837946 trunc from 0x03e849 to 0x016f48 (0x27900 bytes) 000936 1788870566.873509 MIRROR_RESYNC 3 mirrors 000937 1788870567.800226 write 0x02abcc thru 0x036111 (0xb546 bytes) 000941 1788870567.888689 mapwrite 0x001116 thru 0x0061fc (0x50e7 bytes) 000943 1788870567.974371 read 0x000e8f thru 0x00f1bd (0xe32f bytes) 000945 1788870567.988744 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x36112): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36112): Mirror 3: 0x2cd128cf (stale) Mirror 4: 0x2cd128cf (stale) CRC-32 checksum value for chunk [0, 0x36112): Mirror 1: 0x37417a6b Mirror 5: 0x37417a6b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36112: skipped 000946 1788870569.025731 mapwrite 0x00d282 thru 0x0134c8 (0x6247 bytes) 000947 1788870569.213845 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x36112): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36112): Mirror 3: 0x2cd128cf (stale) Mirror 4: 0x2cd128cf (stale) Mirror 5: 0x37417a6b (stale) CRC-32 checksum value for chunk [0, 0x36112): Mirror 1: 0x8ba51b83 Mirror 6: 0x8ba51b83 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36112: skipped 000948 1788870570.283143 write 0x005321 thru 0x00e7f6 (0x94d6 bytes) 000950 1788870570.354540 read 0x01c30a thru 0x01c3d4 (0xcb bytes) 000953 1788870570.360660 write 0x02e4e2 thru 0x033c5a (0x5779 bytes) 000955 1788870570.370715 mapread 0x004e48 thru 0x011389 (0xc542 bytes) 000959 1788870570.390116 mapread 0x001d74 thru 0x006002 (0x428f bytes) 000961 1788870570.397114 write 0x010f50 thru 0x01dd26 (0xcdd7 bytes) 000962 1788870570.402256 mapread 0x0136a5 thru 0x01eb4d (0xb4a9 bytes) 000965 1788870570.411975 mapread 0x02a2ba thru 0x02d7e3 (0x352a bytes) 000968 1788870570.421980 mapwrite 0x03756c thru 0x03ffff (0x8a94 bytes) 000975 1788870570.497840 read 0x01c540 thru 0x01f34e (0x2e0f bytes) 000979 1788870570.504448 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x873c1d25 (stale) Mirror 5: 0x54eb15da (stale) Mirror 6: 0xe9e3f81 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x873c1d25 (stale) Mirror 5: 0x54eb15da (stale) Mirror 6: 0xe9e3f81 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000980 1788870570.953398 mapwrite 0x02c229 thru 0x0352a7 (0x907f bytes) 000982 1788870571.076179 trunc from 0x040000 to 0x00986b (0x36794 bytes) 000983 1788870571.099009 mapread 0x002866 thru 0x00986b (0x7006 bytes) 000984 1788870571.122412 mapread 0x000256 thru 0x00986b (0x9616 bytes) 000985 1788870571.131240 mapread 0x004e46 thru 0x00986b (0x4a26 bytes) 000986 1788870571.132696 read 0x007ff9 thru 0x00986b (0x1873 bytes) 000989 1788870571.135211 write 0x01e45d thru 0x027f9c (0x9b40 bytes) 000990 1788870571.137924 write 0x00356c thru 0x00f256 (0xbceb bytes) 000992 1788870571.144956 mapread 0x0130d7 thru 0x013b47 (0xa71 bytes) 000995 1788870571.156988 read 0x01d458 thru 0x024d19 (0x78c2 bytes) 000996 1788870571.162246 mapread 0x019733 thru 0x021681 (0x7f4f bytes) 000998 1788870571.165136 mapread 0x00441a thru 0x00c2d3 (0x7eba bytes) 000999 1788870571.172064 write 0x02514b thru 0x02810f (0x2fc5 bytes) All operations completed A-OK! PASS 70a (75s) == sanity-flr test 71: check mirror extend parent fid ==== 08:29:37 (1788870577) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 08:29:40 (1788870580) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.163105 s, 12.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184821 s, 5.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 08:29:48 (1788870588) Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre2 Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 8552512 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3387488 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7844480 .. Extending file size to 8631616 .. Extending file size to 7974368 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8068608 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9158656 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9196640 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9289120 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57325 57326 57327 57331 57332 failed 192.168.203.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg332-client.virtnet /mnt/lustre2 (opts:) 192.168.203.132@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg332-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 08:31:10 (1788870670) Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre2 Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8087360 .. Extending file size to 5477056 .. Extending file size to 8533600 .. Extending file size to 5516896 .. Extending file size to 6826016 .. Extending file size to 8886112 .. Extending file size to 8622880 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9177184 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8847136 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8956608 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9188160 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 58843 58844 58846 58848 58851 192.168.203.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg332-client.virtnet /mnt/lustre2 (opts:) 192.168.203.132@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg332-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (75s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 08:32:26 (1788870746) Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104999 s, 20.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.243428 s, 8.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xf5857c6d (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg332-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 08:32:44 (1788870764) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.10902 s, 19.2 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 08:32:51 (1788870771) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fb:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fb:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fc:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 08:32:58 (1788870778) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.191518 s, 11.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 08:33:06 (1788870786) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.228683 s, 9.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ff:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ff:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 08:33:13 (1788870793) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.313968 s, 6.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x300:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x300:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 08:33:46 (1788870826) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg332-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845594 s, 12.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x301:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x301:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x302:0x0] } PASS 204d (53s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 08:34:39 (1788870879) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg332-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x301:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.245681 s, 8.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x301:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x303:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x301:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x303:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (57s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 08:35:36 (1788870936) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg332-server mds-ost sync done. Creating new pool test_204f oleg332-server: Pool lustre.test_204f created Adding targets to pool oleg332-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg332-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.222374 s, 9.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x304:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x304:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg332-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg332-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg332-server: Pool lustre.test_204f destroyed PASS 204f (77s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 08:36:53 (1788871013) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 08:37:00 (1788871020) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 08:37:07 (1788871027) oleg332-server: Pool lustre.test_206 created Adding targets to pool oleg332-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg332-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg332-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg332-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg332-server: Pool lustre.test_206 destroyed PASS 206 (24s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 08:37:31 (1788871051) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816106 s, 12.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249846 s, 42.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x7e238a21 Mirror 3: 0x7e238a21 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 08:37:42 (1788871062) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 08:37:45 (1788871065) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 08:37:49 (1788871069) Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.13433 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1730000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2220000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2220000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2220000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1ce0000 Read this (4 bytes): BBBB 192.168.203.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg332-client.virtnet /mnt/lustre2 (opts:) PASS 209a (35s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 08:38:24 (1788871104) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0297502 s, 275 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 08:38:31 (1788871111) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577208 s, 18.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x227 lmm_fid: [0x200000404:0x227:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x227 lmm_fid: [0x200000404:0x227:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30f:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834471 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 08:38:32.000000000 -0400 Modify: 2026-09-08 08:38:32.000000000 -0400 Change: 2026-09-08 08:38:33.000000000 -0400 Birth: 2026-09-08 08:38:33.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 08:38:39 (1788871119) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 08:38:42 (1788871122) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.62882 s, 65.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0457253 s, 89.6 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 8 08:38 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x229:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 8 08:38 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 08:38:51 (1788871131) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 08:38:58 (1788871138) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1723 sec =========== 08:39:07 (1788871147) === sanity-flr: start cleanup 08:39:09 (1788871149) === === sanity-flr: finish cleanup 08:39:16 (1788871156) ===