-----============= acceptance-small: sanity-flr ============----- Mon Sep 7 10:24:58 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg117-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 10:25:18 (1788791118) === oleg117-client.virtnet: executing check_config_client /mnt/lustre oleg117-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg117-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff93eb84d2b800.idle_timeout=debug osc.lustre-OST0001-osc-ffff93eb84d2b800.idle_timeout=debug disable quota as required oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 10:26:02 (1788791162) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7633] == sanity-flr test 0a: lfs mirror create with -N option == 10:26:04 (1788791164) PASS 0a (20s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:26:24 (1788791184) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:26:27 (1788791187) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 10:26:30 (1788791190) PASS 0d (30s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:27:00 (1788791220) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:27:04 (1788791224) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (5s) == sanity-flr test 0g: lfs mirror create flags support === 10:27:09 (1788791229) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff93eb84d2b800.stats=clear osc.lustre-OST0001-osc-ffff93eb84d2b800.stats=clear PASS 0g (13s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:27:22 (1788791242) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--zerostart] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (16s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:27:38 (1788791258) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:27:47 (1788791267) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 10:27:51 (1788791271) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 10:28:02 (1788791282) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (12s) == sanity-flr test 2: create components from existing files ========================================================== 10:28:13 (1788791293) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:28:22 (1788791302) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (5s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:28:27 (1788791307) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (11s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:28:38 (1788791318) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 10:28:49 (1788791329) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:28:57 (1788791337) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 1.61021 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15235 s, 6.9 MB/s PASS 21 (13s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:29:10 (1788791350) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291421 s, 36.0 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 10:29:19 (1788791359) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:29:32 (1788791372) Stopping /mnt/lustre-ost1 (opts:) on oleg117-server oleg117-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93eb84d2b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93eb84d2b800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg117-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg117-server oleg117-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93eb84d2b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93eb84d2b800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg117-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (84s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:30:56 (1788791456) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.328351 s, 16.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg117-server oleg117-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93eb84d2b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93eb84d2b800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg117-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (49s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:31:45 (1788791505) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg117-server oleg117-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93eb84d2b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93eb84d2b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg117-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg117-server oleg117-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93eb84d2b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93eb84d2b800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg117-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (83s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:33:08 (1788791588) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.75883 s, 13.9 MB/s -rw-r--r-- 1 root root 52428800 Sep 7 10:33 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.192s user 0m0.001s sys 0m0.178s Stopping /mnt/lustre-ost1 (opts:) on oleg117-server oleg117-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93eb84d2b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93eb84d2b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.984s user 0m0.000s sys 0m0.189s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg117-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg117-server oleg117-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93eb84d2b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93eb84d2b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.481s user 0m0.000s sys 0m0.219s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg117-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg117-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93eb84d2b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93eb84d2b800.ost_server_uuid in FULL state after 0 sec PASS 33b (101s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:34:50 (1788791690) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:34:53 (1788791693) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:34:58 (1788791698) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 10:35:02 (1788791702) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 10:35:10 (1788791710) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.923533 s, 13.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:35:24 (1788791724) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337988 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587545 s, 17.8 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:35:32 (1788791732) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:35:40 (1788791740) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00116321 s, 880 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000887375 s, 1.2 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00161681 s, 633 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000996218 s, 1.0 MB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 10:35:52 (1788791752) create test files with size 15755017 .. 72b27d0a398e224e48902f1482feaf63 - 66b8bc6f27c251e83021f9cfebe3668b - 05e627acb08d42abcd180369dac0369a - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg117-client.virtnet /mnt/lustre (opts:) Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 10:36:18 (1788791778) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:36:34 (1788791794) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:36:42 (1788791802) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156107 s, 6.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xd6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526116 s, 19.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xd7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xd8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 10:36:52 (1788791812) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.246802 s, 8.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.520061 s, 8.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.26981 s, 7.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 10:37:11 (1788791831) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 10:37:14 (1788791834) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:37:18 (1788791838) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552615 s, 19.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (15s) == sanity-flr test 44a: lfs mirror split check =========== 10:37:34 (1788791854) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (7s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:37:40 (1788791860) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:37:52 (1788791872) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.43033 s, 24.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg117-client.virtnet /mnt/lustre (opts:) Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (15s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:38:07 (1788791887) llite.lustre-ffff93eb84805000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.402674 s, 26.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg117-client.virtnet /mnt/lustre (opts:) Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff93eb90cee000.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:38:23 (1788791903) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (46s) == sanity-flr test 46: Verify setstripe --copy option ==== 10:39:09 (1788791949) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (48s) == sanity-flr test 47: Verify mirror obj alloc =========== 10:39:57 (1788791997) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 10:40:01 (1788792001) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.239594 s, 13.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.146459 s, 21.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x5bf182d2 (nosync,stale) Mirror 2: 0x5bf182d2 (nosync,stale) ** verify mirror 2 doesn't change original checksum: f4ced89277cf01872d8683a453636185 - mirror 1 checksum: 393aa93621ff0da0c7e7a4bfa46b081a - mirror 2 checksum: f4ced89277cf01872d8683a453636185 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-07T10:40:06-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xf5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-07T10:40:06-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xf6:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: f4ced89277cf01872d8683a453636185 - mirror 1 checksum: 393aa93621ff0da0c7e7a4bfa46b081a - mirror 2 checksum: 393aa93621ff0da0c7e7a4bfa46b081a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-07T10:40:06-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xf5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-07T10:40:06-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xf6:0x0] } PASS 48 (14s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 10:40:17 (1788792017) Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server 10:40:26 (1788792026) shut down facet: mds1 facet_host: oleg117-server facet_failover_host: oleg117-server Failover mds1 to oleg117-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 10:41:02 (1788792062) targets are mounted 10:41:02 (1788792062) facet_failover done oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473677 s, 22.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xf7:0x0] } PASS 50A (67s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 10:41:24 (1788792084) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 10:41:28 (1788792088) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 10:41:32 (1788792092) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0133265 s, 1.2 MB/s PASS 50c (11s) == sanity-flr test 50d: mirror rsync keep holes ========== 10:41:43 (1788792103) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 10:41:47 (1788792107) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00774916 s, 529 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0629907 s, 65.0 kB/s PASS 60a (11s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 10:41:58 (1788792118) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (29s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 10:42:27 (1788792147) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 10:42:58 (1788792178) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 10:43:22 (1788792202) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (27s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 10:43:49 (1788792229) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (10s) == sanity-flr test 70: mirror create and split race ====== 10:43:59 (1788792239) mirror create pid 54360 mirror split pid 54362 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54360 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54362 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 10:45:08 (1788792308) Seed set to 2310 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read 000012 1788792310.723298 mapwrite 0x027b24 thru 0x02f00a (0x74e7 bytes) 000013 1788792310.807440 mapwrite 0x002005 thru 0x00960d (0x7609 bytes) 000019 1788792310.866357 mapwrite 0x025711 thru 0x02d214 (0x7b04 bytes) 000023 1788792310.938421 read 0x026c4c thru 0x02f00a (0x83bf bytes) truncating to largest ever: 0x30602 000025 1788792310.940555 trunc from 0x02f00b to 0x030601 (0x15f7 bytes) 000026 1788792310.970895 write 0x03a0b3 thru 0x03ffff (0x5f4d bytes) 000027 1788792310.983221 MIRROR_EXTEND to 2 mirrors 000029 1788792312.326045 write 0x02552a thru 0x0328ed (0xd3c4 bytes) 000032 1788792312.506785 write 0x00c60c thru 0x0157d5 (0x91ca bytes) 000034 1788792312.527567 mapwrite 0x039af4 thru 0x03ffff (0x650c bytes) 000040 1788792312.680217 read 0x022ced thru 0x023ced (0x1001 bytes) 000041 1788792312.697539 read 0x034c90 thru 0x034e11 (0x182 bytes) 000042 1788792312.698410 mapread 0x00c8a0 thru 0x00e4ad (0x1c0e bytes) 000043 1788792312.699356 mapread 0x032a39 thru 0x03ffff (0xd5c7 bytes) 000046 1788792312.754326 write 0x03f531 thru 0x03ffff (0xacf bytes) 000047 1788792312.755530 write 0x02bb45 thru 0x03612b (0xa5e7 bytes) 000048 1788792312.786749 MIRROR_RESYNC 2 mirrors 000050 1788792313.701469 mapread 0x00ca45 thru 0x0123ec (0x59a8 bytes) 000051 1788792313.754296 mapwrite 0x03be49 thru 0x03ffff (0x41b7 bytes) 000052 1788792314.069452 write 0x00b0e1 thru 0x018dba (0xdcda bytes) 000053 1788792314.109865 read 0x026a14 thru 0x03526b (0xe858 bytes) 000054 1788792314.128375 write 0x01e5c6 thru 0x028868 (0xa2a3 bytes) 000055 1788792314.168383 MIRROR_RESYNC 2 mirrors 000057 1788792316.470922 trunc from 0x040000 to 0x00e773 (0x3188c bytes) 000059 1788792316.769798 mapread 0x006321 thru 0x00e773 (0x8453 bytes) 000060 1788792316.827729 write 0x01ab5b thru 0x028329 (0xd7cf bytes) 000061 1788792316.829503 read 0x00507b thru 0x0082a4 (0x322a bytes) 000062 1788792316.842451 read 0x015da0 thru 0x01f7de (0x9a3f bytes) 000065 1788792316.873112 mapwrite 0x037987 thru 0x03ffff (0x8679 bytes) 000066 1788792317.112129 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16a:0x0]: Invalid argument (22) 000068 1788792317.417001 mapwrite 0x009b01 thru 0x00cd6e (0x326e bytes) 000069 1788792317.562907 mapwrite 0x03f4ef thru 0x03ffff (0xb11 bytes) 000071 1788792317.688084 mapread 0x0270b7 thru 0x02ffa9 (0x8ef3 bytes) 000073 1788792317.711942 read 0x01ff91 thru 0x029400 (0x9470 bytes) 000075 1788792317.739313 trunc from 0x040000 to 0x0149c4 (0x2b63b bytes) 000078 1788792317.784711 mapread 0x00f88b thru 0x0149c4 (0x513a bytes) 000084 1788792317.826299 trunc from 0x0149c5 to 0x02258f (0xdbcb bytes) 000085 1788792317.859867 read 0x0108c5 thru 0x0131aa (0x28e6 bytes) 000086 1788792317.870046 mapwrite 0x020224 thru 0x02be4a (0xbc27 bytes) 000087 1788792318.088376 write 0x022e01 thru 0x02d5b3 (0xa7b3 bytes) 000088 1788792318.096346 read 0x0270f8 thru 0x02cf0d (0x5e16 bytes) 000091 1788792318.100403 trunc from 0x02d5b4 to 0x0286cd (0x4ee6 bytes) 000093 1788792318.122764 mapread 0x00a9a4 thru 0x015ba6 (0xb203 bytes) 000094 1788792318.123853 trunc from 0x0286ce to 0x018a04 (0xfcc9 bytes) 000095 1788792318.137854 read 0x008e2d thru 0x00ad0d (0x1ee1 bytes) 000097 1788792318.139223 mapwrite 0x038335 thru 0x03e69c (0x6368 bytes) 000098 1788792318.303432 mapwrite 0x007457 thru 0x0109cf (0x9579 bytes) 000100 1788792318.464331 read 0x017246 thru 0x01e215 (0x6fd0 bytes) 000101 1788792318.465003 mapwrite 0x02ed36 thru 0x035b83 (0x6e4e bytes) 000102 1788792318.573876 mapwrite 0x0203c5 thru 0x02f334 (0xef70 bytes) 000103 1788792318.690801 mapwrite 0x02b8a3 thru 0x039348 (0xdaa6 bytes) 000105 1788792318.802308 mapread 0x00d813 thru 0x013933 (0x6121 bytes) 000108 1788792318.804219 read 0x00a671 thru 0x01621d (0xbbad bytes) 000110 1788792318.808334 mapread 0x01bb96 thru 0x01dfe5 (0x2450 bytes) 000113 1788792318.811607 mapwrite 0x029b18 thru 0x02ee41 (0x532a bytes) 000115 1788792318.894678 trunc from 0x03e69d to 0x009eed (0x347af bytes) 000118 1788792318.971747 trunc from 0x009eee to 0x007f12 (0x1fdb bytes) 000119 1788792319.008234 trunc from 0x007f13 to 0x007878 (0x69a bytes) 000124 1788792319.083861 MIRROR_EXTEND to 2 mirrors 000125 1788792320.772529 mapwrite 0x001a76 thru 0x0068a3 (0x4e2e bytes) 000126 1788792321.211121 write 0x01189f thru 0x01a7bf (0x8f21 bytes) 000127 1788792321.216984 read 0x012e5a thru 0x0177b7 (0x495e bytes) 000130 1788792321.234693 MIRROR_RESYNC 2 mirrors 000131 1788792322.188604 trunc from 0x01a7c0 to 0x003a00 (0x16dbf bytes) 000133 1788792322.304601 mapwrite 0x010d32 thru 0x012c65 (0x1f34 bytes) 000134 1788792322.445081 read 0x00f45b thru 0x012c65 (0x380b bytes) 000135 1788792322.449072 trunc from 0x012c66 to 0x008ab4 (0xa1b1 bytes) 000136 1788792322.477267 trunc from 0x008ab5 to 0x004961 (0x4153 bytes) 000137 1788792322.504888 write 0x00f870 thru 0x01422e (0x49bf bytes) 000138 1788792322.506332 mapread 0x00cceb thru 0x01422e (0x7544 bytes) 000139 1788792322.531365 MIRROR_RESYNC 2 mirrors truncating to largest ever: 0x3696c 000144 1788792324.046456 trunc from 0x01422f to 0x03696b (0x2273d bytes) 000145 1788792324.283878 mapwrite 0x02f2a0 thru 0x038eb1 (0x9c12 bytes) 000147 1788792324.629297 mapwrite 0x030ce9 thru 0x03dcb0 (0xcfc8 bytes) 000148 1788792324.932089 mapwrite 0x0049cf thru 0x009646 (0x4c78 bytes) 000149 1788792325.132100 trunc from 0x03dcb1 to 0x010920 (0x2d390 bytes) 000151 1788792325.176747 read 0x005e11 thru 0x010920 (0xab10 bytes) 000152 1788792325.183775 mapwrite 0x02ee62 thru 0x03b7c6 (0xc965 bytes) 000153 1788792325.407757 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b7c7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b7c7): Mirror 2: 0x7827140f (stale) CRC-32 checksum value for chunk [0, 0x3b7c7): Mirror 1: 0x2b30e2ba Mirror 3: 0x2b30e2ba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b7c7: skipped 000154 1788792327.202628 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b7c7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b7c7): Mirror 2: 0x7827140f (stale) CRC-32 checksum value for chunk [0, 0x3b7c7): Mirror 1: 0x2b30e2ba Mirror 3: 0x2b30e2ba Mirror 4: 0x2b30e2ba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b7c7: skipped 000158 1788792329.654890 read 0x02dd3f thru 0x02fddc (0x209e bytes) 000160 1788792329.723053 write 0x01b64a thru 0x01c431 (0xde8 bytes) 000162 1788792329.969593 read 0x012806 thru 0x01b228 (0x8a23 bytes) 000165 1788792330.027501 mapread 0x013d28 thru 0x015a33 (0x1d0c bytes) 000166 1788792330.030934 read 0x020ea7 thru 0x025c58 (0x4db2 bytes) 000167 1788792330.035013 read 0x019aca thru 0x028ba9 (0xf0e0 bytes) 000168 1788792330.039281 write 0x026894 thru 0x0348a5 (0xe012 bytes) 000170 1788792330.049634 mapread 0x007898 thru 0x012266 (0xa9cf bytes) 000173 1788792330.058500 trunc from 0x03b7c7 to 0x00ccc6 (0x2eb00 bytes) 000174 1788792330.152613 write 0x034066 thru 0x03ffff (0xbf9a bytes) 000176 1788792330.162030 mapread 0x0115e7 thru 0x013230 (0x1c4a bytes) 000177 1788792330.177901 mapread 0x030001 thru 0x03a736 (0xa736 bytes) 000182 1788792330.191763 MIRROR_RESYNC 4 mirrors 000183 1788792332.844554 write 0x0228b4 thru 0x02c694 (0x9de1 bytes) 000185 1788792332.999618 write 0x01f806 thru 0x02ec45 (0xf440 bytes) 000187 1788792333.022198 trunc from 0x040000 to 0x018a7d (0x27582 bytes) 000188 1788792333.062794 mapread 0x001798 thru 0x00e018 (0xc881 bytes) 000189 1788792333.101493 read 0x00198b thru 0x00ea7a (0xd0f0 bytes) 000192 1788792333.103799 write 0x01c981 thru 0x023469 (0x6ae9 bytes) 000193 1788792333.113898 mapwrite 0x03068b thru 0x035228 (0x4b9e bytes) 000195 1788792333.196400 read 0x0103ae thru 0x012969 (0x25bc bytes) 000196 1788792333.214359 mapwrite 0x01ebb3 thru 0x02e347 (0xf795 bytes) 000198 1788792333.299305 mapread 0x0056db thru 0x0148b4 (0xf1da bytes) 000199 1788792333.300339 mapread 0x032784 thru 0x035228 (0x2aa5 bytes) 000200 1788792333.300963 write 0x03e48f thru 0x03ffff (0x1b71 bytes) 000202 1788792333.308592 read 0x039bf6 thru 0x03ffff (0x640a bytes) 000203 1788792333.351591 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xc3ad9054 (stale) Mirror 4: 0xc3ad9054 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xc3ad9054 (stale) Mirror 4: 0xc3ad9054 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000204 1788792333.917575 MIRROR_RESYNC 3 mirrors truncating to largest ever: 0x3ce40 000205 1788792335.679443 trunc from 0x040000 to 0x03ce3f (0x31c0 bytes) 000206 1788792335.919517 write 0x02d21d thru 0x0309c6 (0x37aa bytes) 000207 1788792335.945956 trunc from 0x03ce40 to 0x00266e (0x3a7d1 bytes) 000208 1788792335.991478 mapwrite 0x010135 thru 0x01706f (0x6f3b bytes) 000210 1788792336.222683 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17070): Mirror 4: 0x67f34d5d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17070): Mirror 4: 0x67f34d5d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17070: skipped 000211 1788792336.521346 mapwrite 0x01f420 thru 0x02cfb8 (0xdb99 bytes) 000212 1788792336.644974 MIRROR_RESYNC 2 mirrors 000213 1788792337.361900 mapwrite 0x03c7ce thru 0x03ffff (0x3832 bytes) 000214 1788792337.497934 read 0x01e394 thru 0x02906b (0xacd8 bytes) 000216 1788792337.509979 mapread 0x00003c thru 0x0017a5 (0x176a bytes) 000217 1788792337.517699 read 0x00af46 thru 0x01a8f4 (0xf9af bytes) 000219 1788792337.528667 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16a:0x0]: Invalid argument (22) 000220 1788792337.681483 mapread 0x01073a thru 0x01fab3 (0xf37a bytes) 000223 1788792337.712333 read 0x02c6f6 thru 0x0342d3 (0x7bde bytes) 000224 1788792337.722132 mapread 0x01f9ba thru 0x02ac97 (0xb2de bytes) 000227 1788792337.731585 trunc from 0x040000 to 0x033067 (0xcf98 bytes) 000230 1788792337.745547 trunc from 0x033068 to 0x00840c (0x2ac5b bytes) 000231 1788792337.766248 mapwrite 0x01ad74 thru 0x021888 (0x6b15 bytes) 000233 1788792337.830419 mapread 0x012152 thru 0x01e313 (0xc1c2 bytes) 000235 1788792337.838319 read 0x0082d4 thru 0x017337 (0xf064 bytes) 000236 1788792337.844487 write 0x00a96d thru 0x00de83 (0x3517 bytes) 000237 1788792337.846375 read 0x01f462 thru 0x021888 (0x2427 bytes) 000239 1788792337.848470 trunc from 0x021889 to 0x0030c0 (0x1e7c8 bytes) 000240 1788792337.866894 write 0x019a1c thru 0x01af5f (0x1544 bytes) 000241 1788792337.875513 MIRROR_EXTEND to 2 mirrors 000243 1788792338.668831 write 0x00ed05 thru 0x01bdae (0xd0aa bytes) 000244 1788792338.743820 write 0x00e111 thru 0x01630a (0x81fa bytes) 000245 1788792338.751055 MIRROR_RESYNC 2 mirrors 000246 1788792339.411987 read 0x016901 thru 0x01bdae (0x54ae bytes) 000247 1788792339.421235 read 0x006b76 thru 0x00aea3 (0x432e bytes) 000248 1788792339.432244 mapread 0x00cd0e thru 0x013a8e (0x6d81 bytes) 000249 1788792339.438019 MIRROR_RESYNC 2 mirrors 000250 1788792339.901096 write 0x02b2c6 thru 0x02bd8b (0xac6 bytes) 000251 1788792340.031596 mapwrite 0x03b192 thru 0x03ffff (0x4e6e bytes) 000252 1788792340.122615 write 0x00b30f thru 0x01a30d (0xefff bytes) 000253 1788792340.150251 mapread 0x01f3e0 thru 0x021e3d (0x2a5e bytes) 000254 1788792340.179406 mapread 0x004b04 thru 0x00626d (0x176a bytes) 000255 1788792340.232704 trunc from 0x040000 to 0x031e2a (0xe1d5 bytes) 000257 1788792340.264141 read 0x018c9c thru 0x022b97 (0x9efc bytes) 000258 1788792340.268634 write 0x027ec5 thru 0x033360 (0xb49c bytes) 000262 1788792340.287326 read 0x029550 thru 0x0329bf (0x9470 bytes) 000264 1788792340.288407 write 0x001c23 thru 0x00fd97 (0xe175 bytes) 000272 1788792340.317116 trunc from 0x033361 to 0x02b572 (0x7dee bytes) 000273 1788792340.357122 write 0x022db4 thru 0x02edf0 (0xc03d bytes) 000274 1788792340.360134 read 0x00b1b9 thru 0x014ecb (0x9d13 bytes) 000276 1788792340.362280 trunc from 0x02edf1 to 0x02ae9f (0x3f51 bytes) 000277 1788792340.387736 read 0x00ac54 thru 0x015cce (0xb07b bytes) 000279 1788792340.394423 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2aea0): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2aea0): Mirror 2: 0x70593587 (stale) CRC-32 checksum value for chunk [0, 0x2aea0): Mirror 1: 0xd422871f Mirror 3: 0xd422871f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2aea0: skipped 000280 1788792341.249981 read 0x02ab99 thru 0x02ae9f (0x307 bytes) 000282 1788792341.259032 trunc from 0x02aea0 to 0x02a996 (0x509 bytes) 000283 1788792341.397564 write 0x00db84 thru 0x00ee88 (0x1305 bytes) 000284 1788792341.406425 trunc from 0x02a997 to 0x035ac6 (0xb130 bytes) 000287 1788792341.421384 read 0x01547f thru 0x0243ac (0xef2e bytes) 000288 1788792341.435280 MIRROR_RESYNC 3 mirrors 000291 1788792342.447140 mapread 0x02857b thru 0x02de67 (0x58ed bytes) 000292 1788792342.478834 mapread 0x01703c thru 0x01ef78 (0x7f3d bytes) 000294 1788792342.489144 mapread 0x009abe thru 0x011ec6 (0x8409 bytes) 000295 1788792342.495819 read 0x011285 thru 0x012d4c (0x1ac8 bytes) 000296 1788792342.496257 write 0x031cef thru 0x03e4be (0xc7d0 bytes) 000297 1788792342.561485 trunc from 0x03e4bf to 0x01b46f (0x2304f bytes) 000298 1788792342.588210 trunc from 0x01b470 to 0x01f87d (0x440e bytes) 000299 1788792342.631061 mapread 0x00bfd9 thru 0x01504d (0x9075 bytes) 000300 1788792342.674984 mapread 0x002404 thru 0x008abb (0x66b8 bytes) 000301 1788792342.690357 trunc from 0x01f87e to 0x000eeb (0x1e992 bytes) 000302 1788792342.718523 write 0x0107b0 thru 0x01ac86 (0xa4d7 bytes) 000304 1788792342.737208 read 0x004aee thru 0x004ffe (0x511 bytes) 000309 1788792342.775285 mapread 0x006729 thru 0x00a039 (0x3911 bytes) 000310 1788792342.779551 trunc from 0x01ac87 to 0x00bfb4 (0xecd2 bytes) 000312 1788792342.809028 mapread 0x002629 thru 0x008c62 (0x663a bytes) 000313 1788792342.810312 trunc from 0x00bfb5 to 0x01f2e1 (0x1332d bytes) 000314 1788792342.829300 mapwrite 0x002062 thru 0x005f8a (0x3f29 bytes) 000315 1788792342.895313 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f2e2): Mirror 3: 0xc9760081 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f2e2): Mirror 3: 0xc9760081 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f2e2: skipped 000317 1788792343.202995 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16a:0x0]: Invalid argument (22) 000319 1788792343.354522 write 0x01f413 thru 0x0261d0 (0x6dbe bytes) 000323 1788792343.374254 mapread 0x007c53 thru 0x01561e (0xd9cc bytes) 000324 1788792343.410391 mapwrite 0x02b869 thru 0x034a82 (0x921a bytes) 000326 1788792343.513652 write 0x00274c thru 0x010298 (0xdb4d bytes) 000327 1788792343.518644 trunc from 0x034a83 to 0x037bb0 (0x312e bytes) 000329 1788792343.556357 read 0x02b3f5 thru 0x03410c (0x8d18 bytes) 000330 1788792343.556894 trunc from 0x037bb1 to 0x00e6a5 (0x2950b bytes) 000331 1788792343.576258 mapread 0x009574 thru 0x00e6a5 (0x5132 bytes) 000333 1788792343.580522 trunc from 0x00e6a6 to 0x03bab7 (0x2d412 bytes) 000334 1788792343.605351 write 0x0213d7 thru 0x0278c9 (0x64f3 bytes) 000335 1788792343.626629 mapread 0x026b07 thru 0x02769f (0xb99 bytes) 000337 1788792343.632578 write 0x032756 thru 0x034fbf (0x286a bytes) 000338 1788792343.647368 read 0x037dbb thru 0x03bab7 (0x3cfd bytes) 000339 1788792343.660066 trunc from 0x03bab8 to 0x027d25 (0x13d92 bytes) 000340 1788792343.700953 mapwrite 0x031550 thru 0x03c0f9 (0xabaa bytes) 000341 1788792343.998275 mapwrite 0x034ecb thru 0x03a810 (0x5946 bytes) 000342 1788792344.146980 read 0x032998 thru 0x03afd4 (0x863d bytes) 000347 1788792344.164843 write 0x016e91 thru 0x01dc18 (0x6d88 bytes) 000353 1788792344.178917 read 0x036d86 thru 0x03c0f9 (0x5374 bytes) 000354 1788792344.182029 mapread 0x0222f7 thru 0x02707c (0x4d86 bytes) 000355 1788792344.191318 read 0x02baec thru 0x032426 (0x693b bytes) 000356 1788792344.205016 write 0x00b545 thru 0x0149af (0x946b bytes) 000358 1788792344.206683 write 0x0219af thru 0x02f77b (0xddcd bytes) 000359 1788792344.209919 mapread 0x02e840 thru 0x03a676 (0xbe37 bytes) 000361 1788792344.212096 read 0x0249c3 thru 0x02bb42 (0x7180 bytes) 000364 1788792344.214118 read 0x00fb98 thru 0x01cd3b (0xd1a4 bytes) 000365 1788792344.215053 read 0x029feb thru 0x02e297 (0x42ad bytes) 000369 1788792344.223546 trunc from 0x03c0fa to 0x00d81d (0x2e8dc bytes) 000370 1788792344.274534 mapwrite 0x02848a thru 0x02c44d (0x3fc4 bytes) 000376 1788792344.437455 trunc from 0x02c44e to 0x00a44c (0x22001 bytes) 000377 1788792344.480917 mapwrite 0x027e18 thru 0x02e39e (0x6587 bytes) 000379 1788792344.654678 read 0x02dd88 thru 0x02e39e (0x617 bytes) 000380 1788792344.657449 mapwrite 0x02501f thru 0x0257a0 (0x782 bytes) 000381 1788792344.736965 trunc from 0x02e39f to 0x03c6b1 (0xe313 bytes) 000384 1788792344.760632 write 0x0291d4 thru 0x02a0d5 (0xf02 bytes) 000385 1788792344.762058 trunc from 0x03c6b2 to 0x03ab1f (0x1b92 bytes) 000386 1788792344.793429 read 0x01f33c thru 0x02cc11 (0xd8d6 bytes) 000388 1788792344.796374 read 0x02aab1 thru 0x036ff3 (0xc543 bytes) 000389 1788792344.811763 read 0x02ae11 thru 0x03276d (0x795d bytes) 000390 1788792344.812947 read 0x01ccc7 thru 0x01f4d5 (0x280f bytes) 000392 1788792344.816693 trunc from 0x03ab20 to 0x017315 (0x2380a bytes) 000393 1788792344.853751 mapread 0x013689 thru 0x017315 (0x3c8d bytes) 000395 1788792344.861128 write 0x00fe9b thru 0x015cf2 (0x5e58 bytes) 000400 1788792344.868558 mapread 0x00dcbe thru 0x017315 (0x9658 bytes) 000402 1788792344.881420 MIRROR_EXTEND to 2 mirrors 000403 1788792346.000413 mapwrite 0x02a01a thru 0x02bc3a (0x1c21 bytes) 000404 1788792346.346063 trunc from 0x02bc3b to 0x003e09 (0x27e31 bytes) 000405 1788792346.403416 trunc from 0x003e0a to 0x015c30 (0x11e27 bytes) 000408 1788792346.466549 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16a:0x0]: Invalid argument (22) 000409 1788792346.700280 mapread 0x005a7d thru 0x013083 (0xd607 bytes) 000410 1788792346.744216 trunc from 0x015c31 to 0x030011 (0x1a3e1 bytes) 000413 1788792346.795432 mapwrite 0x010c56 thru 0x01be96 (0xb241 bytes) 000414 1788792346.903194 read 0x000c39 thru 0x0095bf (0x8987 bytes) 000415 1788792346.930379 mapwrite 0x022ffb thru 0x02449e (0x14a4 bytes) 000416 1788792347.038849 mapwrite 0x017ed5 thru 0x024de2 (0xcf0e bytes) 000417 1788792347.064297 mapwrite 0x03caa4 thru 0x03d479 (0x9d6 bytes) 000418 1788792347.126181 trunc from 0x03d47a to 0x01ba47 (0x21a32 bytes) 000427 1788792347.159552 mapread 0x00c0f1 thru 0x015212 (0x9122 bytes) 000428 1788792347.167528 trunc from 0x01ba48 to 0x02fc45 (0x141fe bytes) 000436 1788792347.215992 write 0x018b66 thru 0x027837 (0xecd2 bytes) 000437 1788792347.230394 mapwrite 0x032465 thru 0x03ffff (0xdb9b bytes) 000439 1788792347.346362 mapread 0x027d33 thru 0x02eae6 (0x6db4 bytes) 000440 1788792347.351903 mapread 0x033568 thru 0x03da58 (0xa4f1 bytes) 000441 1788792347.353321 mapwrite 0x03f1c6 thru 0x03ffff (0xe3a bytes) 000442 1788792347.374845 mapread 0x005dde thru 0x0060b6 (0x2d9 bytes) 000444 1788792347.376049 write 0x02e4c1 thru 0x03d4c2 (0xf002 bytes) 000446 1788792347.377597 read 0x018cfd thru 0x019c10 (0xf14 bytes) 000450 1788792347.381139 read 0x00a8c8 thru 0x011c83 (0x73bc bytes) 000451 1788792347.381966 trunc from 0x040000 to 0x03016c (0xfe93 bytes) 000452 1788792347.394341 mapwrite 0x0201e6 thru 0x03005f (0xfe7a bytes) 000453 1788792347.433240 mapwrite 0x012064 thru 0x017b58 (0x5af5 bytes) 000454 1788792347.478906 mapwrite 0x0110b1 thru 0x012789 (0x16d9 bytes) 000455 1788792347.529278 read 0x01dc4a thru 0x029cf4 (0xc0ab bytes) 000456 1788792347.530362 mapread 0x019e20 thru 0x027200 (0xd3e1 bytes) 000458 1788792347.533519 write 0x02a993 thru 0x03a35c (0xf9ca bytes) 000461 1788792347.538063 mapwrite 0x00f348 thru 0x01816d (0x8e26 bytes) 000464 1788792347.592197 mapread 0x016616 thru 0x01a811 (0x41fc bytes) 000465 1788792347.597343 trunc from 0x03a35d to 0x028b14 (0x11848 bytes) 000467 1788792347.617873 write 0x029551 thru 0x037c45 (0xe6f5 bytes) 000469 1788792347.623596 write 0x029b2f thru 0x030202 (0x66d4 bytes) 000470 1788792347.625038 read 0x0178cc thru 0x0262eb (0xea20 bytes) 000473 1788792347.628771 trunc from 0x037c46 to 0x01923c (0x1ea09 bytes) 000475 1788792347.664532 trunc from 0x01923d to 0x031904 (0x186c8 bytes) 000476 1788792347.691561 mapwrite 0x0019d0 thru 0x006fc0 (0x55f1 bytes) 000477 1788792347.770850 mapwrite 0x00db76 thru 0x00e58c (0xa17 bytes) 000482 1788792347.824967 read 0x026b78 thru 0x02a49f (0x3928 bytes) 000483 1788792347.839502 mapwrite 0x039852 thru 0x03ffff (0x67ae bytes) 000484 1788792347.923589 read 0x00482c thru 0x00dfa9 (0x977e bytes) 000488 1788792347.926856 mapread 0x0006b5 thru 0x009ed3 (0x981f bytes) 000491 1788792347.929830 mapwrite 0x000036 thru 0x008108 (0x80d3 bytes) 000499 1788792347.976858 trunc from 0x040000 to 0x001752 (0x3e8ad bytes) 000501 1788792348.007507 MIRROR_EXTEND to 2 mirrors 000502 1788792349.062098 mapwrite 0x0077ca thru 0x00c19d (0x49d4 bytes) 000503 1788792349.252145 read 0x009712 thru 0x00ba91 (0x2380 bytes) 000505 1788792349.254447 read 0x009518 thru 0x00c19d (0x2c86 bytes) 000506 1788792349.262518 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16a:0x0]: Invalid argument (22) 000508 1788792349.521982 mapwrite 0x028987 thru 0x02b3fd (0x2a77 bytes) 000511 1788792349.681585 read 0x013e46 thru 0x014b44 (0xcff bytes) 000512 1788792349.686878 write 0x012446 thru 0x01ea7c (0xc637 bytes) 000513 1788792349.695988 read 0x000214 thru 0x00d4be (0xd2ab bytes) 000514 1788792349.718435 mapread 0x007771 thru 0x009728 (0x1fb8 bytes) 000515 1788792349.729011 MIRROR_EXTEND to 2 mirrors 000516 1788792350.760039 read 0x00f708 thru 0x019ea9 (0xa7a2 bytes) 000517 1788792350.783282 trunc from 0x02b3fe to 0x02c9f9 (0x15fc bytes) 000519 1788792350.884471 mapwrite 0x034854 thru 0x03e66c (0x9e19 bytes) 000521 1788792351.018119 mapread 0x0098e1 thru 0x01684b (0xcf6b bytes) 000522 1788792351.060566 write 0x010bd2 thru 0x015bb2 (0x4fe1 bytes) 000524 1788792351.065276 trunc from 0x03e66d to 0x00d907 (0x30d65 bytes) 000527 1788792351.100817 trunc from 0x00d908 to 0x000876 (0xd091 bytes) 000529 1788792351.132562 trunc from 0x000877 to 0x01917e (0x18908 bytes) 000531 1788792351.158982 mapwrite 0x0260a4 thru 0x027e5a (0x1db7 bytes) 000532 1788792351.240061 mapwrite 0x03ae19 thru 0x03ffff (0x51e7 bytes) 000534 1788792351.378511 mapwrite 0x02553a thru 0x02a863 (0x532a bytes) 000535 1788792351.439525 mapwrite 0x03c501 thru 0x03ffff (0x3aff bytes) 000536 1788792351.528559 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16a:0x0]: Invalid argument (22) 000537 1788792351.809554 mapread 0x02dd6f thru 0x036a7f (0x8d11 bytes) 000539 1788792351.842379 read 0x000620 thru 0x005f8e (0x596f bytes) 000542 1788792351.857020 mapwrite 0x018d63 thru 0x024d52 (0xbff0 bytes) 000545 1788792351.947027 trunc from 0x040000 to 0x006519 (0x39ae6 bytes) truncating to largest ever: 0x3e779 000548 1788792351.967229 trunc from 0x00651a to 0x03e778 (0x3825f bytes) 000550 1788792351.988467 mapwrite 0x00b037 thru 0x019ccf (0xec99 bytes) 000551 1788792352.111098 trunc from 0x03e779 to 0x03d2b0 (0x14c8 bytes) 000554 1788792352.144670 read 0x0394db thru 0x03d2b0 (0x3dd6 bytes) 000555 1788792352.180344 read 0x02b67c thru 0x038d70 (0xd6f5 bytes) 000556 1788792352.188839 write 0x03eb11 thru 0x03ffff (0x14ef bytes) 000557 1788792352.197188 mapread 0x011f83 thru 0x01ab0d (0x8b8b bytes) 000558 1788792352.201358 trunc from 0x040000 to 0x0165b1 (0x29a4e bytes) 000559 1788792352.247635 mapread 0x006d07 thru 0x00afad (0x42a7 bytes) 000561 1788792352.251443 mapread 0x01229b thru 0x0165b1 (0x4317 bytes) 000562 1788792352.255249 write 0x012529 thru 0x0137c0 (0x1298 bytes) 000563 1788792352.274270 write 0x03bfec thru 0x03ffff (0x4014 bytes) 000567 1788792352.293721 mapwrite 0x0030ad thru 0x007775 (0x46c9 bytes) 000568 1788792352.390205 mapread 0x024620 thru 0x024c33 (0x614 bytes) 000569 1788792352.434957 write 0x03cbb0 thru 0x03ffff (0x3450 bytes) 000575 1788792352.451768 read 0x00894d thru 0x009761 (0xe15 bytes) 000577 1788792352.460680 read 0x0316f3 thru 0x035282 (0x3b90 bytes) 000579 1788792352.463505 mapwrite 0x013428 thru 0x01ab43 (0x771c bytes) 000580 1788792352.535464 mapread 0x009030 thru 0x018a56 (0xfa27 bytes) 000582 1788792352.538748 trunc from 0x040000 to 0x003d29 (0x3c2d6 bytes) 000583 1788792352.596922 MIRROR_EXTEND to 2 mirrors 000584 1788792353.431783 mapread 0x002d7b thru 0x003d29 (0xfaf bytes) 000585 1788792353.472109 mapread 0x0021db thru 0x003d29 (0x1b4f bytes) 000586 1788792353.487595 mapread 0x00028b thru 0x003d29 (0x3a9f bytes) 000587 1788792353.507257 write 0x034505 thru 0x038663 (0x415f bytes) 000588 1788792353.578986 mapread 0x02c940 thru 0x02cb57 (0x218 bytes) 000591 1788792353.597512 write 0x0089e6 thru 0x0183a5 (0xf9c0 bytes) 000592 1788792353.633231 read 0x006c32 thru 0x00cd0c (0x60db bytes) 000594 1788792353.645003 write 0x020c40 thru 0x023fb1 (0x3372 bytes) 000596 1788792353.658914 MIRROR_RESYNC 2 mirrors 000599 1788792354.769409 read 0x031382 thru 0x033d2b (0x29aa bytes) 000600 1788792354.775354 mapwrite 0x03c145 thru 0x03ffff (0x3ebb bytes) 000601 1788792354.917211 write 0x00efc8 thru 0x013479 (0x44b2 bytes) 000602 1788792354.953715 mapwrite 0x01c013 thru 0x026c4a (0xac38 bytes) 000603 1788792355.068485 trunc from 0x040000 to 0x0168d6 (0x29729 bytes) 000606 1788792355.093737 mapread 0x014e25 thru 0x0168d6 (0x1ab2 bytes) 000607 1788792355.104480 mapread 0x0042f3 thru 0x00de7a (0x9b88 bytes) 000608 1788792355.137026 write 0x002326 thru 0x00ef51 (0xcc2c bytes) 000609 1788792355.144189 mapwrite 0x03dab6 thru 0x03ffff (0x254a bytes) 000610 1788792355.266166 trunc from 0x040000 to 0x006737 (0x398c8 bytes) 000611 1788792355.300140 mapwrite 0x01c1b9 thru 0x026404 (0xa24c bytes) 000613 1788792355.502515 trunc from 0x026405 to 0x000728 (0x25cdc bytes) 000615 1788792355.533709 read 0x00036a thru 0x000728 (0x3bf bytes) 000617 1788792355.557349 write 0x03d30b thru 0x03ffff (0x2cf5 bytes) 000618 1788792355.567823 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16a:0x0]: Invalid argument (22) 000622 1788792355.954406 write 0x002c3f thru 0x00ab8d (0x7f4f bytes) 000623 1788792355.998541 write 0x02f130 thru 0x034928 (0x57f9 bytes) 000626 1788792356.016825 write 0x036860 thru 0x03df46 (0x76e7 bytes) 000627 1788792356.032276 write 0x007424 thru 0x00c885 (0x5462 bytes) 000631 1788792356.046456 trunc from 0x040000 to 0x02b883 (0x1477c bytes) 000634 1788792356.093356 read 0x02b5ea thru 0x02b883 (0x29a bytes) 000637 1788792356.100766 mapread 0x020600 thru 0x022b68 (0x2569 bytes) 000638 1788792356.116094 read 0x0254e0 thru 0x02b883 (0x63a4 bytes) 000640 1788792356.117270 mapwrite 0x02a9ea thru 0x02cadb (0x20f2 bytes) 000641 1788792356.240048 mapread 0x0004f8 thru 0x00a072 (0x9b7b bytes) 000643 1788792356.241552 mapwrite 0x01be20 thru 0x0200f7 (0x42d8 bytes) truncating to largest ever: 0x3f047 000644 1788792356.361995 trunc from 0x02cadc to 0x03f046 (0x1256b bytes) 000646 1788792356.409495 write 0x00f6ec thru 0x011f48 (0x285d bytes) 000647 1788792356.414030 mapwrite 0x016950 thru 0x02664a (0xfcfb bytes) 000648 1788792356.549734 trunc from 0x03f047 to 0x02fbcf (0xf477 bytes) 000649 1788792356.585619 mapwrite 0x027ff2 thru 0x031e47 (0x9e56 bytes) 000650 1788792356.685408 read 0x004868 thru 0x0122ea (0xda83 bytes) 000655 1788792356.691117 trunc from 0x031e48 to 0x01990f (0x18538 bytes) 000656 1788792356.710903 write 0x03aee7 thru 0x03cbe0 (0x1cfa bytes) 000657 1788792356.712480 mapread 0x01d586 thru 0x021548 (0x3fc3 bytes) 000659 1788792356.722820 trunc from 0x03cbe1 to 0x03e525 (0x1945 bytes) 000661 1788792356.752941 write 0x01cd79 thru 0x02827d (0xb505 bytes) 000662 1788792356.754968 mapwrite 0x014e5a thru 0x020c3a (0xbde1 bytes) 000664 1788792356.864453 mapread 0x010177 thru 0x010a11 (0x89b bytes) 000665 1788792356.866329 mapread 0x0056cf thru 0x00719c (0x1ace bytes) 000666 1788792356.867772 mapread 0x03922b thru 0x03e525 (0x52fb bytes) 000667 1788792356.876670 mapread 0x01e04b thru 0x029f86 (0xbf3c bytes) 000668 1788792356.878398 trunc from 0x03e526 to 0x03d8c5 (0xc60 bytes) 000669 1788792356.901937 trunc from 0x03d8c6 to 0x015e72 (0x27a53 bytes) 000670 1788792356.937328 read 0x00784c thru 0x00bbf4 (0x43a9 bytes) 000672 1788792356.940352 write 0x01d2a6 thru 0x0215b6 (0x4311 bytes) 000674 1788792356.943881 mapwrite 0x0154a6 thru 0x0209c3 (0xb51e bytes) 000675 1788792357.011444 mapwrite 0x028f98 thru 0x02b4b0 (0x2519 bytes) 000676 1788792357.108188 write 0x02c58f thru 0x03927d (0xccef bytes) 000678 1788792357.117103 trunc from 0x03927e to 0x028179 (0x11104 bytes) 000679 1788792357.146075 write 0x025e62 thru 0x02b8be (0x5a5d bytes) 000681 1788792357.149733 write 0x01df78 thru 0x023a49 (0x5ad2 bytes) 000682 1788792357.152086 write 0x025b85 thru 0x02d83c (0x7cb8 bytes) 000685 1788792357.156908 mapread 0x015c78 thru 0x01e321 (0x86aa bytes) 000687 1788792357.159931 mapread 0x029d33 thru 0x02d83c (0x3b0a bytes) 000688 1788792357.161373 read 0x0124cb thru 0x01915b (0x6c91 bytes) 000692 1788792357.165980 read 0x026eed thru 0x02c559 (0x566d bytes) 000695 1788792357.169421 write 0x02ff2a thru 0x0361fc (0x62d3 bytes) 000696 1788792357.172021 read 0x00e01e thru 0x01d15b (0xf13e bytes) 000697 1788792357.181937 MIRROR_EXTEND to 2 mirrors 000698 1788792358.069598 mapread 0x00b89d thru 0x0102c8 (0x4a2c bytes) 000699 1788792358.108958 mapwrite 0x005497 thru 0x00647d (0xfe7 bytes) 000700 1788792358.421757 read 0x014fba thru 0x01c481 (0x74c8 bytes) 000701 1788792358.452094 MIRROR_RESYNC 2 mirrors 000703 1788792359.437999 read 0x00592f thru 0x00a20f (0x48e1 bytes) 000704 1788792359.457483 read 0x021831 thru 0x030fee (0xf7be bytes) 000705 1788792359.474565 write 0x005219 thru 0x013f2d (0xed15 bytes) 000706 1788792359.544623 mapread 0x013486 thru 0x019e5b (0x69d6 bytes) 000708 1788792359.574086 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16a:0x0]: Invalid argument (22) 000718 1788792359.790414 mapwrite 0x01bbef thru 0x028a00 (0xce12 bytes) 000720 1788792359.944790 trunc from 0x0361fd to 0x004935 (0x318c7 bytes) 000723 1788792360.042292 read 0x001a61 thru 0x0028d1 (0xe71 bytes) 000724 1788792360.049312 mapread 0x00142e thru 0x004935 (0x3508 bytes) 000726 1788792360.058378 trunc from 0x004936 to 0x01cdd6 (0x184a1 bytes) 000727 1788792360.089109 mapwrite 0x01de2e thru 0x01e579 (0x74c bytes) 000728 1788792360.193606 write 0x034806 thru 0x03ffff (0xb7fa bytes) 000730 1788792360.197902 mapwrite 0x0293cd thru 0x02dd39 (0x496d bytes) 000731 1788792360.257949 write 0x0000ea thru 0x000503 (0x41a bytes) 000732 1788792360.263425 trunc from 0x040000 to 0x00de9c (0x32163 bytes) 000733 1788792360.287967 write 0x01cdb4 thru 0x023dcf (0x701c bytes) 000734 1788792360.290819 mapread 0x019069 thru 0x021441 (0x83d9 bytes) 000736 1788792360.302678 write 0x02b35c thru 0x02f0f8 (0x3d9d bytes) 000737 1788792360.304347 mapwrite 0x02434b thru 0x033565 (0xf21b bytes) 000740 1788792360.385574 read 0x02cb1f thru 0x02db5d (0x103f bytes) 000742 1788792360.387057 mapwrite 0x03da78 thru 0x03ffff (0x2588 bytes) 000743 1788792360.453559 read 0x035f94 thru 0x03ffff (0xa06c bytes) 000747 1788792360.456538 mapwrite 0x00426a thru 0x00e3e0 (0xa177 bytes) 000748 1788792360.508470 MIRROR_EXTEND to 2 mirrors 000750 1788792361.640385 mapwrite 0x025fe7 thru 0x0316fe (0xb718 bytes) 000752 1788792361.869101 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16a:0x0]: Invalid argument (22) 000760 1788792362.106957 write 0x013136 thru 0x018bf6 (0x5ac1 bytes) 000763 1788792362.122198 read 0x030df4 thru 0x039ba2 (0x8daf bytes) 000764 1788792362.140774 write 0x008ca1 thru 0x01505b (0xc3bb bytes) 000767 1788792362.161004 trunc from 0x040000 to 0x010f11 (0x2f0ee bytes) 000768 1788792362.235875 mapwrite 0x0330fa thru 0x03ffff (0xcf06 bytes) 000770 1788792362.483583 trunc from 0x040000 to 0x0229c7 (0x1d638 bytes) 000771 1788792362.505529 write 0x000edc thru 0x00acfd (0x9e22 bytes) 000775 1788792362.512601 trunc from 0x0229c8 to 0x006617 (0x1c3b0 bytes) 000779 1788792362.538071 mapwrite 0x039392 thru 0x03ad1f (0x198e bytes) 000780 1788792362.612943 trunc from 0x03ad20 to 0x03a027 (0xcf8 bytes) 000781 1788792362.629846 write 0x0020be thru 0x007378 (0x52bb bytes) 000782 1788792362.637083 write 0x02fafb thru 0x03dd49 (0xe24f bytes) 000783 1788792362.649831 mapread 0x015949 thru 0x01bf98 (0x6650 bytes) 000784 1788792362.660865 mapread 0x023787 thru 0x025a44 (0x22be bytes) 000786 1788792362.676082 trunc from 0x03dd4a to 0x03e449 (0x700 bytes) 000787 1788792362.694933 mapread 0x0348b5 thru 0x0348d6 (0x22 bytes) 000789 1788792362.698064 trunc from 0x03e44a to 0x03b514 (0x2f35 bytes) 000790 1788792362.721046 mapwrite 0x011e3d thru 0x014a27 (0x2beb bytes) 000792 1788792362.813950 mapread 0x011eaa thru 0x01fdd2 (0xdf29 bytes) 000793 1788792362.815353 mapread 0x0134a9 thru 0x01bc88 (0x87e0 bytes) 000795 1788792362.821881 mapwrite 0x0132bb thru 0x019e24 (0x6b6a bytes) 000796 1788792362.912593 mapwrite 0x0180df thru 0x026635 (0xe557 bytes) 000797 1788792362.971836 write 0x03fe26 thru 0x03ffff (0x1da bytes) 000802 1788792362.978083 read 0x00a809 thru 0x00f61a (0x4e12 bytes) 000803 1788792363.008918 mapwrite 0x039e8b thru 0x03ffff (0x6175 bytes) 000804 1788792363.090773 mapread 0x00c5d0 thru 0x01b93b (0xf36c bytes) 000805 1788792363.091947 read 0x034f2d thru 0x037b4d (0x2c21 bytes) 000806 1788792363.092551 mapwrite 0x0388c5 thru 0x03bab5 (0x31f1 bytes) 000808 1788792363.202368 mapread 0x02ffc4 thru 0x036d49 (0x6d86 bytes) 000811 1788792363.204349 write 0x03e840 thru 0x03ffff (0x17c0 bytes) 000812 1788792363.214012 mapwrite 0x0187d8 thru 0x02535f (0xcb88 bytes) 000814 1788792363.318968 trunc from 0x040000 to 0x00bbe6 (0x34419 bytes) 000815 1788792363.367753 MIRROR_EXTEND to 2 mirrors 000816 1788792364.225548 read 0x003921 thru 0x00bbe6 (0x82c6 bytes) 000819 1788792364.282699 read 0x006700 thru 0x00bbe6 (0x54e7 bytes) 000824 1788792364.350406 mapread 0x007505 thru 0x00bbe6 (0x46e2 bytes) 000825 1788792364.375649 mapwrite 0x027661 thru 0x02d011 (0x59b1 bytes) 000827 1788792364.770042 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2d012): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d012): Mirror 2: 0xc2f576c0 (stale) CRC-32 checksum value for chunk [0, 0x2d012): Mirror 1: 0x534d548f Mirror 3: 0x534d548f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d012: skipped 000828 1788792366.395018 trunc from 0x02d012 to 0x0279ac (0x5665 bytes) 000829 1788792366.521453 mapread 0x01cbe8 thru 0x0279ac (0xadc5 bytes) 000830 1788792366.546906 write 0x03f8fd thru 0x03ffff (0x703 bytes) 000831 1788792366.548510 read 0x00c6da thru 0x01b402 (0xed29 bytes) 000834 1788792366.560023 write 0x00537f thru 0x0102d9 (0xaf5b bytes) 000836 1788792366.566043 write 0x02a991 thru 0x038331 (0xd9a1 bytes) 000839 1788792366.581622 MIRROR_RESYNC 3 mirrors 000840 1788792367.694512 read 0x021a8e thru 0x028e9b (0x740e bytes) 000841 1788792367.714732 write 0x024fe3 thru 0x033cb3 (0xecd1 bytes) 000842 1788792367.910780 mapread 0x03c7d2 thru 0x03ffff (0x382e bytes) 000843 1788792368.000773 read 0x00969a thru 0x01645f (0xcdc6 bytes) 000844 1788792368.030914 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x3f0c41c2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x3f0c41c2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000845 1788792368.553188 write 0x0079a7 thru 0x0129a2 (0xaffc bytes) 000847 1788792368.593508 read 0x039c46 thru 0x03ffff (0x63ba bytes) 000851 1788792368.612951 mapread 0x00374e thru 0x0056a3 (0x1f56 bytes) 000853 1788792368.648518 mapread 0x034ea5 thru 0x038194 (0x32f0 bytes) 000856 1788792368.679201 mapread 0x009ee3 thru 0x013b71 (0x9c8f bytes) 000857 1788792368.713345 mapread 0x011b22 thru 0x01478f (0x2c6e bytes) 000858 1788792368.723386 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16a:0x0]: Invalid argument (22) 000859 1788792368.902708 write 0x01a0de thru 0x01ea55 (0x4978 bytes) 000861 1788792368.916259 mapread 0x02f583 thru 0x036f78 (0x79f6 bytes) 000862 1788792368.939985 trunc from 0x040000 to 0x003450 (0x3cbaf bytes) 000863 1788792368.960484 mapread 0x0028e5 thru 0x003450 (0xb6c bytes) 000865 1788792368.970348 write 0x03e100 thru 0x03ffff (0x1f00 bytes) 000867 1788792368.972669 mapread 0x020f29 thru 0x03027e (0xf356 bytes) 000868 1788792368.988009 mapwrite 0x03afaa thru 0x03ffff (0x5056 bytes) 000869 1788792369.038401 mapwrite 0x00e0a4 thru 0x0152c0 (0x721d bytes) 000871 1788792369.121989 write 0x033ecb thru 0x039b58 (0x5c8e bytes) 000874 1788792369.124594 write 0x0073d9 thru 0x011c51 (0xa879 bytes) 000875 1788792369.126198 mapread 0x032f75 thru 0x03495e (0x19ea bytes) 000877 1788792369.136931 trunc from 0x040000 to 0x03e9b5 (0x164a bytes) 000878 1788792369.167750 trunc from 0x03e9b6 to 0x001fc6 (0x3c9ef bytes) 000880 1788792369.191209 write 0x037e35 thru 0x03ffff (0x81cb bytes) 000881 1788792369.195247 mapread 0x03a3ea thru 0x03db39 (0x3750 bytes) 000882 1788792369.196452 read 0x03eda5 thru 0x03ffff (0x125b bytes) 000885 1788792369.200114 mapread 0x016687 thru 0x0172e0 (0xc5a bytes) 000887 1788792369.210299 mapread 0x030402 thru 0x030e4b (0xa4a bytes) 000888 1788792369.220251 mapwrite 0x013227 thru 0x01bd6b (0x8b45 bytes) 000889 1788792369.278979 trunc from 0x040000 to 0x00772c (0x388d3 bytes) 000892 1788792369.307495 mapread 0x006d2e thru 0x00772c (0x9ff bytes) 000894 1788792369.320502 read 0x000109 thru 0x00033e (0x236 bytes) 000895 1788792369.326773 mapread 0x0052ff thru 0x00772c (0x242e bytes) 000896 1788792369.328098 write 0x0378cb thru 0x03bff9 (0x472f bytes) 000897 1788792369.330751 mapwrite 0x031428 thru 0x03397d (0x2556 bytes) 000898 1788792369.398329 mapwrite 0x02d381 thru 0x0333a1 (0x6021 bytes) 000901 1788792369.458888 mapwrite 0x0284d3 thru 0x028dd4 (0x902 bytes) 000903 1788792369.503228 MIRROR_EXTEND to 2 mirrors 000906 1788792370.447891 mapwrite 0x010532 thru 0x016886 (0x6355 bytes) 000907 1788792370.684942 mapread 0x0012cc thru 0x007771 (0x64a6 bytes) 000908 1788792370.711048 MIRROR_RESYNC 2 mirrors 000909 1788792371.780460 write 0x0133fd thru 0x019ac9 (0x66cd bytes) 000910 1788792372.013589 mapwrite 0x030a9d thru 0x03b405 (0xa969 bytes) 000911 1788792372.161281 MIRROR_RESYNC 2 mirrors 000912 1788792372.836221 write 0x01bb9a thru 0x023347 (0x77ae bytes) 000913 1788792372.955795 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3bffa): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bffa): Mirror 2: 0x22d8c77f (stale) CRC-32 checksum value for chunk [0, 0x3bffa): Mirror 1: 0xc3de273f Mirror 3: 0xc3de273f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bffa: skipped 000915 1788792373.789334 mapread 0x022b0d thru 0x02e1c8 (0xb6bc bytes) 000916 1788792373.825544 write 0x015577 thru 0x01ae25 (0x58af bytes) 000917 1788792373.928274 read 0x011b57 thru 0x01c96a (0xae14 bytes) 000919 1788792373.939777 write 0x01dd61 thru 0x02ae5e (0xd0fe bytes) 000921 1788792373.957509 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3bffa): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bffa): Mirror 2: 0x22d8c77f (stale) Mirror 3: 0xc3de273f (stale) CRC-32 checksum value for chunk [0, 0x3bffa): Mirror 1: 0x4790f50f Mirror 4: 0x4790f50f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bffa: skipped 000922 1788792375.557086 write 0x007c94 thru 0x0103fa (0x8767 bytes) 000923 1788792375.786409 mapwrite 0x03c5bc thru 0x03ffff (0x3a44 bytes) 000927 1788792376.016128 mapread 0x01fda8 thru 0x028f4f (0x91a8 bytes) 000928 1788792376.053832 read 0x01aa5a thru 0x01f549 (0x4af0 bytes) 000929 1788792376.057127 mapwrite 0x0114b0 thru 0x017563 (0x60b4 bytes) 000931 1788792376.171096 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc393117 (stale) Mirror 3: 0xffa4354c (stale) Mirror 4: 0x38d9757d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2cf12194 Mirror 5: 0x2cf12194 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000932 1788792377.312710 trunc from 0x040000 to 0x034570 (0xba8f bytes) 000933 1788792377.408026 read 0x018819 thru 0x0231d7 (0xa9bf bytes) 000935 1788792377.425996 MIRROR_RESYNC 5 mirrors 000939 1788792379.150729 read 0x0266b8 thru 0x02c72f (0x6078 bytes) 000941 1788792379.204083 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16a:0x0]: Invalid argument (22) 000942 1788792379.600925 write 0x018afa thru 0x027de8 (0xf2ef bytes) 000943 1788792379.678710 trunc from 0x034571 to 0x03ee06 (0xa896 bytes) 000945 1788792379.704247 mapwrite 0x035e0d thru 0x03b4a4 (0x5698 bytes) 000947 1788792379.775702 write 0x021ee3 thru 0x026b1f (0x4c3d bytes) 000950 1788792379.780117 mapwrite 0x014e68 thru 0x019923 (0x4abc bytes) 000951 1788792379.837299 trunc from 0x03ee07 to 0x02d872 (0x11594 bytes) 000953 1788792379.855220 mapread 0x01bb7e thru 0x02b70c (0xfb8f bytes) 000955 1788792379.872987 read 0x00bb16 thru 0x010cca (0x51b5 bytes) 000956 1788792379.880002 write 0x023bb5 thru 0x02ee9f (0xb2eb bytes) 000958 1788792379.883193 read 0x0043ec thru 0x00ffb5 (0xbbca bytes) 000959 1788792379.892960 read 0x01c0fa thru 0x021844 (0x574b bytes) 000961 1788792379.894239 trunc from 0x02eea0 to 0x026540 (0x895f bytes) 000964 1788792379.914746 mapwrite 0x01bf2c thru 0x025da3 (0x9e78 bytes) 000965 1788792379.972019 mapread 0x01ee5f thru 0x026540 (0x76e2 bytes) 000967 1788792379.974690 mapwrite 0x0162d4 thru 0x01d25b (0x6f88 bytes) 000968 1788792380.014781 read 0x00507c thru 0x00d000 (0x7f85 bytes) 000971 1788792380.023646 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x26541): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26541): Mirror 2: 0x80399946 (stale) Mirror 3: 0x80399946 (stale) Mirror 4: 0x80399946 (stale) CRC-32 checksum value for chunk [0, 0x26541): Mirror 1: 0x6dfb799c Mirror 5: 0x6dfb799c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26541: skipped 000972 1788792381.178696 write 0x010a2e thru 0x019aad (0x9080 bytes) 000974 1788792381.280645 mapwrite 0x01174f thru 0x01c875 (0xb127 bytes) 000975 1788792381.365449 mapread 0x0024e5 thru 0x0110a0 (0xebbc bytes) 000978 1788792381.417196 read 0x026053 thru 0x026540 (0x4ee bytes) 000979 1788792381.437949 mapwrite 0x03c520 thru 0x03df94 (0x1a75 bytes) 000980 1788792381.525591 read 0x023a76 thru 0x0273ee (0x3979 bytes) 000987 1788792381.542444 read 0x0064f5 thru 0x015b49 (0xf655 bytes) 000992 1788792381.552895 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3df95): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3df95): Mirror 2: 0x3db8aeb3 (stale) Mirror 3: 0x3db8aeb3 (stale) Mirror 4: 0x3db8aeb3 (stale) Mirror 5: 0xbb0354c1 (stale) CRC-32 checksum value for chunk [0, 0x3df95): Mirror 1: 0x5a7c0277 Mirror 6: 0x5a7c0277 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3df95: skipped 000994 1788792383.023371 read 0x00829d thru 0x0167be (0xe522 bytes) 000995 1788792383.085878 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3df95): Mirror 2: 0x3db8aeb3 (stale) Mirror 3: 0x3db8aeb3 (stale) Mirror 4: 0x3db8aeb3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3df95): Mirror 2: 0x3db8aeb3 (stale) Mirror 3: 0x3db8aeb3 (stale) Mirror 4: 0x3db8aeb3 (stale) CRC-32 checksum value for chunk [0, 0x3df95): Mirror 1: 0x5a7c0277 Mirror 6: 0x5a7c0277 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3df95: skipped 000996 1788792384.282750 trunc from 0x03df95 to 0x00c88d (0x31707 bytes) 000997 1788792384.432425 mapwrite 0x014e07 thru 0x0189bd (0x3bb7 bytes) 000998 1788792384.536425 mapread 0x003643 thru 0x003d37 (0x6f5 bytes) 001000 1788792384.547076 read 0x00d355 thru 0x00f82d (0x24d9 bytes) All operations completed A-OK! PASS 70a (82s) == sanity-flr test 71: check mirror extend parent fid ==== 10:46:30 (1788792390) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 10:46:35 (1788792395) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x280:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.514881 s, 4.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x280:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173796 s, 6.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x281:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x280:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x281:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x280:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a6:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (14s) == sanity-flr test 200a: stress test ===================== 10:46:49 (1788792409) Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre2 Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5396160 .. Extending file size to 5978752 .. done Extending file size to 7326976 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6834784 .. Extending file size to 6984128 .. Extending file size to 7993120 .. Extending file size to 8718400 .. Extending file size to 7065248 .. Extending file size to 7751808 .. Extending file size to 8287264 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8833824 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9018240 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57249 57250 57253 57254 57255 done 192.168.201.117@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg117-client.virtnet /mnt/lustre2 (opts:) 192.168.201.117@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg117-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 10:48:13 (1788792493) Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre2 Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7634688 .. Extending file size to 3365152 .. Extending file size to 7062304 .. Extending file size to 8618560 .. Extending file size to 7675360 .. Extending file size to 7774592 .. Extending file size to 9150848 .. Extending file size to 8869600 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9161792 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58651 58652 58654 58657 58658 done 192.168.201.117@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg117-client.virtnet /mnt/lustre2 (opts:) 192.168.201.117@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg117-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (96s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 10:49:50 (1788792590) Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.178484 s, 11.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.398632 s, 5.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xfaf97566 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.117@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg117-client.virtnet /mnt/lustre2 (opts:) PASS 200c (25s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 10:50:17 (1788792617) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.165004 s, 12.7 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 10:50:27 (1788792627) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19f:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x297:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x297:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x298:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2bb:0x0] } PASS 203 (11s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 10:50:38 (1788792638) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.349694 s, 6.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x299:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (13s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 10:50:51 (1788792651) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.308179 s, 6.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bf:0x0] } PASS 204b (11s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 10:51:02 (1788792662) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.299378 s, 7.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } PASS 204c (38s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 10:51:41 (1788792701) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg117-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818377 s, 12.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29d:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29d:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29e:0x0] } PASS 204d (72s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 10:52:53 (1788792773) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg117-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.359883 s, 5.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (71s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 10:54:04 (1788792844) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg117-server mds-ost sync done. Creating new pool test_204f oleg117-server: Pool lustre.test_204f created Adding targets to pool oleg117-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg117-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.30599 s, 6.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg117-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg117-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg117-server: Pool lustre.test_204f destroyed PASS 204f (86s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 10:55:30 (1788792930) PASS 205a (12s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 10:55:43 (1788792943) PASS 205b (14s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 10:55:57 (1788792957) oleg117-server: Pool lustre.test_206 created Adding targets to pool oleg117-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg117-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg117-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg117-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg117-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 10:56:26 (1788792986) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12157 s, 8.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983721 s, 10.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x6ddd47aa Mirror 3: 0x6ddd47aa PASS 207 (14s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 10:56:41 (1788793001) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 10:56:45 (1788793005) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 10:56:49 (1788793009) Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.162846 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1790000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1480000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1480000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1480000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x20f0000 Read this (4 bytes): BBBB 192.168.201.117@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg117-client.virtnet /mnt/lustre2 (opts:) PASS 209a (44s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 10:57:33 (1788793053) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0211553 s, 387 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 10:57:42 (1788793062) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246571 s, 42.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1c4 lmm_fid: [0x200000404:0x1c4:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1c4 lmm_fid: [0x200000404:0x1c4:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834372 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 10:57:44.000000000 -0400 Modify: 2026-09-07 10:57:44.000000000 -0400 Change: 2026-09-07 10:57:45.000000000 -0400 Birth: 2026-09-07 10:57:44.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 10:57:51 (1788793071) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 10:57:56 (1788793076) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.482922 s, 84.8 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0510735 s, 80.2 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 7 10:57 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c6:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 7 10:57 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 10:58:05 (1788793085) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 10:58:14 (1788793094) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 2005 sec =========== 10:58:25 (1788793105) === sanity-flr: start cleanup 10:58:27 (1788793107) === === sanity-flr: finish cleanup 10:58:35 (1788793115) ===