-----============= acceptance-small: sanity-flr ============----- Thu Jun 11 21:31:42 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg633-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 21:31:50 (1781227910) === oleg633-client.virtnet: executing check_config_client /mnt/lustre oleg633-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg633-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9ead03fdc000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9ead03fdc000.idle_timeout=debug disable quota as required oleg633-server: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 21:32:04 (1781227924) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7568] == sanity-flr test 0a: lfs mirror create with -N option == 21:32:05 (1781227925) PASS 0a (6s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 21:32:12 (1781227932) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 21:32:13 (1781227933) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 21:32:14 (1781227934) PASS 0d (10s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 21:32:24 (1781227944) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 21:32:25 (1781227945) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 21:32:27 (1781227947) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9ead03fdc000.stats=clear osc.lustre-OST0001-osc-ffff9ead03fdc000.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 21:32:30 (1781227950) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--compress|-Z] [--compress-chunk=SIZE] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 21:32:34 (1781227954) PASS 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 21:32:38 (1781227958) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 21:32:40 (1781227960) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 21:32:43 (1781227963) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 21:32:47 (1781227967) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 21:32:50 (1781227970) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 21:32:52 (1781227972) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 21:32:55 (1781227975) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 21:33:00 (1781227980) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 21:33:02 (1781227982) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.147616 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592886 s, 17.7 MB/s PASS 21 (4s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 21:33:06 (1781227986) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.0912298 s, 69.0 MB/s PASS 22 (4s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 21:33:10 (1781227990) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 21:33:14 (1781227994) Stopping /mnt/lustre-ost1 (opts:) on oleg633-server oleg633-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ead03fdc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ead03fdc000.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg633-server: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg633-client: oleg633-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg633-server: oleg633-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg633-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg633-server oleg633-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ead03fdc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ead03fdc000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg633-server: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg633-client: oleg633-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg633-server: oleg633-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg633-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (28s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 21:33:42 (1781228022) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0819134 s, 51.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg633-server oleg633-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ead03fdc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ead03fdc000.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg633-server: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg633-client: oleg633-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg633-server: oleg633-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg633-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (16s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 21:33:58 (1781228038) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg633-server oleg633-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ead03fdc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ead03fdc000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg633-server: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg633-client: oleg633-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg633-server: oleg633-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg633-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg633-server oleg633-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ead03fdc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ead03fdc000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg633-server: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg633-client: oleg633-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg633-server: oleg633-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg633-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (26s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 21:34:24 (1781228064) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.71724 s, 30.5 MB/s -rw-r--r-- 1 root root 52428800 Jun 11 21:34 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.342s user 0m0.000s sys 0m0.156s Stopping /mnt/lustre-ost1 (opts:) on oleg633-server oleg633-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ead03fdc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ead03fdc000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.356s user 0m0.000s sys 0m0.161s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg633-server: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg633-client: oleg633-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg633-server: oleg633-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg633-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg633-server oleg633-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ead03fdc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ead03fdc000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost1)... real 0m0.381s user 0m0.001s sys 0m0.160s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg633-server: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg633-client: oleg633-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg633-server: oleg633-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg633-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg633-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ead03fdc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ead03fdc000.ost_server_uuid in FULL state after 0 sec PASS 33b (34s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 21:34:58 (1781228098) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 21:34:59 (1781228099) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 21:35:01 (1781228101) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 21:35:02 (1781228102) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 21:35:05 (1781228105) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.198497 s, 63.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (5s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 21:35:10 (1781228110) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178649 s, 58.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207667 s, 50.5 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 21:35:13 (1781228113) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 21:35:17 (1781228117) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000832297 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000732539 s, 1.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000787778 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000964908 s, 1.1 MB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 21:35:21 (1781228121) create test files with size 15751486 .. eb99dda146dc77bbea89e0e68ec0f7ef - 88abf259d7a9625dfb7288c10c882961 - 6f6727574a3d93f037c599071efcfbc0 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg633-client.virtnet /mnt/lustre (opts:) Starting client: oleg633-client.virtnet: -o user_xattr,flock 192.168.206.133@tcp:/lustre /mnt/lustre PASS 37 (9s) == sanity-flr test 38: resync ============================ 21:35:30 (1781228130) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (6s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 21:35:36 (1781228136) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 21:35:39 (1781228139) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360772 s, 29.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120581 s, 87.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 21:35:42 (1781228142) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0626772 s, 33.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.128953 s, 32.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0803697 s, 26.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (6s) == sanity-flr test 42: lfs mirror verify ================= 21:35:48 (1781228148) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 21:35:50 (1781228150) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 21:35:51 (1781228151) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116873 s, 89.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 43c: read prefer uncompressed mirror == 21:35:54 (1781228154) llite.lustre-ffff9ead06bd4000.enable_compression=1 fail_loc=0x1425 fail_val=-1 PASS 43c (2s) == sanity-flr test 44a: lfs mirror split check =========== 21:35:56 (1781228156) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 21:35:57 (1781228157) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe5:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xca:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xca:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 21:36:01 (1781228161) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.110041 s, 95.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x1a:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg633-client.virtnet /mnt/lustre (opts:) Starting client: oleg633-client.virtnet: -o user_xattr,flock 192.168.206.133@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (16s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 21:36:17 (1781228177) llite.lustre-ffff9ead05708800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.100615 s, 104 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 Stopping client oleg633-client.virtnet /mnt/lustre (opts:) Starting client: oleg633-client.virtnet: -o user_xattr,flock 192.168.206.133@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 llite.lustre-ffff9ead29b61000.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 21:36:21 (1781228181) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (14s) == sanity-flr test 46: Verify setstripe --copy option ==== 21:36:35 (1781228195) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (15s) == sanity-flr test 47: Verify mirror obj alloc =========== 21:36:50 (1781228210) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 21:36:51 (1781228211) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0888928 s, 35.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0663255 s, 47.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x4b31b0b3 (nosync,stale) Mirror 2: 0x4b31b0b3 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 9d43d898a96d06024749dcab9cbf176e - mirror 1 checksum: dff9a4c8b43cb5eb5413966b6c99ecf6 - mirror 2 checksum: 9d43d898a96d06024749dcab9cbf176e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jun 11 21:36:52 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jun 11 21:36:52 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 9d43d898a96d06024749dcab9cbf176e - mirror 1 checksum: dff9a4c8b43cb5eb5413966b6c99ecf6 - mirror 2 checksum: dff9a4c8b43cb5eb5413966b6c99ecf6 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jun 11 21:36:52 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jun 11 21:36:52 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } PASS 48 (4s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 21:36:56 (1781228216) Failing mds1 on oleg633-server Stopping /mnt/lustre-mds1 (opts:) on oleg633-server 21:36:58 (1781228218) shut down facet: mds1 facet_host: oleg633-server facet_failover_host: oleg633-server Failover mds1 to oleg633-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg633-server: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg633-client: oleg633-server: ssh exited with exit code 1 Started lustre-MDT0000 21:37:13 (1781228233) targets are mounted 21:37:13 (1781228233) facet_failover done oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137082 s, 76.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe1:0x0] } PASS 50A (24s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 21:37:20 (1781228240) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 21:37:21 (1781228241) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 21:37:22 (1781228242) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00278783 s, 5.9 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 21:37:24 (1781228244) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 21:37:25 (1781228245) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 21:37:27 (1781228247) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (6s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 21:37:33 (1781228253) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 21:37:53 (1781228273) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 21:38:11 (1781228291) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 21:38:29 (1781228309) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 21:38:32 (1781228312) mirror create pid 54037 mirror split pid 54038 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 54037 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 54038 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 21:39:35 (1781228375) Seed set to 8376 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x3f52e 000004 1781228376.279266 trunc from 00000000 to 0x03f52d (0x3f52e bytes) 000005 1781228376.289156 trunc from 0x03f52e to 0x025ffb (0x19532 bytes) 000006 1781228376.298224 mapread 0x017ba4 thru 0x01cb1f (0x4f7c bytes) 000007 1781228376.309882 read 0x003b5e thru 0x00959e (0x5a41 bytes) 000009 1781228376.317348 trunc from 0x025ffc to 0x0046c2 (0x21939 bytes) 000010 1781228376.326177 write 0x0208d1 thru 0x028a26 (0x8156 bytes) 000011 1781228376.327495 read 0x000d3f thru 0x00eeac (0xe16e bytes) 000012 1781228376.332080 mapread 0x007768 thru 0x00e372 (0x6c0b bytes) 000013 1781228376.335775 MIRROR_EXTEND to 2 mirrors 000014 1781228376.602533 write 0x02fc42 thru 0x03d075 (0xd434 bytes) 000016 1781228376.672820 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d076): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d076): Mirror 2: 0x783c5f33 (stale) CRC-32 checksum value for chunk [0, 0x3d076): Mirror 1: 0xc896eaa9 Mirror 3: 0xc896eaa9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d076: skipped 000018 1781228377.073421 write 0x034cfc thru 0x039d11 (0x5016 bytes) 000019 1781228377.113672 trunc from 0x03d076 to 0x00fd66 (0x2d30f bytes) 000024 1781228377.139482 read 0x008766 thru 0x00e870 (0x610b bytes) 000025 1781228377.145420 mapread 0x0067f8 thru 0x00fd66 (0x956f bytes) 000027 1781228377.151209 mapwrite 0x0329f8 thru 0x034647 (0x1c50 bytes) 000032 1781228377.195014 trunc from 0x034648 to 0x017e7b (0x1c7cc bytes) 000034 1781228377.205488 mapwrite 0x03c02d thru 0x03ffff (0x3fd3 bytes) 000036 1781228377.245549 mapwrite 0x00f84a thru 0x01e22f (0xe9e6 bytes) 000037 1781228377.284351 mapwrite 0x02b7bb thru 0x02bcca (0x510 bytes) 000038 1781228377.317536 read 0x000545 thru 0x00e897 (0xe353 bytes) 000039 1781228377.327163 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xdabbad14 (stale) Mirror 3: 0x32ab03f7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdd09f314 Mirror 4: 0xdd09f314 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000041 1781228377.743584 trunc from 0x040000 to 0x000510 (0x3faef bytes) 000042 1781228377.792984 trunc from 0x000511 to 0x029af3 (0x295e3 bytes) 000044 1781228377.803371 mapwrite 0x0225ec thru 0x02605e (0x3a73 bytes) 000045 1781228377.838910 mapread 0x0177e7 thru 0x017a11 (0x22b bytes) 000046 1781228377.844994 trunc from 0x029af4 to 0x02713c (0x29b7 bytes) 000047 1781228377.855249 mapread 0x00dcce thru 0x01a811 (0xcb44 bytes) 000048 1781228377.861425 trunc from 0x02713d to 0x010da3 (0x16399 bytes) 000050 1781228377.876441 read 0x0100ba thru 0x010da3 (0xcea bytes) 000051 1781228377.880964 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x385:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10da4): Mirror 3: 0x5e6c5330 (stale) Mirror 4: 0x990cd9ac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10da4): Mirror 3: 0x5e6c5330 (stale) Mirror 4: 0x990cd9ac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10da4: skipped 000052 1781228378.038206 trunc from 0x010da4 to 0x01e9b4 (0xdc11 bytes) 000053 1781228378.046593 mapread 0x00fcd0 thru 0x01511e (0x544f bytes) 000055 1781228378.060412 MIRROR_RESYNC 3 mirrors 000056 1781228378.350643 trunc from 0x01e9b5 to 0x0061fa (0x187ba bytes) 000059 1781228378.392651 read 0x004b38 thru 0x0061fa (0x16c3 bytes) 000060 1781228378.396129 trunc from 0x0061fb to 0x023da7 (0x1dbad bytes) 000061 1781228378.409005 read 0x003cbb thru 0x010623 (0xc969 bytes) 000063 1781228378.415802 write 0x033055 thru 0x036c15 (0x3bc1 bytes) 000065 1781228378.417524 mapwrite 0x02f204 thru 0x032c40 (0x3a3d bytes) 000066 1781228378.451990 mapwrite 0x03cc1d thru 0x03d0f7 (0x4db bytes) 000067 1781228378.522016 mapread 0x0010e2 thru 0x00ec69 (0xdb88 bytes) 000070 1781228378.524360 write 0x02f852 thru 0x0384e7 (0x8c96 bytes) 000072 1781228378.529256 mapwrite 0x005507 thru 0x011555 (0xc04f bytes) 000073 1781228378.564889 write 0x0062f3 thru 0x014959 (0xe667 bytes) 000075 1781228378.569599 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x385:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d0f8): Mirror 4: 0xd59fcf0f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d0f8): Mirror 4: 0xd59fcf0f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d0f8: skipped 000076 1781228378.762365 mapread 0x03784b thru 0x03d0f7 (0x58ad bytes) 000078 1781228378.782826 mapwrite 0x00620b thru 0x007b53 (0x1949 bytes) 000080 1781228378.828814 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x385:0x0]: Invalid argument (22) 000081 1781228378.909155 write 0x03b242 thru 0x03ffff (0x4dbe bytes) 000082 1781228378.914097 write 0x016f1c thru 0x01b307 (0x43ec bytes) 000083 1781228378.922462 write 0x03d983 thru 0x03ffff (0x267d bytes) 000084 1781228378.923412 mapread 0x013bba thru 0x01c1cd (0x8614 bytes) 000086 1781228378.939646 mapwrite 0x0031c5 thru 0x00b18c (0x7fc8 bytes) 000087 1781228378.980517 trunc from 0x040000 to 0x03a706 (0x58f9 bytes) 000088 1781228378.987812 trunc from 0x03a707 to 0x004de9 (0x3591d bytes) 000089 1781228378.997450 mapwrite 0x034bb7 thru 0x03ffff (0xb449 bytes) 000091 1781228379.035041 read 0x02b23b thru 0x039291 (0xe057 bytes) 000092 1781228379.038397 trunc from 0x040000 to 0x012bb3 (0x2d44c bytes) 000093 1781228379.044868 trunc from 0x012bb4 to 0x024553 (0x119a0 bytes) 000096 1781228379.052266 mapread 0x005f0b thru 0x006c1d (0xd13 bytes) 000100 1781228379.059612 write 0x0124a0 thru 0x018489 (0x5fea bytes) 000104 1781228379.067620 write 0x0017f1 thru 0x00db42 (0xc352 bytes) 000106 1781228379.070168 mapread 0x01ef8a thru 0x01fd70 (0xde7 bytes) 000107 1781228379.076675 mapwrite 0x02e235 thru 0x03baa0 (0xd86c bytes) 000108 1781228379.126005 trunc from 0x03baa1 to 0x01c689 (0x1f417 bytes) 000110 1781228379.139240 trunc from 0x01c68a to 0x00da78 (0xec11 bytes) 000112 1781228379.148530 mapwrite 0x006304 thru 0x014930 (0xe62d bytes) 000115 1781228379.189098 read 0x013840 thru 0x014930 (0x10f1 bytes) 000116 1781228379.192294 MIRROR_EXTEND to 2 mirrors 000117 1781228379.499239 trunc from 0x014931 to 0x02466f (0xfd3f bytes) 000118 1781228379.561878 mapread 0x017760 thru 0x0213b3 (0x9c54 bytes) 000119 1781228379.574592 trunc from 0x024670 to 0x039620 (0x14fb1 bytes) 000121 1781228379.590510 mapwrite 0x006d1a thru 0x00a88e (0x3b75 bytes) 000122 1781228379.631136 MIRROR_RESYNC 2 mirrors 000123 1781228379.909077 write 0x03d513 thru 0x03ffff (0x2aed bytes) 000124 1781228379.941148 trunc from 0x040000 to 0x009ae4 (0x3651b bytes) 000126 1781228379.962032 mapwrite 0x0235ee thru 0x0295ca (0x5fdd bytes) 000127 1781228380.012205 mapread 0x00b0c7 thru 0x012981 (0x78bb bytes) 000128 1781228380.019212 write 0x03c87f thru 0x03ffff (0x3781 bytes) 000129 1781228380.020353 read 0x02d421 thru 0x031062 (0x3c42 bytes) 000134 1781228380.030026 trunc from 0x040000 to 0x01dba3 (0x2245c bytes) 000135 1781228380.039296 read 0x013acd thru 0x01dba3 (0xa0d7 bytes) 000136 1781228380.047328 trunc from 0x01dba4 to 0x024dae (0x720b bytes) 000140 1781228380.061414 read 0x010c7a thru 0x015a17 (0x4d9e bytes) 000141 1781228380.062325 mapread 0x017d73 thru 0x019833 (0x1ac1 bytes) 000142 1781228380.063341 write 0x02ddb8 thru 0x031cbd (0x3f06 bytes) 000143 1781228380.064770 mapread 0x00cdc5 thru 0x013dca (0x7006 bytes) 000144 1781228380.066043 write 0x002770 thru 0x006a04 (0x4295 bytes) 000145 1781228380.066801 trunc from 0x031cbe to 0x03a672 (0x89b5 bytes) 000146 1781228380.077722 write 0x03126b thru 0x0393ef (0x8185 bytes) 000147 1781228380.084239 trunc from 0x03a673 to 0x01bfe3 (0x1e68f bytes) 000148 1781228380.095063 mapwrite 0x01308c thru 0x020c09 (0xdb7e bytes) 000149 1781228380.138664 mapread 0x002a16 thru 0x00f41a (0xca05 bytes) 000150 1781228380.139864 read 0x00d2e2 thru 0x018c5f (0xb97e bytes) 000153 1781228380.144023 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x385:0x0]: Invalid argument (22) 000156 1781228380.210067 write 0x01a606 thru 0x029365 (0xed60 bytes) 000157 1781228380.215387 write 0x0080d1 thru 0x00a970 (0x28a0 bytes) 000158 1781228380.221216 mapwrite 0x0085d8 thru 0x01690e (0xe337 bytes) 000159 1781228380.247578 write 0x00fdf2 thru 0x01e828 (0xea37 bytes) 000162 1781228380.250329 mapread 0x0080aa thru 0x011d1a (0x9c71 bytes) 000163 1781228380.251446 mapread 0x016e71 thru 0x024dd5 (0xdf65 bytes) 000164 1781228380.252829 write 0x001843 thru 0x00bec8 (0xa686 bytes) 000165 1781228380.257218 mapread 0x0195ce thru 0x02515d (0xbb90 bytes) 000166 1781228380.258649 read 0x0087af thru 0x00b4be (0x2d10 bytes) 000167 1781228380.259348 write 0x02c82a thru 0x02ec97 (0x246e bytes) 000169 1781228380.261360 trunc from 0x02ec98 to 0x036e6c (0x81d5 bytes) 000171 1781228380.270924 mapread 0x032308 thru 0x036e6c (0x4b65 bytes) 000173 1781228380.278354 mapwrite 0x016196 thru 0x016c2b (0xa96 bytes) 000174 1781228380.308908 mapwrite 0x001379 thru 0x010da3 (0xfa2b bytes) 000178 1781228380.348058 mapread 0x029a19 thru 0x02f0c5 (0x56ad bytes) 000182 1781228380.357848 MIRROR_EXTEND to 2 mirrors 000185 1781228380.625018 mapwrite 0x004a1f thru 0x013399 (0xe97b bytes) 000187 1781228380.720539 write 0x021c85 thru 0x0243fd (0x2779 bytes) 000189 1781228380.728330 mapwrite 0x016b0c thru 0x018454 (0x1949 bytes) 000190 1781228380.758022 read 0x00d713 thru 0x018e52 (0xb740 bytes) 000191 1781228380.758786 trunc from 0x036e6d to 0x03ac5d (0x3df1 bytes) 000192 1781228380.779553 read 0x00d71c thru 0x01608d (0x8972 bytes) 000193 1781228380.780728 mapread 0x00fead thru 0x010cba (0xe0e bytes) 000194 1781228380.781723 mapwrite 0x00fad0 thru 0x01b6c5 (0xbbf6 bytes) 000195 1781228380.807807 mapread 0x00c8b1 thru 0x01a8f5 (0xe045 bytes) 000196 1781228380.809091 mapread 0x0161aa thru 0x021dfd (0xbc54 bytes) 000198 1781228380.811072 trunc from 0x03ac5e to 0x02b4cc (0xf791 bytes) 000199 1781228380.822563 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x385:0x0]: Invalid argument (22) 000202 1781228380.894081 mapwrite 0x0342a7 thru 0x03ffff (0xbd59 bytes) 000203 1781228380.955601 mapwrite 0x01b668 thru 0x028253 (0xcbec bytes) 000204 1781228381.000195 mapread 0x02b463 thru 0x03806b (0xcc09 bytes) 000205 1781228381.004509 trunc from 0x040000 to 0x028bda (0x17425 bytes) 000206 1781228381.011046 read 0x0069c4 thru 0x01584c (0xee89 bytes) 000207 1781228381.017215 mapwrite 0x0187dd thru 0x01b23e (0x2a62 bytes) 000208 1781228381.035557 mapread 0x00994b thru 0x00dca9 (0x435f bytes) 000210 1781228381.037063 mapread 0x014713 thru 0x022f28 (0xe816 bytes) 000211 1781228381.038128 mapwrite 0x008b6d thru 0x00d86d (0x4d01 bytes) 000214 1781228381.060821 mapwrite 0x00ff6f thru 0x01e3cc (0xe45e bytes) 000215 1781228381.087339 read 0x012567 thru 0x013e9b (0x1935 bytes) 000216 1781228381.087965 write 0x037f3d thru 0x03ffff (0x80c3 bytes) 000220 1781228381.090550 read 0x015784 thru 0x0237e1 (0xe05e bytes) 000221 1781228381.091082 mapwrite 0x0196c0 thru 0x019d9d (0x6de bytes) 000223 1781228381.118195 mapread 0x009266 thru 0x017afc (0xe897 bytes) 000225 1781228381.119857 mapread 0x0276fe thru 0x02ed35 (0x7638 bytes) 000226 1781228381.128232 MIRROR_EXTEND to 2 mirrors 000229 1781228381.365547 mapread 0x000f50 thru 0x00d987 (0xca38 bytes) 000231 1781228381.379165 write 0x00c912 thru 0x0155bd (0x8cac bytes) 000233 1781228381.415622 write 0x024515 thru 0x031726 (0xd212 bytes) 000234 1781228381.427285 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x385:0x0]: Invalid argument (22) 000239 1781228381.507416 trunc from 0x040000 to 0x001ca1 (0x3e35e bytes) 000242 1781228381.530502 read 0x001147 thru 0x001ca1 (0xb5b bytes) 000244 1781228381.536279 trunc from 0x001ca2 to 0x024172 (0x224d1 bytes) 000246 1781228381.550005 read 0x02265b thru 0x022adc (0x482 bytes) 000247 1781228381.558594 mapread 0x015e35 thru 0x018fd3 (0x319f bytes) 000250 1781228381.561201 read 0x01254f thru 0x014509 (0x1fbb bytes) 000251 1781228381.561949 write 0x010484 thru 0x018f3d (0x8aba bytes) 000252 1781228381.562959 mapwrite 0x010f24 thru 0x013776 (0x2853 bytes) 000253 1781228381.586317 write 0x03dd2a thru 0x03ffff (0x22d6 bytes) 000254 1781228381.587413 read 0x03878a thru 0x03ffff (0x7876 bytes) 000255 1781228381.591917 trunc from 0x040000 to 0x03f41a (0xbe5 bytes) 000257 1781228381.600528 trunc from 0x03f41b to 0x003de9 (0x3b631 bytes) 000259 1781228381.607871 write 0x00db41 thru 0x0125f5 (0x4ab5 bytes) 000262 1781228381.610238 mapread 0x00848c thru 0x0125f5 (0xa16a bytes) 000264 1781228381.625308 mapwrite 0x03aa42 thru 0x03ffff (0x55be bytes) 000265 1781228381.677683 read 0x00dba1 thru 0x012d7d (0x51dd bytes) 000267 1781228381.678974 read 0x02dafd thru 0x03c21a (0xe71e bytes) 000271 1781228381.687029 read 0x0105f8 thru 0x01cd66 (0xc76f bytes) 000275 1781228381.689811 write 0x02830b thru 0x0366a4 (0xe39a bytes) 000282 1781228381.694353 trunc from 0x040000 to 0x03e491 (0x1b6e bytes) 000284 1781228381.703949 read 0x0360f5 thru 0x03e491 (0x839d bytes) 000285 1781228381.704504 read 0x027cf2 thru 0x03120b (0x951a bytes) 000289 1781228381.709533 MIRROR_EXTEND to 2 mirrors 000290 1781228381.960583 mapwrite 0x01dd0a thru 0x02b0f4 (0xd3eb bytes) 000291 1781228382.055454 MIRROR_RESYNC 2 mirrors 000293 1781228382.373586 mapread 0x02f7da thru 0x03bc82 (0xc4a9 bytes) 000295 1781228382.387865 read 0x004e11 thru 0x014364 (0xf554 bytes) 000296 1781228382.395108 MIRROR_RESYNC 2 mirrors 000297 1781228382.570801 trunc from 0x03e492 to 0x00eae2 (0x2f9af bytes) 000298 1781228382.606337 mapwrite 0x032c81 thru 0x03f785 (0xcb05 bytes) 000299 1781228382.657461 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f786): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f786): Mirror 2: 0x4b3da75d (stale) CRC-32 checksum value for chunk [0, 0x3f786): Mirror 1: 0xeb5da883 Mirror 3: 0xeb5da883 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f786: skipped 000300 1781228382.989570 write 0x0231b9 thru 0x0298cb (0x6713 bytes) 000303 1781228383.026794 write 0x014b0d thru 0x019ade (0x4fd2 bytes) 000304 1781228383.036568 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f786): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f786): Mirror 2: 0x4b3da75d (stale) Mirror 3: 0xeb5da883 (stale) CRC-32 checksum value for chunk [0, 0x3f786): Mirror 1: 0x79ceceb4 Mirror 4: 0x79ceceb4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f786: skipped 000305 1781228383.464798 mapwrite 0x00c562 thru 0x017c96 (0xb735 bytes) 000307 1781228383.575993 write 0x00f332 thru 0x0137bf (0x448e bytes) 000311 1781228383.578451 mapwrite 0x03f179 thru 0x03ffff (0xe87 bytes) 000312 1781228383.632583 write 0x01ae97 thru 0x0234f6 (0x8660 bytes) 000314 1781228383.637408 mapread 0x00a736 thru 0x00a8a3 (0x16e bytes) 000316 1781228383.639253 mapread 0x02e5bb thru 0x02ea46 (0x48c bytes) 000317 1781228383.649050 mapread 0x02ce46 thru 0x0311de (0x4399 bytes) 000319 1781228383.650509 trunc from 0x040000 to 0x015acf (0x2a530 bytes) 000320 1781228383.666534 write 0x014088 thru 0x01afcd (0x6f46 bytes) 000322 1781228383.671333 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1afce): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1afce): Mirror 2: 0xd8784f20 (stale) Mirror 3: 0x3c5a5062 (stale) Mirror 4: 0x51653216 (stale) CRC-32 checksum value for chunk [0, 0x1afce): Mirror 1: 0xd6087e95 Mirror 5: 0xd6087e95 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1afce: skipped 000323 1781228384.183780 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x385:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1afce): Mirror 3: 0x3c5a5062 (stale) Mirror 4: 0x51653216 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1afce): Mirror 3: 0x3c5a5062 (stale) Mirror 4: 0x51653216 (stale) CRC-32 checksum value for chunk [0, 0x1afce): Mirror 1: 0xd6087e95 Mirror 5: 0xd6087e95 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1afce: skipped 000324 1781228384.427205 write 0x0144e9 thru 0x01dbc2 (0x96da bytes) 000325 1781228384.460115 mapread 0x00d2c1 thru 0x01b78d (0xe4cd bytes) 000326 1781228384.468699 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x385:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1dbc3): Mirror 4: 0xd175c016 (stale) Mirror 5: 0x55649f18 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dbc3): Mirror 4: 0xd175c016 (stale) Mirror 5: 0x55649f18 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dbc3: skipped 000327 1781228384.639696 mapread 0x010664 thru 0x012b55 (0x24f2 bytes) 000328 1781228384.651431 trunc from 0x01dbc3 to 0x03c820 (0x1ec5e bytes) 000329 1781228384.658539 mapwrite 0x019624 thru 0x01d7a0 (0x417d bytes) 000331 1781228384.690693 write 0x02b474 thru 0x02e795 (0x3322 bytes) 000332 1781228384.699777 write 0x037025 thru 0x03ffff (0x8fdb bytes) 000333 1781228384.705610 read 0x03b0c0 thru 0x03ffff (0x4f40 bytes) 000335 1781228384.706718 write 0x02ccf4 thru 0x02ed80 (0x208d bytes) 000336 1781228384.707515 read 0x005c46 thru 0x00e21e (0x85d9 bytes) 000341 1781228384.715554 mapread 0x007deb thru 0x0121c9 (0xa3df bytes) 000343 1781228384.717343 read 0x0289a1 thru 0x033d49 (0xb3a9 bytes) 000344 1781228384.718041 trunc from 0x040000 to 0x01c4ec (0x23b13 bytes) 000346 1781228384.727658 mapread 0x0067cc thru 0x00fae0 (0x9315 bytes) 000348 1781228384.729305 write 0x03790b thru 0x03ffff (0x86f5 bytes) 000349 1781228384.730450 mapread 0x0196a1 thru 0x025c86 (0xc5e6 bytes) 000353 1781228384.738614 trunc from 0x040000 to 0x01de83 (0x2217c bytes) 000357 1781228384.751834 write 0x00c639 thru 0x00e0a9 (0x1a71 bytes) 000358 1781228384.753027 mapwrite 0x027488 thru 0x030ae1 (0x965a bytes) 000359 1781228384.811222 read 0x018606 thru 0x01eec1 (0x68bc bytes) 000360 1781228384.818052 MIRROR_RESYNC 3 mirrors 000362 1781228385.148227 read 0x028c47 thru 0x030ae1 (0x7e9b bytes) 000364 1781228385.158566 write 0x007010 thru 0x00ec28 (0x7c19 bytes) 000365 1781228385.216263 trunc from 0x030ae2 to 0x0181c4 (0x1891d bytes) 000366 1781228385.227081 write 0x01e0ee thru 0x0298f6 (0xb809 bytes) 000367 1781228385.228424 mapwrite 0x013e83 thru 0x01968e (0x580c bytes) 000368 1781228385.259176 mapwrite 0x026ea8 thru 0x0359d1 (0xeb2a bytes) 000369 1781228385.309109 read 0x01c54a thru 0x0223ee (0x5ea5 bytes) 000370 1781228385.315366 mapwrite 0x01160e thru 0x014710 (0x3103 bytes) 000371 1781228385.340019 write 0x026d9b thru 0x02b54f (0x47b5 bytes) 000372 1781228385.341081 mapwrite 0x0089bd thru 0x017612 (0xec56 bytes) 000374 1781228385.378718 mapwrite 0x0154b4 thru 0x019b0c (0x4659 bytes) 000376 1781228385.411088 write 0x032dc3 thru 0x0357bb (0x29f9 bytes) 000378 1781228385.416722 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x385:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x359d2): Mirror 5: 0xd4ea5530 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x359d2): Mirror 5: 0xd4ea5530 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x359d2: skipped 000381 1781228385.597172 read 0x01a246 thru 0x01b148 (0xf03 bytes) 000384 1781228385.603138 write 0x037650 thru 0x03ffff (0x89b0 bytes) 000389 1781228385.610783 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x385:0x0]: Invalid argument (22) 000398 1781228385.704974 mapread 0x01794e thru 0x01fc5a (0x830d bytes) 000402 1781228385.724434 mapwrite 0x03bcc9 thru 0x03ffff (0x4337 bytes) 000404 1781228385.757933 mapwrite 0x02c277 thru 0x03761d (0xb3a7 bytes) 000406 1781228385.792438 mapwrite 0x0283a6 thru 0x02ae43 (0x2a9e bytes) 000407 1781228385.816960 mapread 0x01effe thru 0x02a0db (0xb0de bytes) 000411 1781228385.822779 mapread 0x029738 thru 0x034c13 (0xb4dc bytes) 000414 1781228385.825117 mapwrite 0x012c48 thru 0x0199e4 (0x6d9d bytes) 000415 1781228385.854276 mapread 0x03eb28 thru 0x03ffff (0x14d8 bytes) 000416 1781228385.855272 write 0x014bc6 thru 0x01f72e (0xab69 bytes) 000417 1781228385.859916 MIRROR_EXTEND to 2 mirrors 000418 1781228386.181493 read 0x02f44a thru 0x03a544 (0xb0fb bytes) 000419 1781228386.188250 read 0x035853 thru 0x03c723 (0x6ed1 bytes) 000422 1781228386.196047 mapread 0x00c1e9 thru 0x013317 (0x712f bytes) 000423 1781228386.196923 read 0x02e588 thru 0x03cef3 (0xe96c bytes) 000424 1781228386.197310 trunc from 0x040000 to 0x010bda (0x2f425 bytes) 000425 1781228386.243154 trunc from 0x010bdb to 0x01a7a0 (0x9bc6 bytes) 000427 1781228386.254687 write 0x0014e9 thru 0x003852 (0x236a bytes) 000428 1781228386.260946 mapread 0x015ec1 thru 0x01a7a0 (0x48e0 bytes) 000436 1781228386.265671 read 0x010567 thru 0x011b3d (0x15d7 bytes) 000439 1781228386.267306 read 0x01745a thru 0x01a7a0 (0x3347 bytes) 000440 1781228386.267930 mapwrite 0x03d079 thru 0x03ffff (0x2f87 bytes) 000441 1781228386.308542 trunc from 0x040000 to 0x03b924 (0x46db bytes) 000443 1781228386.322199 read 0x039c69 thru 0x03b924 (0x1cbc bytes) 000444 1781228386.325383 MIRROR_RESYNC 2 mirrors 000445 1781228386.589220 write 0x02f4d6 thru 0x03d46b (0xdf96 bytes) 000446 1781228386.624929 trunc from 0x03d46c to 0x03a96c (0x2aff bytes) 000447 1781228386.648442 trunc from 0x03a96d to 0x02b230 (0xf73c bytes) 000448 1781228386.655438 read 0x009240 thru 0x011cfa (0x8abb bytes) 000451 1781228386.661524 read 0x00d5e9 thru 0x012d44 (0x575c bytes) 000452 1781228386.666887 trunc from 0x02b231 to 0x01c12c (0xf104 bytes) 000454 1781228386.675185 mapwrite 0x00a89c thru 0x00ba4b (0x11b0 bytes) 000455 1781228386.700646 mapread 0x00341f thru 0x011255 (0xde37 bytes) 000456 1781228386.701947 trunc from 0x01c12d to 0x02dfb9 (0x11e8d bytes) 000457 1781228386.713236 mapwrite 0x03e8a9 thru 0x03ffff (0x1757 bytes) 000458 1781228386.750433 trunc from 0x040000 to 0x034fe0 (0xb01f bytes) 000460 1781228386.757504 read 0x006fe5 thru 0x01579b (0xe7b7 bytes) 000461 1781228386.760684 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x385:0x0]: Invalid argument (22) 000462 1781228386.825025 write 0x013cb6 thru 0x01accf (0x701a bytes) 000466 1781228386.832097 write 0x028137 thru 0x034b22 (0xc9ec bytes) 000467 1781228386.842031 mapwrite 0x01ac09 thru 0x02550e (0xa906 bytes) 000472 1781228386.870429 trunc from 0x034fe1 to 0x0058ad (0x2f733 bytes) 000477 1781228386.879938 read 0x002eca thru 0x0058ad (0x29e4 bytes) 000480 1781228386.884634 trunc from 0x0058ae to 0x0160af (0x10802 bytes) 000482 1781228386.894996 mapwrite 0x012446 thru 0x02103c (0xebf7 bytes) 000484 1781228386.931536 write 0x02c273 thru 0x032eeb (0x6c79 bytes) 000485 1781228386.932944 write 0x00b2f5 thru 0x00c731 (0x143d bytes) 000488 1781228386.935214 trunc from 0x032eec to 0x00f950 (0x2359b bytes) 000492 1781228386.948381 MIRROR_EXTEND to 2 mirrors 000493 1781228387.240527 MIRROR_RESYNC 2 mirrors 000495 1781228387.381439 trunc from 0x00f951 to 0x034eb4 (0x25564 bytes) 000497 1781228387.410670 read 0x02426a thru 0x027f5e (0x3cf5 bytes) 000499 1781228387.415285 read 0x02322a thru 0x032a96 (0xf86d bytes) 000500 1781228387.420371 read 0x01a14a thru 0x01ffc5 (0x5e7c bytes) 000501 1781228387.421107 read 0x01bce1 thru 0x026f7a (0xb29a bytes) 000505 1781228387.423503 mapwrite 0x02a73f thru 0x035f5f (0xb821 bytes) 000506 1781228387.453351 trunc from 0x035f60 to 0x00b6f8 (0x2a867 bytes) 000507 1781228387.461564 read 0x00948a thru 0x00b6f8 (0x226f bytes) 000508 1781228387.464192 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xb6f9): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb6f9): Mirror 2: 0x623ae160 (stale) CRC-32 checksum value for chunk [0, 0xb6f9): Mirror 1: 0x623ae160 Mirror 3: 0x623ae160 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb6f9: skipped 000510 1781228387.755885 read 0x00985a thru 0x00b6f8 (0x1e9f bytes) 000511 1781228387.760413 mapwrite 0x037fac thru 0x03dcf2 (0x5d47 bytes) 000512 1781228387.836390 write 0x03f7af thru 0x03ffff (0x851 bytes) 000513 1781228387.837330 write 0x0009a5 thru 0x0096f6 (0x8d52 bytes) 000514 1781228387.846726 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x385:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x17175c8b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x17175c8b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000515 1781228387.984511 write 0x0215b4 thru 0x02bfcf (0xaa1c bytes) 000516 1781228387.991101 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x17175c8b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x18d48c1a Mirror 3: 0x18d48c1a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000518 1781228388.377466 mapread 0x012b46 thru 0x022135 (0xf5f0 bytes) 000519 1781228388.394405 mapread 0x0123c4 thru 0x01df9d (0xbbda bytes) 000520 1781228388.395241 write 0x03e3ff thru 0x03ffff (0x1c01 bytes) 000521 1781228388.433410 read 0x018282 thru 0x01c60e (0x438d bytes) 000522 1781228388.439510 mapwrite 0x01cda6 thru 0x025ff4 (0x924f bytes) 000523 1781228388.480169 read 0x0133a6 thru 0x013da8 (0xa03 bytes) 000524 1781228388.489313 mapwrite 0x03e532 thru 0x03ffff (0x1ace bytes) 000527 1781228388.520410 MIRROR_RESYNC 3 mirrors 000530 1781228388.846912 write 0x02a3d9 thru 0x0364fd (0xc125 bytes) 000531 1781228388.892537 read 0x0252a9 thru 0x028e3f (0x3b97 bytes) 000532 1781228388.897446 write 0x02ad38 thru 0x03a6e1 (0xf9aa bytes) 000533 1781228388.904468 mapread 0x03c3a3 thru 0x03ffff (0x3c5d bytes) 000535 1781228388.911296 mapread 0x0152ff thru 0x0164b7 (0x11b9 bytes) 000537 1781228388.921254 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x385:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x119826b1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x119826b1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000538 1781228389.083412 trunc from 0x040000 to 0x001e78 (0x3e187 bytes) 000539 1781228389.101298 mapwrite 0x00722e thru 0x007f2f (0xd02 bytes) 000540 1781228389.130931 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x7f30): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7f30): Mirror 3: 0xa156ef92 (stale) CRC-32 checksum value for chunk [0, 0x7f30): Mirror 1: 0xbf28d67c Mirror 4: 0xbf28d67c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7f30: skipped 000542 1781228389.492668 read 0x005a3d thru 0x007f2f (0x24f3 bytes) 000543 1781228389.499137 mapread 0x006fe7 thru 0x007f2f (0xf49 bytes) 000546 1781228389.500520 write 0x034042 thru 0x03ea4d (0xaa0c bytes) 000547 1781228389.541356 MIRROR_RESYNC 3 mirrors 000548 1781228389.873278 mapwrite 0x0247e9 thru 0x02a7f7 (0x600f bytes) 000549 1781228389.952432 write 0x02cfba thru 0x02d2ec (0x333 bytes) 000552 1781228389.954264 read 0x02fc8a thru 0x03ae37 (0xb1ae bytes) 000553 1781228389.967632 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x385:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ea4e): Mirror 4: 0xdfc3f63 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ea4e): Mirror 4: 0xdfc3f63 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ea4e: skipped 000554 1781228390.108450 mapwrite 0x039763 thru 0x03ffff (0x689d bytes) 000556 1781228390.153766 trunc from 0x040000 to 0x01d4d8 (0x22b27 bytes) 000558 1781228390.167620 write 0x02a2b9 thru 0x034161 (0x9ea9 bytes) 000559 1781228390.169400 mapread 0x0068b9 thru 0x01113f (0xa887 bytes) 000560 1781228390.181295 mapwrite 0x00faae thru 0x015066 (0x55b9 bytes) 000562 1781228390.210140 trunc from 0x034162 to 0x0063ff (0x2dd62 bytes) 000563 1781228390.220461 mapwrite 0x0065c4 thru 0x011176 (0xabb3 bytes) 000564 1781228390.266395 write 0x004629 thru 0x013280 (0xec58 bytes) 000565 1781228390.270260 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x385:0x0]: Invalid argument (22) 000566 1781228390.341369 read 0x012513 thru 0x013280 (0xd6e bytes) 000569 1781228390.345445 write 0x009902 thru 0x0169b3 (0xd0b2 bytes) 000571 1781228390.350368 write 0x00ac11 thru 0x012e92 (0x8282 bytes) 000577 1781228390.353814 mapread 0x00c50d thru 0x0169b3 (0xa4a7 bytes) 000578 1781228390.354730 mapread 0x012d4b thru 0x0169b3 (0x3c69 bytes) 000579 1781228390.355386 mapwrite 0x005508 thru 0x0065a6 (0x109f bytes) 000586 1781228390.387252 mapread 0x00e631 thru 0x0141ad (0x5b7d bytes) 000587 1781228390.388295 read 0x00ec5f thru 0x0167c0 (0x7b62 bytes) 000592 1781228390.391050 mapwrite 0x02d4e2 thru 0x035083 (0x7ba2 bytes) 000594 1781228390.443457 mapread 0x013a10 thru 0x01f4f8 (0xbae9 bytes) 000595 1781228390.444825 write 0x03ea45 thru 0x03ffff (0x15bb bytes) 000596 1781228390.445775 read 0x0185d8 thru 0x02842b (0xfe54 bytes) 000599 1781228390.447292 write 0x02fcc2 thru 0x03aa90 (0xadcf bytes) 000601 1781228390.451962 trunc from 0x040000 to 0x0359bd (0xa642 bytes) 000602 1781228390.459707 read 0x02e44f thru 0x034029 (0x5bdb bytes) 000603 1781228390.462872 MIRROR_EXTEND to 2 mirrors 000604 1781228390.697166 mapwrite 0x027e2e thru 0x02ee89 (0x705c bytes) 000605 1781228390.784192 MIRROR_RESYNC 2 mirrors 000609 1781228391.052741 write 0x016058 thru 0x02443d (0xe3e6 bytes) 000613 1781228391.107909 mapread 0x0265d4 thru 0x03230c (0xbd39 bytes) 000614 1781228391.125668 mapwrite 0x03b49b thru 0x03cefa (0x1a60 bytes) 000615 1781228391.175263 trunc from 0x03cefb to 0x00df76 (0x2ef84 bytes) 000616 1781228391.186145 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x385:0x0]: Invalid argument (22) 000618 1781228391.245875 mapread 0x00c592 thru 0x00df76 (0x19e5 bytes) 000624 1781228391.257190 read 0x0043d5 thru 0x005dd6 (0x1a02 bytes) 000627 1781228391.262693 mapread 0x001abf thru 0x00df76 (0xc4b8 bytes) 000628 1781228391.268532 mapwrite 0x026892 thru 0x027880 (0xfef bytes) 000629 1781228391.305611 mapread 0x01db98 thru 0x0230a1 (0x550a bytes) 000630 1781228391.310940 trunc from 0x027881 to 0x02a084 (0x2804 bytes) 000633 1781228391.319488 write 0x01888a thru 0x024855 (0xbfcc bytes) 000634 1781228391.320884 trunc from 0x02a085 to 0x01ffdb (0xa0a9 bytes) 000636 1781228391.330595 read 0x013f86 thru 0x01ffdb (0xc056 bytes) 000637 1781228391.336365 mapread 0x002ce8 thru 0x0072e8 (0x4601 bytes) 000638 1781228391.337659 trunc from 0x01ffdc to 0x01e3e9 (0x1bf2 bytes) 000639 1781228391.346842 trunc from 0x01e3ea to 0x0045fb (0x19dee bytes) 000640 1781228391.355877 write 0x02e8d1 thru 0x03d205 (0xe935 bytes) 000642 1781228391.358056 read 0x00fbe2 thru 0x01a917 (0xad36 bytes) 000643 1781228391.362550 write 0x036b3f thru 0x038de1 (0x22a3 bytes) 000644 1781228391.363330 trunc from 0x03d206 to 0x02d4c1 (0xfd44 bytes) 000645 1781228391.370079 mapread 0x002df8 thru 0x0053df (0x25e8 bytes) 000649 1781228391.372937 write 0x00f855 thru 0x01ef2e (0xf6da bytes) 000650 1781228391.374228 mapwrite 0x039c8b thru 0x03ffff (0x6375 bytes) 000652 1781228391.415608 mapread 0x001937 thru 0x0077ea (0x5eb4 bytes) 000655 1781228391.417300 trunc from 0x040000 to 0x000199 (0x3fe66 bytes) 000656 1781228391.424384 read 0x00011c thru 0x000199 (0x7e bytes) 000659 1781228391.425554 mapwrite 0x01a51e thru 0x0243de (0x9ec1 bytes) 000661 1781228391.460872 trunc from 0x0243df to 0x028e5e (0x4a80 bytes) 000662 1781228391.469657 write 0x025d6b thru 0x02a583 (0x4819 bytes) 000663 1781228391.474065 mapwrite 0x031d56 thru 0x03bdb2 (0xa05d bytes) 000665 1781228391.517831 read 0x037af7 thru 0x03bdb2 (0x42bc bytes) 000670 1781228391.520621 trunc from 0x03bdb3 to 0x012996 (0x2941c bytes) 000673 1781228391.536002 read 0x00f5de thru 0x01265f (0x3082 bytes) 000674 1781228391.536603 trunc from 0x012997 to 0x0189e6 (0x6050 bytes) 000680 1781228391.552941 trunc from 0x0189e7 to 0x01b93a (0x2f54 bytes) 000681 1781228391.562399 mapread 0x0120b0 thru 0x01b93a (0x988b bytes) 000683 1781228391.569824 write 0x02fd53 thru 0x03d627 (0xd8d5 bytes) 000684 1781228391.571278 trunc from 0x03d628 to 0x0391ab (0x447c bytes) 000685 1781228391.583286 read 0x01a3d6 thru 0x01b5dd (0x1208 bytes) 000686 1781228391.583982 mapread 0x0173fb thru 0x023ffb (0xcc01 bytes) 000687 1781228391.591716 trunc from 0x0391ac to 0x0339c0 (0x57eb bytes) 000690 1781228391.603828 write 0x0346db thru 0x03db3a (0x9460 bytes) 000691 1781228391.606135 trunc from 0x03db3b to 0x013144 (0x2a9f6 bytes) 000692 1781228391.617326 read 0x00bba0 thru 0x00c255 (0x6b6 bytes) 000694 1781228391.618439 trunc from 0x013145 to 0x00f370 (0x3dd4 bytes) 000699 1781228391.631187 trunc from 0x00f371 to 0x02a730 (0x1b3c0 bytes) 000702 1781228391.641334 trunc from 0x02a731 to 0x00829b (0x22495 bytes) 000703 1781228391.651693 trunc from 0x00829c to 0x026b87 (0x1e8ec bytes) 000706 1781228391.661535 mapwrite 0x0021ab thru 0x008642 (0x6498 bytes) 000707 1781228391.692717 trunc from 0x026b88 to 0x029726 (0x2b9f bytes) 000708 1781228391.701623 mapwrite 0x0083c1 thru 0x00f5e8 (0x7228 bytes) 000711 1781228391.734008 trunc from 0x029727 to 0x02e16c (0x4a46 bytes) 000712 1781228391.744190 write 0x033cc3 thru 0x03999f (0x5cdd bytes) 000713 1781228391.745807 read 0x004a8d thru 0x005579 (0xaed bytes) 000714 1781228391.746607 write 0x00d883 thru 0x01b304 (0xda82 bytes) 000715 1781228391.747972 trunc from 0x0399a0 to 0x024fd3 (0x149cc bytes) 000716 1781228391.759435 write 0x021ed3 thru 0x030e2d (0xef5b bytes) 000717 1781228391.761253 trunc from 0x030e2e to 0x02639e (0xaa8f bytes) 000718 1781228391.770201 trunc from 0x02639f to 0x00414d (0x22251 bytes) 000719 1781228391.781723 mapread 0x00296a thru 0x00414d (0x17e4 bytes) 000720 1781228391.782731 trunc from 0x00414e to 0x03bcbf (0x37b72 bytes) 000721 1781228391.792686 trunc from 0x03bcc0 to 0x0182e1 (0x239de bytes) 000724 1781228391.800600 mapread 0x00a02d thru 0x014af6 (0xaaca bytes) 000725 1781228391.806863 read 0x00dc5c thru 0x00fd2a (0x20cf bytes) 000726 1781228391.807651 mapread 0x015485 thru 0x0182e1 (0x2e5d bytes) 000729 1781228391.812463 MIRROR_EXTEND to 2 mirrors 000730 1781228392.042158 trunc from 0x0182e2 to 0x0104a6 (0x7e3b bytes) 000731 1781228392.098365 trunc from 0x0104a7 to 0x029dbc (0x19916 bytes) 000732 1781228392.108347 mapread 0x01cf94 thru 0x01e320 (0x138d bytes) 000734 1781228392.126505 MIRROR_RESYNC 2 mirrors 000736 1781228392.378645 mapwrite 0x03d8f2 thru 0x03ffff (0x270e bytes) 000737 1781228392.458728 trunc from 0x040000 to 0x017ec5 (0x2813a bytes) 000738 1781228392.478053 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x17ec6): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17ec6): Mirror 2: 0xcba05ae3 (stale) CRC-32 checksum value for chunk [0, 0x17ec6): Mirror 1: 0xcba05ae3 Mirror 3: 0xcba05ae3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17ec6: skipped 000740 1781228392.823659 mapread 0x007ac8 thru 0x0111c4 (0x96fd bytes) 000741 1781228392.837460 read 0x014f2e thru 0x017ec5 (0x2f98 bytes) 000743 1781228392.845633 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x17ec6): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17ec6): Mirror 2: 0xcba05ae3 (stale) CRC-32 checksum value for chunk [0, 0x17ec6): Mirror 1: 0xcba05ae3 Mirror 3: 0xcba05ae3 Mirror 4: 0xcba05ae3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17ec6: skipped 000746 1781228393.227875 mapread 0x0041fe thru 0x00f98a (0xb78d bytes) 000747 1781228393.241633 mapwrite 0x0007f9 thru 0x004fc1 (0x47c9 bytes) 000748 1781228393.334057 write 0x003104 thru 0x00d220 (0xa11d bytes) 000749 1781228393.337419 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x17ec6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17ec6): Mirror 2: 0xcba05ae3 (stale) Mirror 3: 0xcba05ae3 (stale) Mirror 4: 0xcba05ae3 (stale) CRC-32 checksum value for chunk [0, 0x17ec6): Mirror 1: 0x4d8c8818 Mirror 5: 0x4d8c8818 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17ec6: skipped 000751 1781228393.777080 read 0x00a8ba thru 0x017ec5 (0xd60c bytes) 000752 1781228393.785063 trunc from 0x017ec6 to 0x01c2ec (0x4427 bytes) 000753 1781228393.839365 mapwrite 0x016d0d thru 0x01c3aa (0x569e bytes) 000755 1781228393.912931 mapread 0x00a847 thru 0x015570 (0xad2a bytes) 000759 1781228393.927527 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1c3ab): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c3ab): Mirror 2: 0x93b1b741 (stale) Mirror 3: 0x93b1b741 (stale) Mirror 4: 0x93b1b741 (stale) Mirror 5: 0xc63fdb0d (stale) CRC-32 checksum value for chunk [0, 0x1c3ab): Mirror 1: 0x1c4595c5 Mirror 6: 0x1c4595c5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c3ab: skipped 000760 1781228394.472307 mapread 0x01816e thru 0x01ad09 (0x2b9c bytes) 000762 1781228394.487789 read 0x007852 thru 0x011842 (0x9ff1 bytes) 000764 1781228394.492575 trunc from 0x01c3ab to 0x03c45c (0x200b2 bytes) 000765 1781228394.539421 trunc from 0x03c45d to 0x03ce3e (0x9e2 bytes) 000766 1781228394.550466 read 0x0116c0 thru 0x01846b (0x6dac bytes) 000767 1781228394.558447 mapread 0x016165 thru 0x024dac (0xec48 bytes) 000768 1781228394.559908 mapread 0x03c9c9 thru 0x03ce3e (0x476 bytes) 000769 1781228394.560848 read 0x0148f6 thru 0x023208 (0xe913 bytes) 000776 1781228394.565038 trunc from 0x03ce3f to 0x0005ce (0x3c870 bytes) 000777 1781228394.574830 write 0x01b6da thru 0x020435 (0x4d5c bytes) 000778 1781228394.576116 trunc from 0x020436 to 0x033ad0 (0x1369b bytes) 000779 1781228394.588777 read 0x01dd32 thru 0x02bbfb (0xdeca bytes) 000780 1781228394.597066 mapwrite 0x00fc28 thru 0x0196fd (0x9ad6 bytes) 000782 1781228394.629062 read 0x015256 thru 0x021023 (0xbdce bytes) 000783 1781228394.629801 read 0x00cf0a thru 0x011636 (0x472d bytes) 000784 1781228394.630354 mapread 0x017862 thru 0x024ced (0xd48c bytes) 000785 1781228394.634146 mapread 0x01a879 thru 0x022eb7 (0x863f bytes) 000787 1781228394.636452 mapread 0x010e56 thru 0x020da5 (0xff50 bytes) 000790 1781228394.638686 write 0x016785 thru 0x020b8c (0xa408 bytes) 000791 1781228394.639919 write 0x0058b5 thru 0x0122ef (0xca3b bytes) 000792 1781228394.641219 read 0x01e93d thru 0x023d98 (0x545c bytes) 000795 1781228394.642784 read 0x020e23 thru 0x025a49 (0x4c27 bytes) 000798 1781228394.644577 mapwrite 0x00c348 thru 0x01a06e (0xdd27 bytes) 000799 1781228394.667367 mapread 0x00210d thru 0x00340b (0x12ff bytes) 000801 1781228394.668896 mapread 0x0173d0 thru 0x01a099 (0x2cca bytes) 000802 1781228394.669977 write 0x021c9e thru 0x021cce (0x31 bytes) 000805 1781228394.671857 mapwrite 0x035a0e thru 0x03ffff (0xa5f2 bytes) 000807 1781228394.719789 mapwrite 0x03b90c thru 0x03ffff (0x46f4 bytes) 000810 1781228394.750605 mapread 0x00d5e8 thru 0x015f0b (0x8924 bytes) 000813 1781228394.756555 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x385:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x5392e311 (stale) Mirror 4: 0x5392e311 (stale) Mirror 5: 0xd4053333 (stale) Mirror 6: 0xb2564fc0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x5392e311 (stale) Mirror 4: 0x5392e311 (stale) Mirror 5: 0xd4053333 (stale) Mirror 6: 0xb2564fc0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000816 1781228394.952590 trunc from 0x040000 to 0x035c23 (0xa3dc bytes) 000818 1781228394.965320 write 0x00ea43 thru 0x015457 (0x6a15 bytes) 000819 1781228394.972038 trunc from 0x035c24 to 0x03ae96 (0x5273 bytes) 000820 1781228394.984610 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3ae97): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ae97): Mirror 3: 0x5425837e (stale) Mirror 4: 0x5425837e (stale) Mirror 5: 0xe535a3d (stale) Mirror 6: 0x5824e26a (stale) CRC-32 checksum value for chunk [0, 0x3ae97): Mirror 1: 0xc01679be Mirror 7: 0xc01679be lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ae97: skipped 000821 1781228395.445053 write 0x024ace thru 0x034086 (0xf5b9 bytes) 000822 1781228395.488181 write 0x03474f thru 0x03ca94 (0x8346 bytes) 000824 1781228395.489920 trunc from 0x03ca95 to 0x00d876 (0x2f21e bytes) 000825 1781228395.501240 mapwrite 0x01402b thru 0x01536d (0x1343 bytes) 000826 1781228395.541246 read 0x012dc8 thru 0x01536d (0x25a6 bytes) 000827 1781228395.545725 MIRROR_RESYNC 6 mirrors 000829 1781228395.988420 read 0x00f7f0 thru 0x01536d (0x5b7e bytes) 000832 1781228395.995603 mapread 0x001235 thru 0x006947 (0x5713 bytes) 000833 1781228396.010396 MIRROR_EXTEND to 7 mirrors 000836 1781228396.393862 mapwrite 0x022fb1 thru 0x02bd46 (0x8d96 bytes) 000837 1781228396.494346 trunc from 0x02bd47 to 0x037de8 (0xc0a2 bytes) 000838 1781228396.508390 read 0x02c33c thru 0x037de8 (0xbaad bytes) 000839 1781228396.514720 mapread 0x016c2c thru 0x020996 (0x9d6b bytes) 000840 1781228396.522435 write 0x011248 thru 0x01e300 (0xd0b9 bytes) 000841 1781228396.523775 mapwrite 0x03f6d8 thru 0x03ffff (0x928 bytes) 000842 1781228396.586622 read 0x02e52e thru 0x0355b8 (0x708b bytes) 000843 1781228396.587409 write 0x01503b thru 0x023b3e (0xeb04 bytes) 000844 1781228396.590726 mapwrite 0x02381c thru 0x02c8c5 (0x90aa bytes) 000845 1781228396.623273 write 0x013c52 thru 0x01d45a (0x9809 bytes) 000846 1781228396.624898 read 0x00ca7d thru 0x00fe5a (0x33de bytes) 000847 1781228396.639895 MIRROR_RESYNC 7 mirrors 000852 1781228397.241608 trunc from 0x040000 to 0x0309f3 (0xf60c bytes) 000854 1781228397.287006 trunc from 0x0309f4 to 0x03b310 (0xa91d bytes) 000855 1781228397.305181 read 0x0042ff thru 0x01009e (0xbda0 bytes) 000856 1781228397.313075 read 0x032bb9 thru 0x0363c0 (0x3808 bytes) 000857 1781228397.332099 write 0x02959a thru 0x035dc8 (0xc82f bytes) 000858 1781228397.333486 trunc from 0x03b311 to 0x00ed46 (0x2c5ca bytes) 000859 1781228397.344740 trunc from 0x00ed47 to 0x008995 (0x63b1 bytes) 000860 1781228397.358027 MIRROR_RESYNC 7 mirrors 000863 1781228397.943397 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x385:0x0]: Invalid argument (22) 000866 1781228398.162699 read 0x0049b8 thru 0x008012 (0x365b bytes) 000867 1781228398.169644 mapwrite 0x025849 thru 0x02d09d (0x7855 bytes) 000868 1781228398.327090 read 0x0291c2 thru 0x02d09d (0x3edc bytes) 000869 1781228398.327648 write 0x039380 thru 0x03e459 (0x50da bytes) 000870 1781228398.328722 mapwrite 0x0304b2 thru 0x03a2af (0x9dfe bytes) 000876 1781228398.367739 read 0x001d45 thru 0x00a685 (0x8941 bytes) 000877 1781228398.368504 trunc from 0x03e45a to 0x0352ca (0x918f bytes) 000878 1781228398.383248 read 0x00c5bc thru 0x00ed96 (0x27db bytes) 000879 1781228398.384080 mapwrite 0x025579 thru 0x0344c5 (0xef4d bytes) 000881 1781228398.427355 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x352cb): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x352cb): Mirror 3: 0x46c07305 (stale) Mirror 4: 0x46c07305 (stale) Mirror 5: 0x46c07305 (stale) Mirror 6: 0x46c07305 (stale) Mirror 7: 0x46c07305 (stale) CRC-32 checksum value for chunk [0, 0x352cb): Mirror 1: 0xadb81804 Mirror 8: 0xadb81804 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x352cb: skipped 000882 1781228398.915535 MIRROR_RESYNC 7 mirrors 000883 1781228399.544394 write 0x03c2ba thru 0x03ffff (0x3d46 bytes) 000884 1781228399.580654 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x385:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe2dda845 (stale) Mirror 4: 0xe2dda845 (stale) Mirror 6: 0xe2dda845 (stale) Mirror 7: 0xe2dda845 (stale) Mirror 8: 0xe2dda845 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe2dda845 (stale) Mirror 4: 0xe2dda845 (stale) Mirror 6: 0xe2dda845 (stale) Mirror 7: 0xe2dda845 (stale) Mirror 8: 0xe2dda845 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000886 1781228399.833774 mapread 0x010ed9 thru 0x015f58 (0x5080 bytes) 000888 1781228399.846477 read 0x02b18c thru 0x02dfdd (0x2e52 bytes) 000889 1781228399.850167 mapread 0x02f88f thru 0x03ef6f (0xf6e1 bytes) 000890 1781228399.862052 read 0x03e028 thru 0x03ffff (0x1fd8 bytes) 000893 1781228399.863582 trunc from 0x040000 to 0x00b20a (0x34df5 bytes) 000895 1781228399.874924 mapread 0x001f8a thru 0x00b20a (0x9281 bytes) 000896 1781228399.880772 write 0x03de55 thru 0x03ffff (0x21ab bytes) 000897 1781228399.881830 mapwrite 0x0086c2 thru 0x0114b5 (0x8df4 bytes) 000899 1781228399.916786 mapwrite 0x03fdcc thru 0x03ffff (0x234 bytes) 000903 1781228399.942876 write 0x01757e thru 0x01a437 (0x2eba bytes) 000905 1781228399.944846 mapwrite 0x02af86 thru 0x02e3c8 (0x3443 bytes) 000906 1781228399.982485 mapread 0x018cd1 thru 0x01d409 (0x4739 bytes) 000907 1781228399.983601 mapread 0x012cc5 thru 0x014b9d (0x1ed9 bytes) 000908 1781228399.984641 mapread 0x008f65 thru 0x013804 (0xa8a0 bytes) 000910 1781228399.986085 read 0x037e62 thru 0x03998a (0x1b29 bytes) 000912 1781228399.987189 read 0x02914c thru 0x02c052 (0x2f07 bytes) 000914 1781228399.988417 mapread 0x01869d thru 0x026fbb (0xe91f bytes) 000915 1781228399.989706 trunc from 0x040000 to 0x0380a1 (0x7f5e bytes) 000916 1781228400.002033 write 0x01e297 thru 0x0272e7 (0x9051 bytes) 000917 1781228400.005617 trunc from 0x0380a2 to 0x0186d2 (0x1f9cf bytes) 000918 1781228400.017802 trunc from 0x0186d3 to 0x01fd5e (0x768c bytes) 000920 1781228400.027091 mapwrite 0x021bbf thru 0x02c0b5 (0xa4f7 bytes) 000922 1781228400.091128 write 0x00521d thru 0x01232e (0xd112 bytes) 000923 1781228400.092705 trunc from 0x02c0b6 to 0x0172af (0x14e06 bytes) 000925 1781228400.103935 mapread 0x006259 thru 0x0155b9 (0xf361 bytes) 000927 1781228400.105596 mapwrite 0x01dee7 thru 0x028b0a (0xac24 bytes) 000928 1781228400.195228 write 0x007eaf thru 0x016042 (0xe194 bytes) 000929 1781228400.196551 mapread 0x013e36 thru 0x01a7e5 (0x69b0 bytes) 000930 1781228400.210137 trunc from 0x028b0b to 0x0002c8 (0x28842 bytes) 000931 1781228400.221116 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2c9): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c9): Mirror 3: 0x147ddee2 (stale) Mirror 4: 0x147ddee2 (stale) Mirror 6: 0x147ddee2 (stale) Mirror 7: 0x147ddee2 (stale) Mirror 8: 0x147ddee2 (stale) CRC-32 checksum value for chunk [0, 0x2c9): Mirror 1: 0x147ddee2 Mirror 9: 0x147ddee2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c9: skipped 000932 1781228400.724411 mapwrite 0x02fcfc thru 0x03a20d (0xa512 bytes) 000933 1781228400.808980 mapwrite 0x022a0c thru 0x027870 (0x4e65 bytes) 000934 1781228400.844495 mapwrite 0x02fb5b thru 0x03021c (0x6c2 bytes) 000935 1781228400.866132 write 0x00bd08 thru 0x015308 (0x9601 bytes) 000937 1781228400.880323 mapwrite 0x018c8d thru 0x027b6d (0xeee1 bytes) 000938 1781228400.918377 trunc from 0x03a20e to 0x00b33b (0x2eed2 bytes) 000939 1781228400.933855 mapread 0x000fec thru 0x002a1a (0x1a2f bytes) 000940 1781228400.938413 trunc from 0x00b33c to 0x031251 (0x25f16 bytes) 000941 1781228400.948118 trunc from 0x031252 to 0x027e36 (0x941b bytes) 000942 1781228400.959888 write 0x03814e thru 0x03ffff (0x7eb2 bytes) 000943 1781228400.961159 trunc from 0x040000 to 0x0260f5 (0x19f0a bytes) 000944 1781228400.971658 trunc from 0x0260f6 to 0x011a77 (0x1467e bytes) 000948 1781228400.987011 mapread 0x000b7f thru 0x00bb3f (0xafc1 bytes) 000949 1781228400.988565 read 0x0038c3 thru 0x00ba5c (0x819a bytes) 000950 1781228400.989632 mapwrite 0x037ba2 thru 0x03e79c (0x6bfb bytes) 000952 1781228401.034362 mapread 0x02ce8c thru 0x0394e3 (0xc658 bytes) 000954 1781228401.042810 mapread 0x0328c7 thru 0x03c342 (0x9a7c bytes) 000957 1781228401.044598 mapwrite 0x023f4a thru 0x0280f4 (0x41ab bytes) 000962 1781228401.077546 mapread 0x03cd2d thru 0x03e79c (0x1a70 bytes) 000964 1781228401.079160 read 0x01523a thru 0x01b03a (0x5e01 bytes) 000965 1781228401.087789 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3e79d): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e79d): Mirror 3: 0xdc99279a (stale) Mirror 4: 0xdc99279a (stale) Mirror 6: 0xdc99279a (stale) Mirror 7: 0xdc99279a (stale) Mirror 8: 0xdc99279a (stale) Mirror 9: 0x3423d7d3 (stale) CRC-32 checksum value for chunk [0, 0x3e79d): Mirror 1: 0xe804411c Mirror 10: 0xe804411c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e79d: skipped 000966 1781228401.500159 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3e79d): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e79d): Mirror 3: 0xdc99279a (stale) Mirror 4: 0xdc99279a (stale) Mirror 6: 0xdc99279a (stale) Mirror 7: 0xdc99279a (stale) Mirror 8: 0xdc99279a (stale) Mirror 9: 0x3423d7d3 (stale) CRC-32 checksum value for chunk [0, 0x3e79d): Mirror 1: 0xe804411c Mirror 10: 0xe804411c Mirror 11: 0xe804411c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e79d: skipped 000967 1781228401.858160 MIRROR_RESYNC 9 mirrors 000968 1781228402.373282 mapwrite 0x029f32 thru 0x032642 (0x8711 bytes) 000970 1781228402.472061 read 0x031042 thru 0x037e4a (0x6e09 bytes) 000971 1781228402.481925 mapwrite 0x022934 thru 0x02443e (0x1b0b bytes) 000972 1781228402.507939 mapread 0x00d430 thru 0x017646 (0xa217 bytes) 000974 1781228402.523248 read 0x019fd5 thru 0x023ef6 (0x9f22 bytes) 000975 1781228402.524767 write 0x03cee3 thru 0x03ffff (0x311d bytes) 000976 1781228402.531775 trunc from 0x040000 to 0x028647 (0x179b8 bytes) 000977 1781228402.544716 trunc from 0x028648 to 0x035d5a (0xd713 bytes) 000979 1781228402.558648 trunc from 0x035d5b to 0x00be32 (0x29f28 bytes) 000980 1781228402.567487 read 0x003aab thru 0x006309 (0x285f bytes) 000983 1781228402.569317 read 0x00a87a thru 0x00bc53 (0x13da bytes) 000984 1781228402.570056 trunc from 0x00be33 to 0x02e604 (0x227d2 bytes) 000985 1781228402.579073 write 0x00b1e3 thru 0x01a2ec (0xf10a bytes) 000987 1781228402.585456 mapread 0x01d827 thru 0x023fb3 (0x678d bytes) 000990 1781228402.594324 read 0x01214f thru 0x013b15 (0x19c7 bytes) 000991 1781228402.598921 MIRROR_RESYNC 9 mirrors 000992 1781228403.262479 mapwrite 0x03a215 thru 0x03aa7b (0x867 bytes) 000993 1781228403.334424 read 0x022a8a thru 0x02c5af (0x9b26 bytes) 000994 1781228403.339099 mapread 0x0092be thru 0x00c30d (0x3050 bytes) 000995 1781228403.353804 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3aa7c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aa7c): Mirror 3: 0x6ae5329f (stale) Mirror 4: 0x6ae5329f (stale) Mirror 6: 0x6ae5329f (stale) Mirror 7: 0x6ae5329f (stale) Mirror 8: 0x6ae5329f (stale) Mirror 9: 0x6ae5329f (stale) Mirror 10: 0x6ae5329f (stale) Mirror 11: 0x6ae5329f (stale) CRC-32 checksum value for chunk [0, 0x3aa7c): Mirror 1: 0xa23a1d80 Mirror 12: 0xa23a1d80 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aa7c: skipped 000996 1781228403.821202 mapread 0x038d7c thru 0x03aa7b (0x1d00 bytes) All operations completed A-OK! PASS 70a (31s) == sanity-flr test 71: check mirror extend parent fid ==== 21:40:06 (1781228406) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 72: check append on FLR file ========== 21:40:07 (1781228407) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5e1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.133438 s, 15.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5fe:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5e1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094674 s, 11.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5fe:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5e2:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5e1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5fe:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5e2:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5e1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5ff:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (4s) == sanity-flr test 200a: stress test ===================== 21:40:11 (1781228411) Starting client: oleg633-client.virtnet: -o user_xattr,flock 192.168.206.133@tcp:/lustre /mnt/lustre2 Starting client: oleg633-client.virtnet: -o user_xattr,flock 192.168.206.133@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3932704 .. Extending file size to 3305344 .. Extending file size to 5597472 .. Extending file size to 6095584 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7181984 .. failed Extending file size to 7344384 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 6515168 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7272960 .. Extending file size to 8540192 .. done Extending file size to 8462240 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9236544 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8727552 .. Extending file size to 9118080 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9237440 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 9256992 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9171392 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 58934 58935 58936 58939 58941 done 192.168.206.133@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg633-client.virtnet /mnt/lustre2 (opts:) 192.168.206.133@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg633-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (69s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 21:41:20 (1781228480) Starting client: oleg633-client.virtnet: -o user_xattr,flock 192.168.206.133@tcp:/lustre /mnt/lustre2 Starting client: oleg633-client.virtnet: -o user_xattr,flock 192.168.206.133@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1988608 .. Extending file size to 5157088 .. done Extending file size to 3260672 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 5639552 .. Extending file size to 7290880 .. Extending file size to 7371264 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7663488 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8919488 .. Extending file size to 7885312 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7947904 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8769440 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9000896 .. Extending file size to 9110080 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9165056 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 61968 61969 61970 61974 61975 192.168.206.133@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg633-client.virtnet /mnt/lustre2 (opts:) 192.168.206.133@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg633-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (66s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 21:42:26 (1781228546) Starting client: oleg633-client.virtnet: -o user_xattr,flock 192.168.206.133@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3328 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3328 2205056 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0529149 s, 39.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.122178 s, 17.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x46a42fd8 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.206.133@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg633-client.virtnet /mnt/lustre2 (opts:) PASS 200c (8s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 21:42:34 (1781228554) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0414292 s, 50.6 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 21:42:37 (1781228557) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c5:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5fe:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5fe:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5ff:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x61a:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 21:42:40 (1781228560) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0580058 s, 36.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x600:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x61b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x61c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 21:42:43 (1781228563) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0811593 s, 25.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x601:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x602:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x61d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x601:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x602:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x61d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x61e:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 21:42:46 (1781228566) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0872762 s, 24.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x603:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x61f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x603:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x61f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x620:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 21:43:11 (1781228591) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg633-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410195 s, 25.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x604:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x622:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x621:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x604:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x622:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x621:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x605:0x0] } PASS 204d (43s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 21:43:54 (1781228634) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg633-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x623:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x624:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0836675 s, 25.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x623:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x624:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x606:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x623:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x624:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x606:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (45s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 21:44:39 (1781228679) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg633-server mds-ost sync done. Creating new pool test_204f oleg633-server: Pool lustre.test_204f created Adding targets to pool oleg633-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg633-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.06545 s, 32.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x625:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x607:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x625:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x607:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg633-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg633-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg633-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (60s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 21:45:39 (1781228739) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 21:45:42 (1781228742) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 21:45:45 (1781228745) oleg633-server: Pool lustre.test_206 created Adding targets to pool oleg633-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg633-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg633-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg633-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg633-server: Pool lustre.test_206 destroyed PASS 206 (17s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 21:46:02 (1781228762) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327538 s, 32.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182077 s, 57.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x9bd223e3 Mirror 3: 0x9bd223e3 PASS 207 (5s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 21:46:07 (1781228767) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 21:46:09 (1781228769) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 21:46:11 (1781228771) Starting client: oleg633-client.virtnet: -o user_xattr,flock 192.168.206.133@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0591388 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x13f0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xf60000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xf60000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xf60000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x850000 Read this (4 bytes): BBBB 192.168.206.133@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg633-client.virtnet /mnt/lustre2 (opts:) PASS 209a (22s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 21:46:33 (1781228793) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0150422 s, 545 kB/s the very first read read with warm cache PASS 209b (4s) == sanity-flr test 210a: handle broken mirrored lovea ==== 21:46:37 (1781228797) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210702 s, 49.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x3ea lmm_fid: [0x200000404:0x3ea:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x3ea lmm_fid: [0x200000404:0x3ea:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x611:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834922 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-11 21:46:37.000000000 -0400 Modify: 2026-06-11 21:46:37.000000000 -0400 Change: 2026-06-11 21:46:38.000000000 -0400 Birth: 2026-06-11 21:46:37.000000000 -0400 PASS 210a (4s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 21:46:41 (1781228801) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 21:46:43 (1781228803) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.213807 s, 192 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0189678 s, 216 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 11 21:46 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ec:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 11 21:46 /mnt/lustre/f211.sanity-flr PASS 211 (4s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 21:46:47 (1781228807) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (4s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 21:46:51 (1781228811) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (5s) == sanity-flr test complete, duration 913 sec ============ 21:46:56 (1781228816) === sanity-flr: start cleanup 21:46:57 (1781228817) === === sanity-flr: finish cleanup 21:47:00 (1781228820) ===