-----============= acceptance-small: sanity-flr ============----- Fri May 29 16:21:14 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg346-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 16:21:22 (1780086082) === oleg346-client.virtnet: executing check_config_client /mnt/lustre oleg346-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg346-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff950ec7ece000.idle_timeout=debug osc.lustre-OST0001-osc-ffff950ec7ece000.idle_timeout=debug disable quota as required oleg346-server: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 16:21:37 (1780086097) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7550] == sanity-flr test 0a: lfs mirror create with -N option == 16:21:38 (1780086098) PASS 0a (6s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 16:21:44 (1780086104) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 16:21:46 (1780086106) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 16:21:47 (1780086107) PASS 0d (12s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 16:21:59 (1780086119) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 16:22:01 (1780086121) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 16:22:03 (1780086123) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff950ec7ece000.stats=clear osc.lustre-OST0001-osc-ffff950ec7ece000.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 16:22:06 (1780086126) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (6s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 16:22:12 (1780086132) PASS 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 16:22:16 (1780086136) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 16:22:18 (1780086138) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 16:22:22 (1780086142) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 16:22:27 (1780086147) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 16:22:30 (1780086150) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 16:22:31 (1780086151) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 16:22:34 (1780086154) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 16:22:39 (1780086159) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 16:22:41 (1780086161) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.318568 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053293 s, 19.7 MB/s PASS 21 (5s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 16:22:46 (1780086166) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 0.17968 s, 81.7 MB/s PASS 22 (3s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 16:22:49 (1780086169) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (5s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 16:22:54 (1780086174) Stopping /mnt/lustre-ost1 (opts:) on oleg346-server oleg346-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff950ec7ece000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff950ec7ece000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg346-server: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg346-server: oleg346-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg346-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg346-server oleg346-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff950ec7ece000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff950ec7ece000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg346-server: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg346-server: oleg346-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg346-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (29s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 16:23:23 (1780086203) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.119144 s, 44.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg346-server oleg346-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff950ec7ece000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff950ec7ece000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg346-server: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg346-server: oleg346-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg346-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (15s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 16:23:38 (1780086218) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg346-server oleg346-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff950ec7ece000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff950ec7ece000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg346-server: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg346-server: oleg346-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg346-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg346-server oleg346-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff950ec7ece000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff950ec7ece000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg346-server: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg346-server: oleg346-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg346-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (30s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 16:24:08 (1780086248) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.28247 s, 23.0 MB/s -rw-r--r-- 1 root root 52428800 May 29 16:24 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.506s user 0m0.002s sys 0m0.174s Stopping /mnt/lustre-ost1 (opts:) on oleg346-server oleg346-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff950ec7ece000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff950ec7ece000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... real 0m0.501s user 0m0.002s sys 0m0.177s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg346-server: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg346-server: oleg346-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg346-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg346-server oleg346-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff950ec7ece000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff950ec7ece000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.585s user 0m0.000s sys 0m0.160s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg346-server: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg346-server: oleg346-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg346-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg346-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff950ec7ece000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff950ec7ece000.ost_server_uuid in FULL state after 0 sec PASS 33b (41s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 16:24:49 (1780086289) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 16:24:50 (1780086290) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 16:24:52 (1780086292) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 16:24:53 (1780086293) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 16:24:56 (1780086296) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.196647 s, 64.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (5s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 16:25:01 (1780086301) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147927 s, 70.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184388 s, 56.9 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 16:25:04 (1780086304) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 16:25:08 (1780086308) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00089811 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000769375 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00081433 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000635608 s, 1.6 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 16:25:11 (1780086311) create test files with size 15737902 .. d318c5fe12f2d4ad52fc42bade6a590c - f915485c460db9831b38d53d062e2ca6 - 7e189e7ec74cc54b365a88921ed35fa7 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre PASS 37 (7s) == sanity-flr test 38: resync ============================ 16:25:19 (1780086319) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (6s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 16:25:24 (1780086324) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 16:25:27 (1780086327) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369168 s, 28.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xca:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xc8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137868 s, 76.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xc9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xca:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 16:25:31 (1780086331) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0474588 s, 44.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.149658 s, 28.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0798194 s, 26.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (6s) == sanity-flr test 42: lfs mirror verify ================= 16:25:37 (1780086337) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 16:25:38 (1780086338) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 16:25:40 (1780086340) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133716 s, 78.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 16:25:42 (1780086342) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 16:25:44 (1780086344) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd1:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd3:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd3:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 16:25:47 (1780086347) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.116032 s, 90.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (18s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 16:26:05 (1780086365) llite.lustre-ffff950ec39cf000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0897808 s, 117 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff950ec4d33000.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 16:26:10 (1780086370) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (15s) == sanity-flr test 46: Verify setstripe --copy option ==== 16:26:25 (1780086385) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (14s) == sanity-flr test 47: Verify mirror obj alloc =========== 16:26:39 (1780086399) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 16:26:40 (1780086400) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0767149 s, 41.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0646514 s, 48.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x1484052f (nosync,stale) Mirror 2: 0x1484052f (nosync,stale) ** verify mirror 2 doesn't change original checksum: 060d7bf2b6ea541a623e9f52b87991d9 - mirror 1 checksum: 108d3110ebc1d7660d313a2c83480155 - mirror 2 checksum: 060d7bf2b6ea541a623e9f52b87991d9 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri May 29 16:26:42 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri May 29 16:26:42 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe9:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 060d7bf2b6ea541a623e9f52b87991d9 - mirror 1 checksum: 108d3110ebc1d7660d313a2c83480155 - mirror 2 checksum: 108d3110ebc1d7660d313a2c83480155 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri May 29 16:26:42 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri May 29 16:26:42 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe9:0x0] } PASS 48 (4s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 16:26:45 (1780086405) Failing mds1 on oleg346-server Stopping /mnt/lustre-mds1 (opts:) on oleg346-server 16:26:47 (1780086407) shut down facet: mds1 facet_host: oleg346-server facet_failover_host: oleg346-server Failover mds1 to oleg346-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg346-server: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-MDT0000 16:27:02 (1780086422) targets are mounted 16:27:02 (1780086422) facet_failover done oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152017 s, 69.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xea:0x0] } PASS 50A (24s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 16:27:09 (1780086429) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 16:27:11 (1780086431) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 16:27:12 (1780086432) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00454066 s, 3.6 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 16:27:15 (1780086435) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 16:27:16 (1780086436) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 16:27:18 (1780086438) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (8s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 16:27:26 (1780086446) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (21s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 16:27:47 (1780086467) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 16:28:05 (1780086485) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 16:28:24 (1780086504) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 16:28:27 (1780086507) mirror create pid 53404 mirror split pid 53405 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 53404 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 53405 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 16:29:30 (1780086570) Seed set to 6571 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x6288 000002 1780086571.465503 trunc from 00000000 to 0x006287 (0x6288 bytes) 000003 1780086571.472071 mapwrite 0x0263ef thru 0x02d7aa (0x73bc bytes) 000004 1780086571.513819 mapread 0x00a76a thru 0x012fb5 (0x884c bytes) 000006 1780086571.521209 mapread 0x019802 thru 0x021234 (0x7a33 bytes) 000007 1780086571.526042 read 0x02d59b thru 0x02d7aa (0x210 bytes) 000009 1780086571.527061 write 0x00f154 thru 0x019978 (0xa825 bytes) 000013 1780086571.529747 trunc from 0x02d7ab to 0x004da8 (0x28a02 bytes) truncating to largest ever: 0xf3a6 000014 1780086571.539439 trunc from 0x004da9 to 0x00f3a5 (0xa5fd bytes) 000015 1780086571.548201 MIRROR_EXTEND to 2 mirrors 000016 1780086571.795388 mapread 0x004f3f thru 0x00f3a5 (0xa467 bytes) truncating to largest ever: 0x14925 000017 1780086571.806038 trunc from 0x00f3a6 to 0x014924 (0x557f bytes) truncating to largest ever: 0x3e054 000018 1780086571.846541 trunc from 0x014925 to 0x03e053 (0x2972f bytes) 000019 1780086571.858213 mapread 0x0252f3 thru 0x02ab3a (0x5848 bytes) 000021 1780086571.865065 read 0x01fc2b thru 0x024f12 (0x52e8 bytes) 000022 1780086571.867725 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e054): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e054): Mirror 2: 0x55831322 (stale) CRC-32 checksum value for chunk [0, 0x3e054): Mirror 1: 0x55831322 Mirror 3: 0x55831322 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e054: skipped 000023 1780086572.229203 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3e054): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e054): Mirror 2: 0x55831322 (stale) CRC-32 checksum value for chunk [0, 0x3e054): Mirror 1: 0x55831322 Mirror 3: 0x55831322 Mirror 4: 0x55831322 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e054: skipped 000025 1780086572.550541 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e054): Mirror 2: 0x55831322 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e054): Mirror 2: 0x55831322 (stale) CRC-32 checksum value for chunk [0, 0x3e054): Mirror 1: 0x55831322 Mirror 3: 0x55831322 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e054: skipped 000028 1780086572.751213 write 0x02d899 thru 0x03c6d8 (0xee40 bytes) 000029 1780086572.782972 mapwrite 0x023e12 thru 0x026ec7 (0x30b6 bytes) 000030 1780086572.815957 mapwrite 0x00dabe thru 0x00f602 (0x1b45 bytes) 000032 1780086572.848482 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e054): Mirror 3: 0x55831322 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e054): Mirror 3: 0x55831322 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e054: skipped 000033 1780086572.966900 trunc from 0x03e054 to 0x034218 (0x9e3b bytes) 000034 1780086572.973038 write 0x00529c thru 0x006735 (0x149a bytes) 000038 1780086572.979315 read 0x02f653 thru 0x034218 (0x4bc6 bytes) 000040 1780086572.984063 mapwrite 0x012f55 thru 0x01cb96 (0x9c42 bytes) 000042 1780086573.019317 write 0x004977 thru 0x00aaef (0x6179 bytes) 000044 1780086573.025556 trunc from 0x034219 to 0x015663 (0x1ebb5 bytes) 000045 1780086573.035716 read 0x013bdf thru 0x015663 (0x1a85 bytes) 000047 1780086573.036732 mapwrite 0x031cf0 thru 0x03ffff (0xe310 bytes) 000048 1780086573.075479 write 0x00fdae thru 0x01289f (0x2af2 bytes) 000052 1780086573.077944 mapwrite 0x02fd3a thru 0x03b50a (0xb7d1 bytes) 000055 1780086573.107168 MIRROR_RESYNC 2 mirrors 000058 1780086573.333253 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c7:0x0]: Invalid argument (22) 000061 1780086573.396510 mapread 0x00e07c thru 0x010adc (0x2a61 bytes) 000063 1780086573.407310 mapwrite 0x0089d8 thru 0x0106f7 (0x7d20 bytes) 000065 1780086573.451392 trunc from 0x040000 to 0x00d0f1 (0x32f0e bytes) 000066 1780086573.477857 read 0x0028bb thru 0x00b6ce (0x8e14 bytes) 000067 1780086573.478709 write 0x00007d thru 0x009ff4 (0x9f78 bytes) 000070 1780086573.484123 read 0x00455c thru 0x004db2 (0x857 bytes) 000071 1780086573.484853 mapwrite 0x035045 thru 0x03d784 (0x8740 bytes) 000072 1780086573.530525 read 0x00e115 thru 0x01d509 (0xf3f5 bytes) 000073 1780086573.531249 mapread 0x032a50 thru 0x03d784 (0xad35 bytes) 000074 1780086573.532319 read 0x00f41b thru 0x016e37 (0x7a1d bytes) 000079 1780086573.535334 trunc from 0x03d785 to 0x021feb (0x1b799 bytes) 000080 1780086573.542315 mapwrite 0x01e7c8 thru 0x02d461 (0xec9a bytes) 000081 1780086573.572141 mapread 0x01e029 thru 0x02af77 (0xcf4f bytes) 000082 1780086573.573307 mapread 0x01b1b8 thru 0x020d4e (0x5b97 bytes) 000083 1780086573.574112 trunc from 0x02d462 to 0x0217c7 (0xbc9a bytes) 000084 1780086573.581797 trunc from 0x0217c8 to 0x026002 (0x483b bytes) 000086 1780086573.595104 write 0x01ee62 thru 0x02990b (0xaaaa bytes) 000087 1780086573.596461 read 0x018e52 thru 0x02059f (0x774e bytes) 000088 1780086573.597332 mapread 0x001d5e thru 0x00b2c6 (0x9569 bytes) 000094 1780086573.601116 trunc from 0x02990c to 0x02ec1a (0x530f bytes) 000095 1780086573.609819 mapwrite 0x0232c6 thru 0x032be8 (0xf923 bytes) 000096 1780086573.649765 mapread 0x003869 thru 0x00a7aa (0x6f42 bytes) 000098 1780086573.651233 mapwrite 0x015a06 thru 0x01d699 (0x7c94 bytes) 000099 1780086573.672313 trunc from 0x032be9 to 0x00e228 (0x249c0 bytes) 000104 1780086573.682599 mapwrite 0x03ac83 thru 0x03ffff (0x537d bytes) 000110 1780086573.717388 mapwrite 0x034da0 thru 0x038f37 (0x4198 bytes) 000112 1780086573.742334 mapwrite 0x03672a thru 0x0373fb (0xcd2 bytes) 000113 1780086573.760084 mapwrite 0x007906 thru 0x015ec1 (0xe5bc bytes) 000114 1780086573.800240 mapwrite 0x039698 thru 0x039ba9 (0x512 bytes) 000116 1780086573.818279 write 0x018b42 thru 0x01a517 (0x19d6 bytes) 000118 1780086573.825235 mapread 0x009828 thru 0x00b313 (0x1aec bytes) 000119 1780086573.826213 mapwrite 0x021021 thru 0x022f3d (0x1f1d bytes) 000124 1780086573.852175 mapread 0x012947 thru 0x01ea9d (0xc157 bytes) 000126 1780086573.866478 trunc from 0x040000 to 0x019b17 (0x264e8 bytes) 000127 1780086573.873908 read 0x010f97 thru 0x019b17 (0x8b81 bytes) 000128 1780086573.874326 mapwrite 0x026c87 thru 0x02b878 (0x4bf2 bytes) 000129 1780086573.909423 mapwrite 0x018071 thru 0x0273b3 (0xf343 bytes) 000130 1780086573.932922 read 0x021248 thru 0x02b878 (0xa631 bytes) 000135 1780086573.935388 mapwrite 0x02fba6 thru 0x0363bf (0x681a bytes) 000137 1780086573.970479 trunc from 0x0363c0 to 0x00f757 (0x26c68 bytes) 000138 1780086573.978670 mapwrite 0x00cda3 thru 0x01b219 (0xe477 bytes) 000139 1780086574.029813 trunc from 0x01b21a to 0x02ed3e (0x13b25 bytes) 000141 1780086574.040026 read 0x0239e4 thru 0x02de0e (0xa42b bytes) 000142 1780086574.046307 mapwrite 0x00fb04 thru 0x00fc73 (0x170 bytes) 000144 1780086574.064589 read 0x0169e2 thru 0x025c3e (0xf25d bytes) 000150 1780086574.068756 mapread 0x01c7e0 thru 0x021106 (0x4927 bytes) 000151 1780086574.069643 mapread 0x02d54a thru 0x02ed3e (0x17f5 bytes) 000153 1780086574.070776 mapwrite 0x03ce01 thru 0x03ffff (0x31ff bytes) 000154 1780086574.104378 mapread 0x00c61f thru 0x013644 (0x7026 bytes) 000155 1780086574.105851 trunc from 0x040000 to 0x00f554 (0x30aab bytes) 000156 1780086574.115374 mapread 0x00dacb thru 0x00f554 (0x1a8a bytes) 000157 1780086574.116513 trunc from 0x00f555 to 0x034746 (0x251f2 bytes) 000158 1780086574.125963 mapwrite 0x03eb48 thru 0x03ffff (0x14b8 bytes) 000162 1780086574.162107 trunc from 0x040000 to 0x0300bd (0xff42 bytes) 000163 1780086574.171687 read 0x023a0a thru 0x028bf7 (0x51ee bytes) 000165 1780086574.179048 mapread 0x01b019 thru 0x0206df (0x56c7 bytes) 000167 1780086574.180537 read 0x01b6a3 thru 0x025e39 (0xa797 bytes) 000168 1780086574.181245 trunc from 0x0300be to 0x02826c (0x7e51 bytes) 000169 1780086574.189951 write 0x00a0dd thru 0x0198c1 (0xf7e5 bytes) 000170 1780086574.191412 mapread 0x014d4a thru 0x01e2b9 (0x9570 bytes) 000172 1780086574.193470 read 0x01cea5 thru 0x02733a (0xa496 bytes) 000173 1780086574.194194 read 0x00f9ff thru 0x01b2ed (0xb8ef bytes) 000175 1780086574.195340 trunc from 0x02826d to 0x00b65d (0x1cc0f bytes) 000178 1780086574.209343 MIRROR_EXTEND to 2 mirrors 000180 1780086574.431906 trunc from 0x00b65e to 0x00847a (0x31e3 bytes) 000181 1780086574.474547 write 0x033ef6 thru 0x034d33 (0xe3e bytes) 000182 1780086574.475670 write 0x00b45b thru 0x0192b4 (0xde5a bytes) 000187 1780086574.483766 mapwrite 0x01693d thru 0x019408 (0x2acc bytes) 000192 1780086574.504356 write 0x03e35f thru 0x03ffff (0x1ca1 bytes) 000193 1780086574.505698 mapread 0x003b7a thru 0x011363 (0xd7ea bytes) 000195 1780086574.513431 read 0x031980 thru 0x03a7b2 (0x8e33 bytes) 000196 1780086574.522823 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xd93ed8e5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x64df494c Mirror 3: 0x64df494c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000197 1780086574.883392 trunc from 0x040000 to 0x004f0d (0x3b0f2 bytes) 000198 1780086574.943379 mapwrite 0x00a3ee thru 0x014492 (0xa0a5 bytes) 000199 1780086574.982539 mapwrite 0x01c137 thru 0x021f27 (0x5df1 bytes) 000200 1780086575.020457 mapwrite 0x003fd2 thru 0x008899 (0x48c8 bytes) 000201 1780086575.044094 mapwrite 0x034e9c thru 0x03dc13 (0x8d78 bytes) 000202 1780086575.087088 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dc14): Mirror 3: 0x719ecd3c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dc14): Mirror 3: 0x719ecd3c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dc14: skipped 000203 1780086575.202696 write 0x030141 thru 0x03ebee (0xeaae bytes) 000204 1780086575.207033 mapread 0x01bd82 thru 0x01e7bf (0x2a3e bytes) 000207 1780086575.217947 MIRROR_RESYNC 2 mirrors 000213 1780086575.459449 read 0x039cbf thru 0x03ab83 (0xec5 bytes) 000214 1780086575.467444 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c7:0x0]: Invalid argument (22) 000215 1780086575.535841 mapread 0x0242c4 thru 0x027088 (0x2dc5 bytes) 000218 1780086575.546853 mapread 0x01baff thru 0x02540d (0x990f bytes) 000219 1780086575.554226 mapread 0x019bc6 thru 0x01e110 (0x454b bytes) 000221 1780086575.555643 mapread 0x031d59 thru 0x033bf8 (0x1ea0 bytes) 000224 1780086575.562604 write 0x03463a thru 0x03826f (0x3c36 bytes) 000226 1780086575.572246 mapwrite 0x0245a0 thru 0x0339ae (0xf40f bytes) 000229 1780086575.604101 read 0x0236ce thru 0x031668 (0xdf9b bytes) 000232 1780086575.607324 read 0x03d417 thru 0x03ebee (0x17d8 bytes) 000233 1780086575.613273 trunc from 0x03ebef to 0x022a31 (0x1c1bd bytes) 000234 1780086575.629833 mapread 0x01d8c9 thru 0x022a31 (0x5169 bytes) 000235 1780086575.630956 mapread 0x00ddf2 thru 0x0184fa (0xa709 bytes) 000236 1780086575.632050 mapwrite 0x0116f4 thru 0x013782 (0x208f bytes) 000239 1780086575.650450 mapread 0x01f8bf thru 0x022a31 (0x3173 bytes) 000240 1780086575.651372 mapwrite 0x02f344 thru 0x038531 (0x91ee bytes) 000244 1780086575.709748 trunc from 0x038532 to 0x02d31b (0xb216 bytes) 000245 1780086575.721166 mapread 0x013cc6 thru 0x0238d8 (0xfc13 bytes) 000247 1780086575.730139 MIRROR_EXTEND to 2 mirrors 000248 1780086575.954421 write 0x003e40 thru 0x00aa50 (0x6c11 bytes) 000249 1780086575.983554 trunc from 0x02d31c to 0x009ec0 (0x2345b bytes) 000251 1780086575.999529 write 0x036864 thru 0x03ffff (0x979c bytes) 000252 1780086576.000667 mapread 0x01164a thru 0x01455b (0x2f12 bytes) 000254 1780086576.008678 MIRROR_RESYNC 2 mirrors 000256 1780086576.228809 mapread 0x031d07 thru 0x03ffff (0xe2f9 bytes) 000257 1780086576.242537 mapwrite 0x018241 thru 0x01fbb4 (0x7974 bytes) 000259 1780086576.300285 read 0x03a6c5 thru 0x03e0bf (0x39fb bytes) 000261 1780086576.305814 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c7:0x0]: Invalid argument (22) 000262 1780086576.356226 mapwrite 0x00f7f2 thru 0x0106fc (0xf0b bytes) 000264 1780086576.384753 read 0x014666 thru 0x0206eb (0xc086 bytes) 000265 1780086576.392365 write 0x018f66 thru 0x027d81 (0xee1c bytes) 000266 1780086576.396036 mapread 0x0068d7 thru 0x0087ee (0x1f18 bytes) 000267 1780086576.402403 write 0x034ab0 thru 0x0361f2 (0x1743 bytes) 000268 1780086576.408643 read 0x026645 thru 0x02ea26 (0x83e2 bytes) 000270 1780086576.414046 write 0x03f740 thru 0x03ffff (0x8c0 bytes) truncating to largest ever: 0x3f63c 000272 1780086576.416163 trunc from 0x040000 to 0x03f63b (0x9c4 bytes) 000273 1780086576.426187 trunc from 0x03f63c to 0x037251 (0x83ea bytes) 000274 1780086576.433230 mapread 0x01def4 thru 0x02d273 (0xf380 bytes) 000276 1780086576.435484 mapread 0x02cae9 thru 0x037251 (0xa769 bytes) 000277 1780086576.436699 mapwrite 0x034de2 thru 0x03b1f4 (0x6413 bytes) 000278 1780086576.472658 write 0x035e2e thru 0x03e1b4 (0x8387 bytes) 000279 1780086576.473774 write 0x003a3d thru 0x00ac10 (0x71d4 bytes) 000281 1780086576.474995 mapread 0x025bbf thru 0x033f97 (0xe3d9 bytes) 000286 1780086576.478459 trunc from 0x03e1b5 to 0x001b97 (0x3c61d bytes) 000291 1780086576.486831 trunc from 0x001b98 to 0x0341aa (0x32613 bytes) 000295 1780086576.494847 mapwrite 0x025dbe thru 0x03176e (0xb9b1 bytes) 000296 1780086576.534704 write 0x02c975 thru 0x039331 (0xc9bd bytes) 000298 1780086576.536515 read 0x001dc5 thru 0x00be28 (0xa064 bytes) 000299 1780086576.540246 trunc from 0x039332 to 0x01aacc (0x1e865 bytes) 000301 1780086576.546662 mapread 0x00a5ab thru 0x015b87 (0xb5dd bytes) 000302 1780086576.547889 mapread 0x00d1d6 thru 0x01a050 (0xce7b bytes) 000303 1780086576.549061 read 0x00929b thru 0x011e3a (0x8ba0 bytes) 000306 1780086576.550710 trunc from 0x01aacd to 0x01295d (0x816f bytes) 000309 1780086576.557284 trunc from 0x01295e to 0x03ca29 (0x2a0cc bytes) 000310 1780086576.563329 trunc from 0x03ca2a to 0x022553 (0x1a4d6 bytes) 000311 1780086576.569304 trunc from 0x022554 to 0x03711b (0x14bc8 bytes) 000312 1780086576.576521 read 0x008541 thru 0x013251 (0xad11 bytes) 000315 1780086576.585302 MIRROR_EXTEND to 2 mirrors 000316 1780086576.803244 mapwrite 0x031fdc thru 0x0330ff (0x1124 bytes) 000318 1780086576.871276 write 0x026f45 thru 0x027f2e (0xfea bytes) 000319 1780086576.876323 read 0x00ef4a thru 0x00f6d7 (0x78e bytes) 000322 1780086576.881360 mapwrite 0x012b4f thru 0x01497b (0x1e2d bytes) 000323 1780086576.911947 trunc from 0x03711c to 0x03442e (0x2ced bytes) 000324 1780086576.934040 read 0x0146f2 thru 0x01b6ce (0x6fdd bytes) 000325 1780086576.940113 read 0x01b0e9 thru 0x02a12f (0xf047 bytes) 000326 1780086576.940843 write 0x03ac0e thru 0x03c843 (0x1c36 bytes) 000328 1780086576.942590 mapread 0x02bd61 thru 0x02e48f (0x272f bytes) 000330 1780086576.944160 mapwrite 0x02040b thru 0x0233b7 (0x2fad bytes) 000332 1780086576.969814 read 0x016283 thru 0x01fcc6 (0x9a44 bytes) 000335 1780086576.971569 mapwrite 0x021f00 thru 0x028947 (0x6a48 bytes) 000337 1780086577.001269 mapread 0x02fe83 thru 0x032451 (0x25cf bytes) 000338 1780086577.002295 mapread 0x00ca21 thru 0x01c434 (0xfa14 bytes) 000342 1780086577.005356 trunc from 0x03c844 to 0x02fed1 (0xc972 bytes) 000343 1780086577.014651 read 0x02c4ff thru 0x02fed1 (0x39d3 bytes) 000346 1780086577.016518 mapwrite 0x03eb7c thru 0x03ffff (0x1484 bytes) 000347 1780086577.052971 write 0x0088ef thru 0x00bb8c (0x329e bytes) 000349 1780086577.054552 mapwrite 0x03f988 thru 0x03ffff (0x678 bytes) 000351 1780086577.078412 mapread 0x0345dc thru 0x03a6d5 (0x60fa bytes) 000354 1780086577.083277 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c7:0x0]: Invalid argument (22) 000355 1780086577.152588 write 0x017d05 thru 0x026a1d (0xed19 bytes) 000356 1780086577.157117 mapwrite 0x004056 thru 0x011d6e (0xdd19 bytes) 000358 1780086577.187591 read 0x005117 thru 0x014136 (0xf020 bytes) 000359 1780086577.188279 mapread 0x00491a thru 0x01179a (0xce81 bytes) 000360 1780086577.189348 trunc from 0x040000 to 0x026530 (0x19acf bytes) 000361 1780086577.197292 mapread 0x01f41e thru 0x026530 (0x7113 bytes) 000364 1780086577.199310 mapwrite 0x00f0fa thru 0x0150a7 (0x5fae bytes) 000365 1780086577.220746 trunc from 0x026531 to 0x00fa31 (0x16aff bytes) 000366 1780086577.228921 read 0x00e300 thru 0x00fa31 (0x1732 bytes) 000367 1780086577.229663 trunc from 0x00fa32 to 0x0199ab (0x9f7a bytes) 000369 1780086577.238313 mapread 0x00d4f1 thru 0x010f5f (0x3a6f bytes) 000370 1780086577.243510 write 0x01065e thru 0x0204e8 (0xfe8b bytes) 000373 1780086577.247377 mapread 0x011ccf thru 0x0187e7 (0x6b19 bytes) 000375 1780086577.248961 trunc from 0x0204e9 to 0x00da56 (0x12a92 bytes) 000376 1780086577.255560 read 0x0098ea thru 0x00da56 (0x416d bytes) 000383 1780086577.259769 trunc from 0x00da57 to 0x022d63 (0x1530d bytes) 000385 1780086577.268671 read 0x00fba6 thru 0x0104e2 (0x93d bytes) 000386 1780086577.273707 trunc from 0x022d64 to 0x01412f (0xec34 bytes) 000387 1780086577.281414 read 0x0099e2 thru 0x00d0a3 (0x36c2 bytes) 000388 1780086577.282112 mapwrite 0x026cc4 thru 0x0337d0 (0xcb0d bytes) 000392 1780086577.311801 mapread 0x005a1d thru 0x00b13b (0x571f bytes) 000393 1780086577.312950 write 0x02a3a5 thru 0x02b106 (0xd62 bytes) 000395 1780086577.316846 MIRROR_EXTEND to 2 mirrors 000396 1780086577.533945 mapread 0x0164ac thru 0x019289 (0x2dde bytes) 000397 1780086577.544841 trunc from 0x0337d1 to 0x03b872 (0x80a2 bytes) 000398 1780086577.594293 mapread 0x004222 thru 0x005b87 (0x1966 bytes) 000399 1780086577.602997 MIRROR_RESYNC 2 mirrors 000400 1780086577.860488 mapwrite 0x0163b3 thru 0x023d39 (0xd987 bytes) 000404 1780086577.955745 trunc from 0x03b873 to 0x03660e (0x5264 bytes) 000405 1780086577.975460 MIRROR_RESYNC 2 mirrors 000407 1780086578.272226 trunc from 0x03660f to 0x01a3f1 (0x1c21d bytes) 000408 1780086578.328487 read 0x016b62 thru 0x01a3f1 (0x3890 bytes) 000410 1780086578.335777 write 0x013400 thru 0x018c7d (0x587e bytes) 000411 1780086578.339092 write 0x00cb55 thru 0x010170 (0x361c bytes) 000414 1780086578.351782 mapread 0x011d60 thru 0x01801f (0x62c0 bytes) 000415 1780086578.355937 mapread 0x00b378 thru 0x012ad6 (0x775f bytes) 000416 1780086578.356969 mapwrite 0x01d7f3 thru 0x0245bf (0x6dcd bytes) 000419 1780086578.399035 read 0x015829 thru 0x01deaf (0x8687 bytes) 000420 1780086578.399697 mapwrite 0x018948 thru 0x01f9ec (0x70a5 bytes) 000422 1780086578.424471 trunc from 0x0245c0 to 0x00f529 (0x15096 bytes) 000423 1780086578.437238 MIRROR_RESYNC 2 mirrors 000424 1780086578.674557 read 0x002abb thru 0x00f529 (0xca6f bytes) 000427 1780086578.680981 mapwrite 0x001592 thru 0x00ab68 (0x95d7 bytes) 000428 1780086578.763313 trunc from 0x00f52a to 0x0233ef (0x13ec6 bytes) 000429 1780086578.782976 mapwrite 0x01d767 thru 0x02abad (0xd447 bytes) 000430 1780086578.823312 trunc from 0x02abae to 0x00dfdc (0x1cbd1 bytes) 000431 1780086578.835847 mapwrite 0x03f785 thru 0x03ffff (0x87b bytes) 000432 1780086578.872452 trunc from 0x040000 to 0x0297a5 (0x1685a bytes) 000433 1780086578.881021 write 0x017fb3 thru 0x01d63a (0x5688 bytes) 000435 1780086578.887822 mapread 0x01b86b thru 0x01cd4b (0x14e1 bytes) 000436 1780086578.888855 trunc from 0x0297a6 to 0x00314f (0x26656 bytes) 000437 1780086578.899797 read 0x00170d thru 0x00314f (0x1a43 bytes) 000438 1780086578.900499 trunc from 0x003150 to 0x018fc3 (0x15e74 bytes) 000440 1780086578.909978 trunc from 0x018fc4 to 0x0010d0 (0x17ef3 bytes) 000442 1780086578.918033 trunc from 0x0010d1 to 0x013b3b (0x12a6b bytes) 000443 1780086578.926394 write 0x00d258 thru 0x00fab1 (0x285a bytes) 000444 1780086578.930716 write 0x0097a2 thru 0x00fd9e (0x65fd bytes) 000445 1780086578.931739 write 0x025aea thru 0x0279c6 (0x1edd bytes) 000446 1780086578.932642 mapread 0x0263c0 thru 0x0279c6 (0x1607 bytes) 000447 1780086578.933593 mapread 0x023071 thru 0x0279c6 (0x4956 bytes) 000448 1780086578.938493 trunc from 0x0279c7 to 0x027dfa (0x434 bytes) 000449 1780086578.949711 mapread 0x01f300 thru 0x023bb6 (0x48b7 bytes) 000450 1780086578.950787 read 0x01e8cb thru 0x027cf2 (0x9428 bytes) 000451 1780086578.951462 mapread 0x001e15 thru 0x00e1d5 (0xc3c1 bytes) 000452 1780086578.952735 read 0x0229dc thru 0x027dfa (0x541f bytes) 000454 1780086578.953907 read 0x014d9b thru 0x021fd2 (0xd238 bytes) 000457 1780086578.955337 trunc from 0x027dfb to 0x02ac2a (0x2e30 bytes) 000458 1780086578.963138 write 0x01a61f thru 0x021c66 (0x7648 bytes) 000459 1780086578.964162 trunc from 0x02ac2b to 0x002539 (0x286f1 bytes) 000461 1780086578.974108 trunc from 0x00253a to 0x007f77 (0x5a3e bytes) 000463 1780086578.983366 trunc from 0x007f78 to 0x00bcfe (0x3d87 bytes) 000464 1780086578.992234 trunc from 0x00bcff to 0x03ed00 (0x33002 bytes) 000466 1780086579.002684 mapwrite 0x00d9a2 thru 0x0149fa (0x7059 bytes) 000467 1780086579.033295 mapread 0x03afc5 thru 0x03ed00 (0x3d3c bytes) 000470 1780086579.035533 read 0x023c3c thru 0x033719 (0xfade bytes) 000473 1780086579.037468 mapwrite 0x0151db thru 0x0201bc (0xafe2 bytes) 000477 1780086579.064841 write 0x0027aa thru 0x00f53b (0xcd92 bytes) 000478 1780086579.068575 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c7:0x0]: Invalid argument (22) 000480 1780086579.156640 mapread 0x03a167 thru 0x03ed00 (0x4b9a bytes) 000481 1780086579.167981 trunc from 0x03ed01 to 0x005f37 (0x38dc9 bytes) 000482 1780086579.177039 mapwrite 0x006007 thru 0x013308 (0xd302 bytes) 000486 1780086579.230667 trunc from 0x013309 to 0x0116d5 (0x1c33 bytes) 000488 1780086579.238292 mapwrite 0x00fca2 thru 0x01bc2f (0xbf8e bytes) 000490 1780086579.285132 trunc from 0x01bc30 to 0x0369c0 (0x1ad91 bytes) 000492 1780086579.298513 read 0x01fdd5 thru 0x01ffc5 (0x1f1 bytes) 000494 1780086579.302742 read 0x0090ac thru 0x00e3a6 (0x52fb bytes) 000495 1780086579.303771 read 0x035ece thru 0x0369c0 (0xaf3 bytes) 000496 1780086579.309986 write 0x01ed56 thru 0x02c14b (0xd3f6 bytes) 000497 1780086579.311989 trunc from 0x0369c1 to 0x00f89b (0x27125 bytes) 000503 1780086579.325570 mapwrite 0x03347d thru 0x03f724 (0xc2a8 bytes) 000504 1780086579.376468 mapread 0x010fbb thru 0x01ccc3 (0xbd09 bytes) 000506 1780086579.411327 mapread 0x020865 thru 0x024718 (0x3eb4 bytes) 000507 1780086579.426117 mapwrite 0x03d4a9 thru 0x03ffff (0x2b57 bytes) 000508 1780086579.457637 mapwrite 0x01331e thru 0x01c39c (0x907f bytes) 000509 1780086579.484982 write 0x024c52 thru 0x0318aa (0xcc59 bytes) 000511 1780086579.488902 trunc from 0x040000 to 0x022b4f (0x1d4b0 bytes) 000512 1780086579.501760 MIRROR_EXTEND to 2 mirrors 000513 1780086579.786294 write 0x03cbdd thru 0x03ce5e (0x282 bytes) 000514 1780086579.830886 trunc from 0x03ce5f to 0x0189f3 (0x2446b bytes) 000515 1780086579.856374 trunc from 0x0189f4 to 0x0377d1 (0x1edde bytes) 000516 1780086579.872757 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x377d2): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x377d2): Mirror 2: 0xaffba7d (stale) CRC-32 checksum value for chunk [0, 0x377d2): Mirror 1: 0xda30a203 Mirror 3: 0xda30a203 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x377d2: skipped 000517 1780086580.259361 read 0x02f0cb thru 0x0305bd (0x14f3 bytes) 000518 1780086580.265216 write 0x02b721 thru 0x033725 (0x8005 bytes) 000522 1780086580.302550 MIRROR_RESYNC 3 mirrors 000523 1780086580.695844 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c7:0x0]: Invalid argument (22) 000524 1780086580.837467 trunc from 0x0377d2 to 0x01d594 (0x1a23d bytes) 000525 1780086580.877367 trunc from 0x01d595 to 0x026422 (0x8e8e bytes) 000526 1780086580.889905 MIRROR_RESYNC 2 mirrors 000528 1780086581.213017 trunc from 0x026423 to 0x003e92 (0x22590 bytes) 000530 1780086581.254347 MIRROR_RESYNC 2 mirrors 000531 1780086581.551103 mapread 0x001ee3 thru 0x003e92 (0x1fb0 bytes) 000532 1780086581.560279 trunc from 0x003e93 to 0x03cee4 (0x39052 bytes) 000533 1780086581.604215 read 0x017a41 thru 0x01ff0c (0x84cc bytes) 000535 1780086581.612455 read 0x01f7bc thru 0x028bf1 (0x9436 bytes) 000536 1780086581.621269 trunc from 0x03cee5 to 0x00cea2 (0x30042 bytes) 000537 1780086581.636618 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c7:0x0]: Invalid argument (22) 000542 1780086581.721791 write 0x02a89b thru 0x031ffe (0x7764 bytes) 000544 1780086581.723495 write 0x0271c1 thru 0x02e970 (0x77b0 bytes) 000549 1780086581.728914 mapwrite 0x00ae52 thru 0x00d700 (0x28af bytes) 000550 1780086581.760399 trunc from 0x031fff to 0x03838b (0x638d bytes) 000554 1780086581.769556 write 0x00c425 thru 0x012f63 (0x6b3f bytes) 000555 1780086581.770473 trunc from 0x03838c to 0x02d4e7 (0xaea4 bytes) 000557 1780086581.779793 trunc from 0x02d4e8 to 0x0265c4 (0x6f23 bytes) 000558 1780086581.787740 mapwrite 0x02edb4 thru 0x02ee40 (0x8d bytes) 000559 1780086581.834710 read 0x00e3e4 thru 0x01500e (0x6c2b bytes) 000561 1780086581.840469 read 0x016265 thru 0x01ed02 (0x8a9e bytes) 000562 1780086581.846725 mapwrite 0x0211c3 thru 0x02d2bc (0xc0fa bytes) 000564 1780086581.880976 write 0x01c163 thru 0x020f79 (0x4e17 bytes) 000566 1780086581.882532 write 0x015ef4 thru 0x02352d (0xd63a bytes) 000567 1780086581.883896 read 0x014ea6 thru 0x022f0b (0xe066 bytes) 000574 1780086581.888223 trunc from 0x02ee41 to 0x0119e9 (0x1d457 bytes) 000575 1780086581.897529 trunc from 0x0119ea to 0x036beb (0x25202 bytes) 000577 1780086581.906317 trunc from 0x036bec to 0x001eab (0x34d40 bytes) 000580 1780086581.916449 mapwrite 0x0084ac thru 0x00d450 (0x4fa5 bytes) 000589 1780086581.962499 mapwrite 0x012b25 thru 0x0228e3 (0xfdbf bytes) 000590 1780086582.015695 mapread 0x0182dd thru 0x01d63e (0x5362 bytes) 000591 1780086582.019187 MIRROR_EXTEND to 2 mirrors 000593 1780086582.321151 mapwrite 0x03ead6 thru 0x03ffff (0x152a bytes) 000596 1780086582.392390 write 0x0010e7 thru 0x00f6bb (0xe5d5 bytes) 000598 1780086582.400387 mapwrite 0x01c286 thru 0x027922 (0xb69d bytes) 000599 1780086582.438348 mapread 0x001c0d thru 0x00354f (0x1943 bytes) 000600 1780086582.439612 trunc from 0x040000 to 0x03a359 (0x5ca6 bytes) 000601 1780086582.451697 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3a35a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a35a): Mirror 2: 0x2e5bfe61 (stale) CRC-32 checksum value for chunk [0, 0x3a35a): Mirror 1: 0xd67737ae Mirror 3: 0xd67737ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a35a: skipped 000602 1780086582.769035 mapwrite 0x00929c thru 0x015902 (0xc667 bytes) 000603 1780086582.871852 mapwrite 0x009eb7 thru 0x014d82 (0xaecc bytes) 000604 1780086582.899785 mapread 0x015d3a thru 0x0223eb (0xc6b2 bytes) 000605 1780086582.907545 mapread 0x014496 thru 0x019414 (0x4f7f bytes) 000607 1780086582.909069 read 0x02d701 thru 0x03a359 (0xcc59 bytes) 000610 1780086582.915849 write 0x01cdec thru 0x02bfbd (0xf1d2 bytes) 000613 1780086582.925445 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a35a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a35a): Mirror 2: 0x2e5bfe61 (stale) Mirror 3: 0xd67737ae (stale) CRC-32 checksum value for chunk [0, 0x3a35a): Mirror 1: 0xd9d56047 Mirror 4: 0xd9d56047 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a35a: skipped 000614 1780086583.311392 read 0x02deff thru 0x038421 (0xa523 bytes) 000615 1780086583.323561 trunc from 0x03a35a to 0x010787 (0x29bd2 bytes) 000623 1780086583.378192 read 0x00f4be thru 0x010787 (0x12ca bytes) 000624 1780086583.389245 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10788): Mirror 2: 0x30cbb157 (stale) Mirror 4: 0x265f0e04 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10788): Mirror 2: 0x30cbb157 (stale) Mirror 4: 0x265f0e04 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10788: skipped 000625 1780086583.602858 read 0x00221e thru 0x0066b4 (0x4497 bytes) 000627 1780086583.611481 read 0x005ab2 thru 0x00c465 (0x69b4 bytes) 000630 1780086583.622785 trunc from 0x010788 to 0x032936 (0x221af bytes) 000631 1780086583.640123 MIRROR_RESYNC 3 mirrors 000634 1780086584.083161 read 0x024f59 thru 0x02c40b (0x74b3 bytes) 000636 1780086584.089559 mapread 0x00af5b thru 0x0180fa (0xd1a0 bytes) 000637 1780086584.108726 write 0x020aca thru 0x02f16e (0xe6a5 bytes) 000638 1780086584.154308 mapread 0x00dfb0 thru 0x01ca1f (0xea70 bytes) 000639 1780086584.164451 read 0x02279c thru 0x027680 (0x4ee5 bytes) 000641 1780086584.165615 read 0x01d33e thru 0x028dcb (0xba8e bytes) 000642 1780086584.166230 trunc from 0x032937 to 0x000d83 (0x31bb3 bytes) 000643 1780086584.196039 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd84): Mirror 2: 0x45ff80a5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd84): Mirror 2: 0x45ff80a5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd84: skipped 000644 1780086584.360408 write 0x00e6f9 thru 0x016c64 (0x856c bytes) 000645 1780086584.362161 write 0x013fb9 thru 0x0164df (0x2527 bytes) 000647 1780086584.363710 read 0x003b52 thru 0x00ba2b (0x7eda bytes) 000648 1780086584.373529 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c7:0x0]: Invalid argument (22) 000649 1780086584.465407 mapwrite 0x03233c thru 0x032409 (0xce bytes) 000650 1780086584.500447 MIRROR_EXTEND to 2 mirrors 000651 1780086584.798533 read 0x0009a6 thru 0x00b1fe (0xa859 bytes) 000652 1780086584.808424 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c7:0x0]: Invalid argument (22) 000653 1780086584.890209 read 0x01e99d thru 0x02586f (0x6ed3 bytes) 000654 1780086584.901525 read 0x01660e thru 0x02296c (0xc35f bytes) 000655 1780086584.912564 write 0x03ca6a thru 0x03ffff (0x3596 bytes) 000656 1780086584.923299 mapwrite 0x02e581 thru 0x038d4a (0xa7ca bytes) 000658 1780086584.976583 mapwrite 0x01f766 thru 0x02282b (0x30c6 bytes) 000659 1780086585.011661 read 0x016e6b thru 0x0183a9 (0x153f bytes) 000660 1780086585.021342 read 0x0227ee thru 0x02bf2a (0x973d bytes) 000661 1780086585.022087 mapread 0x03f8f9 thru 0x03ffff (0x707 bytes) 000662 1780086585.025988 MIRROR_EXTEND to 2 mirrors 000666 1780086585.382109 MIRROR_RESYNC 2 mirrors 000667 1780086585.543758 read 0x007a21 thru 0x01709c (0xf67c bytes) 000669 1780086585.550925 mapwrite 0x0132fc thru 0x016d0d (0x3a12 bytes) 000674 1780086585.636707 mapwrite 0x014bf1 thru 0x01f9db (0xadeb bytes) 000675 1780086585.670144 mapread 0x03efb2 thru 0x03ffff (0x104e bytes) 000676 1780086585.684382 mapwrite 0x015db8 thru 0x01626d (0x4b6 bytes) 000677 1780086585.711985 mapread 0x035f63 thru 0x03ffff (0xa09d bytes) 000678 1780086585.723531 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x54a60edf (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6607903c Mirror 3: 0x6607903c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000679 1780086586.055735 read 0x01222b thru 0x01468c (0x2462 bytes) 000681 1780086586.065963 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x54a60edf (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6607903c Mirror 3: 0x6607903c Mirror 4: 0x6607903c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000684 1780086586.446388 read 0x03b174 thru 0x03cb74 (0x1a01 bytes) 000685 1780086586.449732 trunc from 0x040000 to 0x01c6a1 (0x2395e bytes) 000691 1780086586.512882 read 0x00f69f thru 0x0193a0 (0x9d02 bytes) 000692 1780086586.521525 trunc from 0x01c6a2 to 0x02a995 (0xe2f4 bytes) 000693 1780086586.546086 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a996): Mirror 2: 0x38c6e24d (stale) Mirror 4: 0xcc9c18a6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a996): Mirror 2: 0x38c6e24d (stale) Mirror 4: 0xcc9c18a6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a996: skipped 000694 1780086586.734781 mapwrite 0x01d231 thru 0x021628 (0x43f8 bytes) 000696 1780086586.790509 read 0x01491f thru 0x01b5ea (0x6ccc bytes) 000697 1780086586.795194 mapwrite 0x0382ed thru 0x03ffff (0x7d13 bytes) 000698 1780086586.842540 write 0x01468b thru 0x023bfc (0xf572 bytes) 000699 1780086586.845042 mapread 0x02bf87 thru 0x031b25 (0x5b9f bytes) 000700 1780086586.854617 write 0x02e7aa thru 0x0342c5 (0x5b1c bytes) 000704 1780086586.858704 mapread 0x0133d2 thru 0x014b8d (0x17bc bytes) 000708 1780086586.869120 write 0x0217bf thru 0x02258c (0xdce bytes) 000709 1780086586.870370 trunc from 0x040000 to 0x03e0a9 (0x1f56 bytes) 000710 1780086586.885967 mapread 0x020ec3 thru 0x02a34c (0x948a bytes) 000711 1780086586.887375 trunc from 0x03e0aa to 0x004234 (0x39e75 bytes) 000712 1780086586.897752 write 0x01f6b6 thru 0x02ddfc (0xe747 bytes) 000715 1780086586.900583 mapread 0x00d1ff thru 0x00e563 (0x1365 bytes) 000716 1780086586.906585 mapread 0x02aa9a thru 0x02c2ff (0x1866 bytes) 000718 1780086586.908414 read 0x024c26 thru 0x02ddfc (0x91d7 bytes) 000719 1780086586.909166 mapread 0x008a9f thru 0x01710c (0xe66e bytes) 000720 1780086586.918174 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2ddfd): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ddfd): Mirror 2: 0x1a954bcc (stale) Mirror 4: 0x2a4d5eff (stale) CRC-32 checksum value for chunk [0, 0x2ddfd): Mirror 1: 0x83ee377 Mirror 5: 0x83ee377 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ddfd: skipped 000721 1780086587.407735 mapwrite 0x00ffd7 thru 0x01dab2 (0xdadc bytes) 000722 1780086587.527212 mapwrite 0x03187c thru 0x03aaa4 (0x9229 bytes) 000725 1780086587.583454 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3aaa5): Mirror 4: 0x30b0fe76 (stale) Mirror 5: 0xc08ecb72 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aaa5): Mirror 4: 0x30b0fe76 (stale) Mirror 5: 0xc08ecb72 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aaa5: skipped 000726 1780086587.778911 read 0x01238c thru 0x0154d8 (0x314d bytes) 000730 1780086587.786974 write 0x001a30 thru 0x009830 (0x7e01 bytes) 000733 1780086587.795641 write 0x009fde thru 0x00ebf0 (0x4c13 bytes) 000734 1780086587.799802 read 0x01dab5 thru 0x01f276 (0x17c2 bytes) 000737 1780086587.808833 mapwrite 0x02ad31 thru 0x03497d (0x9c4d bytes) 000740 1780086587.845129 read 0x017b03 thru 0x01da57 (0x5f55 bytes) 000741 1780086587.845807 read 0x023e25 thru 0x027c71 (0x3e4d bytes) 000744 1780086587.847529 mapread 0x001af7 thru 0x002329 (0x833 bytes) 000746 1780086587.849061 trunc from 0x03aaa5 to 0x03d991 (0x2eed bytes) 000748 1780086587.860238 read 0x0150ba thru 0x015f35 (0xe7c bytes) 000749 1780086587.860980 mapwrite 0x035769 thru 0x03ffff (0xa897 bytes) 000750 1780086587.900779 read 0x01aba0 thru 0x026f06 (0xc367 bytes) 000752 1780086587.901977 read 0x015165 thru 0x01bd07 (0x6ba3 bytes) 000755 1780086587.903731 trunc from 0x040000 to 0x03208f (0xdf70 bytes) 000756 1780086587.911891 trunc from 0x032090 to 0x01e177 (0x13f18 bytes) 000757 1780086587.924394 mapwrite 0x00c744 thru 0x0121cf (0x5a8c bytes) 000760 1780086587.957767 mapwrite 0x0268cb thru 0x033130 (0xc866 bytes) 000761 1780086588.017733 mapwrite 0x033d03 thru 0x039073 (0x5371 bytes) 000763 1780086588.064516 write 0x009ddb thru 0x00c91f (0x2b45 bytes) 000766 1780086588.069223 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39074): Mirror 4: 0x85924675 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39074): Mirror 4: 0x85924675 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39074: skipped 000768 1780086588.266007 write 0x02d7a6 thru 0x0381b3 (0xaa0e bytes) 000769 1780086588.272489 mapread 0x024aba thru 0x033e6b (0xf3b2 bytes) 000771 1780086588.278915 read 0x02f110 thru 0x034e7b (0x5d6c bytes) 000774 1780086588.281285 read 0x025434 thru 0x02eee7 (0x9ab4 bytes) 000775 1780086588.285474 MIRROR_RESYNC 2 mirrors 000777 1780086588.584785 write 0x03d0d1 thru 0x03ffff (0x2f2f bytes) 000780 1780086588.619300 trunc from 0x040000 to 0x01d255 (0x22daa bytes) 000781 1780086588.644801 write 0x009675 thru 0x010df5 (0x7781 bytes) 000782 1780086588.657429 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c7:0x0]: Invalid argument (22) 000784 1780086588.764844 mapread 0x006f80 thru 0x00e24b (0x72cc bytes) 000786 1780086588.783354 read 0x00afc9 thru 0x013d50 (0x8d88 bytes) 000787 1780086588.788785 mapwrite 0x032554 thru 0x034a80 (0x252d bytes) 000788 1780086588.841309 read 0x02b593 thru 0x02c0c3 (0xb31 bytes) 000791 1780086588.843156 mapread 0x01816f thru 0x0211e6 (0x9078 bytes) 000792 1780086588.852379 mapwrite 0x01b30e thru 0x02553c (0xa22f bytes) 000793 1780086588.893521 trunc from 0x034a81 to 0x00816d (0x2c913 bytes) 000795 1780086588.906650 trunc from 0x00816e to 0x01d5cb (0x1545e bytes) 000796 1780086588.914839 read 0x0142a3 thru 0x01ac02 (0x6960 bytes) 000799 1780086588.929215 write 0x002e17 thru 0x00f8e7 (0xcad1 bytes) 000800 1780086588.932447 mapread 0x008f2a thru 0x015bd4 (0xccab bytes) 000802 1780086588.935268 trunc from 0x01d5cc to 0x0270be (0x9af3 bytes) 000803 1780086588.951550 mapread 0x00275f thru 0x0046f9 (0x1f9b bytes) 000804 1780086588.955431 MIRROR_EXTEND to 2 mirrors 000805 1780086589.326520 mapwrite 0x02671d thru 0x02e82e (0x8112 bytes) 000806 1780086589.411370 mapread 0x0290bd thru 0x02d2ae (0x41f2 bytes) 000808 1780086589.413203 write 0x0230c7 thru 0x02dbb4 (0xaaee bytes) 000809 1780086589.414684 write 0x02cb99 thru 0x03bf02 (0xf36a bytes) 000810 1780086589.416901 trunc from 0x03bf03 to 0x03b20d (0xcf5 bytes) 000811 1780086589.428936 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b20e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b20e): Mirror 2: 0xf374cf2a (stale) CRC-32 checksum value for chunk [0, 0x3b20e): Mirror 1: 0xf46ecb43 Mirror 3: 0xf46ecb43 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b20e: skipped 000812 1780086589.898889 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b20e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b20e): Mirror 2: 0xf374cf2a (stale) CRC-32 checksum value for chunk [0, 0x3b20e): Mirror 1: 0xf46ecb43 Mirror 3: 0xf46ecb43 Mirror 4: 0xf46ecb43 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b20e: skipped 000814 1780086590.374120 write 0x02dac0 thru 0x034b5c (0x709d bytes) 000815 1780086590.442583 MIRROR_RESYNC 4 mirrors 000816 1780086590.969550 trunc from 0x03b20e to 0x00f2d5 (0x2bf38 bytes) 000817 1780086591.014812 mapread 0x001386 thru 0x00dfae (0xcc29 bytes) 000818 1780086591.039090 read 0x00b770 thru 0x00f2d5 (0x3b66 bytes) 000820 1780086591.040261 mapread 0x005636 thru 0x00f2d5 (0x9ca0 bytes) 000824 1780086591.046857 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xf2d6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf2d6): Mirror 2: 0xe72aa323 (stale) Mirror 3: 0xe72aa323 (stale) Mirror 4: 0xe72aa323 (stale) CRC-32 checksum value for chunk [0, 0xf2d6): Mirror 1: 0xe72aa323 Mirror 5: 0xe72aa323 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf2d6: skipped 000826 1780086591.608369 write 0x0283d4 thru 0x037135 (0xed62 bytes) 000828 1780086591.651002 trunc from 0x037136 to 0x0152e1 (0x21e54 bytes) 000833 1780086591.669225 write 0x023496 thru 0x02b241 (0x7dac bytes) 000837 1780086591.676948 MIRROR_RESYNC 5 mirrors 000838 1780086592.242427 trunc from 0x02b242 to 0x031b87 (0x6946 bytes) 000840 1780086592.296336 mapwrite 0x02568c thru 0x03383d (0xe1b2 bytes) 000841 1780086592.374841 trunc from 0x03383e to 0x02df66 (0x58d7 bytes) 000842 1780086592.387730 mapwrite 0x0319f8 thru 0x03a0a0 (0x86a9 bytes) 000843 1780086592.450281 mapwrite 0x02c84a thru 0x02feaf (0x3666 bytes) 000844 1780086592.473427 MIRROR_RESYNC 5 mirrors 000846 1780086592.919421 write 0x0264f4 thru 0x035218 (0xed25 bytes) 000847 1780086592.959347 write 0x011c49 thru 0x0192cd (0x7685 bytes) 000848 1780086592.964210 write 0x011963 thru 0x021485 (0xfb23 bytes) 000849 1780086592.971188 trunc from 0x03a0a1 to 0x028fcf (0x110d1 bytes) 000850 1780086593.004040 mapwrite 0x0092a5 thru 0x00b1b4 (0x1f10 bytes) 000851 1780086593.053171 MIRROR_RESYNC 5 mirrors 000852 1780086593.555287 MIRROR_EXTEND to 6 mirrors 000857 1780086593.897806 mapwrite 0x01a520 thru 0x01bb07 (0x15e8 bytes) 000859 1780086594.038537 trunc from 0x028fd0 to 0x031a2b (0x8a5c bytes) 000860 1780086594.060521 trunc from 0x031a2c to 0x000a71 (0x30fba bytes) 000861 1780086594.082024 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xa72): Mirror 2: 0xec910481 (stale) Mirror 4: 0xec910481 (stale) Mirror 5: 0xec910481 (stale) Mirror 6: 0xec910481 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa72): Mirror 2: 0xec910481 (stale) Mirror 4: 0xec910481 (stale) Mirror 5: 0xec910481 (stale) Mirror 6: 0xec910481 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa72: skipped 000862 1780086594.303392 trunc from 0x000a72 to 0x00b40e (0xa99d bytes) 000863 1780086594.317782 trunc from 0x00b40f to 0x01456f (0x9161 bytes) 000866 1780086594.334660 mapread 0x01389f thru 0x01456f (0xcd1 bytes) 000867 1780086594.345420 mapread 0x00993c thru 0x00e5bd (0x4c82 bytes) 000868 1780086594.353520 trunc from 0x014570 to 0x016407 (0x1e98 bytes) 000871 1780086594.370737 mapread 0x015681 thru 0x016407 (0xd87 bytes) 000872 1780086594.379539 trunc from 0x016408 to 0x03be7b (0x25a74 bytes) 000873 1780086594.393295 MIRROR_RESYNC 5 mirrors 000876 1780086594.878774 mapwrite 0x00e91d thru 0x014878 (0x5f5c bytes) 000877 1780086594.973402 trunc from 0x03be7c to 0x02fad2 (0xc3a9 bytes) 000878 1780086594.991358 write 0x0259f4 thru 0x027c47 (0x2254 bytes) 000881 1780086595.008947 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2fad3): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fad3): Mirror 2: 0x3b477d4a (stale) Mirror 4: 0x3b477d4a (stale) Mirror 5: 0x3b477d4a (stale) Mirror 6: 0x3b477d4a (stale) CRC-32 checksum value for chunk [0, 0x2fad3): Mirror 1: 0xfcef0bd0 Mirror 7: 0xfcef0bd0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fad3: skipped 000882 1780086595.509785 mapread 0x01f85d thru 0x024482 (0x4c26 bytes) 000885 1780086595.541123 trunc from 0x02fad3 to 0x009d2b (0x25da7 bytes) 000886 1780086595.574842 mapwrite 0x019bdb thru 0x02638a (0xc7b0 bytes) 000888 1780086595.647773 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2638b): Mirror 2: 0xa358e617 (stale) Mirror 4: 0xa358e617 (stale) Mirror 6: 0xa358e617 (stale) Mirror 7: 0xec0d8fb1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2638b): Mirror 2: 0xa358e617 (stale) Mirror 4: 0xa358e617 (stale) Mirror 6: 0xa358e617 (stale) Mirror 7: 0xec0d8fb1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2638b: skipped 000891 1780086595.853601 write 0x02129a thru 0x02e2c6 (0xd02d bytes) 000892 1780086595.858496 trunc from 0x02e2c7 to 0x001f56 (0x2c370 bytes) 000893 1780086595.868827 read 0x0018c7 thru 0x001f56 (0x690 bytes) 000894 1780086595.872076 mapread 0x00040d thru 0x001f56 (0x1b4a bytes) 000895 1780086595.876617 trunc from 0x001f57 to 0x01f27c (0x1d326 bytes) 000896 1780086595.885542 mapread 0x015509 thru 0x01dc31 (0x8729 bytes) 000897 1780086595.897352 MIRROR_RESYNC 5 mirrors 000899 1780086596.351347 read 0x00c796 thru 0x019b79 (0xd3e4 bytes) 000902 1780086596.360159 MIRROR_RESYNC 5 mirrors 000903 1780086596.594845 MIRROR_EXTEND to 6 mirrors 000904 1780086596.926202 mapread 0x00193c thru 0x001d67 (0x42c bytes) 000907 1780086596.938677 MIRROR_RESYNC 6 mirrors 000909 1780086597.185988 MIRROR_EXTEND to 7 mirrors 000910 1780086597.546068 mapwrite 0x01ed72 thru 0x020a42 (0x1cd1 bytes) 000911 1780086597.633709 mapread 0x018148 thru 0x020a42 (0x88fb bytes) 000912 1780086597.640198 mapwrite 0x022712 thru 0x0254d6 (0x2dc5 bytes) 000914 1780086597.686362 mapread 0x003a01 thru 0x0099ba (0x5fba bytes) 000915 1780086597.692353 mapread 0x0253d1 thru 0x0254d6 (0x106 bytes) 000916 1780086597.693334 read 0x01ffa3 thru 0x020bdb (0xc39 bytes) 000919 1780086597.696334 mapread 0x020690 thru 0x0254d6 (0x4e47 bytes) 000920 1780086597.697488 mapwrite 0x027bea thru 0x02bdbc (0x41d3 bytes) 000922 1780086597.740786 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2bdbd): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bdbd): Mirror 2: 0xda77df95 (stale) Mirror 4: 0xda77df95 (stale) Mirror 6: 0xda77df95 (stale) Mirror 7: 0xda77df95 (stale) Mirror 8: 0xda77df95 (stale) Mirror 9: 0xda77df95 (stale) CRC-32 checksum value for chunk [0, 0x2bdbd): Mirror 1: 0x9b07b295 Mirror 10: 0x9b07b295 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bdbd: skipped 000923 1780086598.167529 trunc from 0x02bdbd to 0x027b13 (0x42a9 bytes) 000926 1780086598.211697 write 0x0070ed thru 0x014b2d (0xda41 bytes) 000927 1780086598.218361 mapwrite 0x00aba6 thru 0x013b13 (0x8f6e bytes) 000928 1780086598.251133 MIRROR_RESYNC 8 mirrors 000929 1780086598.799040 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c7:0x0]: Invalid argument (22) 000931 1780086598.956548 trunc from 0x027b14 to 0x013826 (0x142ed bytes) 000934 1780086598.995069 mapread 0x007f97 thru 0x00fb62 (0x7bcc bytes) 000935 1780086599.008746 mapwrite 0x01eec6 thru 0x02002a (0x1165 bytes) 000937 1780086599.051370 read 0x016e6d thru 0x01b9e5 (0x4b79 bytes) 000938 1780086599.055351 mapwrite 0x00c3e1 thru 0x012199 (0x5db9 bytes) 000940 1780086599.090165 MIRROR_RESYNC 7 mirrors 000941 1780086599.539910 mapwrite 0x037641 thru 0x037bcd (0x58d bytes) 000942 1780086599.631744 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37bce): Mirror 4: 0x5378f7ad (stale) Mirror 6: 0x5378f7ad (stale) Mirror 7: 0x5378f7ad (stale) Mirror 8: 0x5378f7ad (stale) Mirror 9: 0x5378f7ad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37bce): Mirror 4: 0x5378f7ad (stale) Mirror 6: 0x5378f7ad (stale) Mirror 7: 0x5378f7ad (stale) Mirror 8: 0x5378f7ad (stale) Mirror 9: 0x5378f7ad (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37bce: skipped 000943 1780086599.854728 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x37bce): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37bce): Mirror 4: 0x5378f7ad (stale) Mirror 6: 0x5378f7ad (stale) Mirror 7: 0x5378f7ad (stale) Mirror 8: 0x5378f7ad (stale) Mirror 9: 0x5378f7ad (stale) CRC-32 checksum value for chunk [0, 0x37bce): Mirror 1: 0x96a1835e Mirror 10: 0x96a1835e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37bce: skipped 000944 1780086600.347778 mapread 0x031813 thru 0x037bcd (0x63bb bytes) 000945 1780086600.371355 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x37bce): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37bce): Mirror 4: 0x5378f7ad (stale) Mirror 6: 0x5378f7ad (stale) Mirror 7: 0x5378f7ad (stale) Mirror 8: 0x5378f7ad (stale) Mirror 9: 0x5378f7ad (stale) CRC-32 checksum value for chunk [0, 0x37bce): Mirror 1: 0x96a1835e Mirror 10: 0x96a1835e Mirror 11: 0x96a1835e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37bce: skipped 000946 1780086600.896291 mapwrite 0x03bae1 thru 0x03d03a (0x155a bytes) 000947 1780086601.005194 write 0x009da8 thru 0x015ac6 (0xbd1f bytes) 000948 1780086601.016710 mapread 0x023570 thru 0x0244db (0xf6c bytes) 000950 1780086601.029923 mapwrite 0x032458 thru 0x032801 (0x3aa bytes) 000951 1780086601.069191 mapread 0x01c56f thru 0x024624 (0x80b6 bytes) 000952 1780086601.070691 write 0x02e95b thru 0x037fcc (0x9672 bytes) 000953 1780086601.072246 mapread 0x0383f7 thru 0x03d03a (0x4c44 bytes) 000955 1780086601.074551 write 0x02a3ab thru 0x02c316 (0x1f6c bytes) 000956 1780086601.082502 MIRROR_RESYNC 8 mirrors 000958 1780086601.745647 mapwrite 0x034124 thru 0x03ff8e (0xbe6b bytes) 000959 1780086601.869200 write 0x00769d thru 0x0134c4 (0xbe28 bytes) 000960 1780086601.881312 trunc from 0x03ff8f to 0x01f7c7 (0x207c7 bytes) 000962 1780086601.902707 read 0x008650 thru 0x00edcc (0x677d bytes) 000963 1780086601.907274 MIRROR_RESYNC 8 mirrors 000964 1780086602.549561 trunc from 0x01f7c8 to 0x026e3b (0x7674 bytes) 000965 1780086602.603478 write 0x004129 thru 0x007ccb (0x3ba3 bytes) 000966 1780086602.610457 read 0x00b166 thru 0x013e58 (0x8cf3 bytes) 000968 1780086602.619596 mapread 0x00baf2 thru 0x00c0b8 (0x5c7 bytes) 000970 1780086602.621289 write 0x00cb63 thru 0x018a3a (0xbed8 bytes) 000972 1780086602.625634 trunc from 0x026e3c to 0x03a857 (0x13a1c bytes) 000973 1780086602.639357 mapwrite 0x00ef5d thru 0x01e23d (0xf2e1 bytes) 000977 1780086602.679569 mapread 0x037b50 thru 0x03a857 (0x2d08 bytes) 000980 1780086602.690906 write 0x03d707 thru 0x03ffff (0x28f9 bytes) 000981 1780086602.691950 mapwrite 0x039625 thru 0x03f846 (0x6222 bytes) 000982 1780086602.737836 mapwrite 0x00fa63 thru 0x016e62 (0x7400 bytes) 000983 1780086602.772150 trunc from 0x040000 to 0x01a39c (0x25c63 bytes) 000984 1780086602.784325 trunc from 0x01a39d to 0x03d93d (0x235a1 bytes) 000985 1780086602.799269 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3d93e): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d93e): Mirror 4: 0xabfe79e5 (stale) Mirror 6: 0xabfe79e5 (stale) Mirror 7: 0xabfe79e5 (stale) Mirror 8: 0xabfe79e5 (stale) Mirror 9: 0xabfe79e5 (stale) Mirror 10: 0xabfe79e5 (stale) Mirror 11: 0xabfe79e5 (stale) CRC-32 checksum value for chunk [0, 0x3d93e): Mirror 1: 0x5e506da5 Mirror 12: 0x5e506da5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d93e: skipped 000989 1780086603.263646 read 0x021ec1 thru 0x0258ac (0x39ec bytes) 000990 1780086603.275126 mapwrite 0x038969 thru 0x03ffff (0x7697 bytes) 000993 1780086603.352344 mapwrite 0x02db1b thru 0x03c1ef (0xe6d5 bytes) 000994 1780086603.383076 write 0x023a94 thru 0x0285ab (0x4b18 bytes) 000996 1780086603.392570 mapread 0x0038c0 thru 0x00d1db (0x991c bytes) 001000 1780086603.400330 mapwrite 0x038c16 thru 0x03d521 (0x490c bytes) All operations completed A-OK! PASS 70a (35s) == sanity-flr test 71: check mirror extend parent fid ==== 16:30:05 (1780086605) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 72: check append on FLR file ========== 16:30:06 (1780086606) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5f9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.126088 s, 16.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5fe:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5f9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899069 s, 11.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5fe:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5fa:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5f9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5fe:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5fa:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5f9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5ff:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (4s) == sanity-flr test 200a: stress test ===================== 16:30:10 (1780086610) Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6506368 .. Extending file size to 5551712 .. failed Extending file size to 8759520 .. Extending file size to 6578944 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8508768 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9343008 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9304320 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9411808 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 58091 58092 58093 58096 58098 failed 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) 192.168.203.146@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (67s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 16:31:17 (1780086677) Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3319296 .. Extending file size to 1069824 .. done Extending file size to 5443200 .. Extending file size to 1196256 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 5496096 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7090944 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5717280 .. done Extending file size to 5755520 .. Extending file size to 7624288 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8867904 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 7309248 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7725920 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7852064 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8886592 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8779072 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9053504 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8867232 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9371616 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 60978 60979 60981 60983 60985 done 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) 192.168.203.146@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (68s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 16:32:25 (1780086745) Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3200 2205184 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3200 2205184 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0523878 s, 40.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.102139 s, 20.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x7e14bce3 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 16:32:32 (1780086752) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0374002 s, 56.1 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 16:32:34 (1780086754) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x40c:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x610:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x610:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x611:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x614:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 16:32:37 (1780086757) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0700529 s, 29.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x612:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x615:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x616:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 16:32:40 (1780086760) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0551504 s, 38.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x613:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x614:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x617:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x613:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x614:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x617:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x618:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 16:32:43 (1780086763) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0682556 s, 30.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x615:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x619:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x615:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x619:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x61a:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 16:33:08 (1780086788) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg346-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253647 s, 41.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x616:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x61c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x61b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x616:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x61c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x61b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x617:0x0] } PASS 204d (44s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 16:33:52 (1780086832) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg346-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x618:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x619:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0520666 s, 40.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x618:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x619:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x61d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x618:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x619:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x61d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (47s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 16:34:39 (1780086879) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg346-server mds-ost sync done. Creating new pool test_204f oleg346-server: Pool lustre.test_204f created Adding targets to pool oleg346-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg346-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0495839 s, 42.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x61a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x61e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x61a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x61e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg346-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg346-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg346-server: Pool lustre.test_204f destroyed PASS 204f (58s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 16:35:37 (1780086937) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 16:35:39 (1780086939) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 16:35:41 (1780086941) oleg346-server: Pool lustre.test_206 created Adding targets to pool oleg346-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg346-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg346-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg346-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg346-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (16s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 16:35:57 (1780086957) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258374 s, 40.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132324 s, 79.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x53bbe57f Mirror 3: 0x53bbe57f PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 16:36:00 (1780086960) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 16:36:01 (1780086961) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 16:36:03 (1780086963) Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0242633 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1970000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x14b0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x14b0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x14b0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1d20000 Read this (4 bytes): BBBB 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) PASS 209a (16s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 16:36:19 (1780086979) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00402903 s, 2.0 MB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 16:36:22 (1780086982) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151129 s, 69.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x431 lmm_fid: [0x200000404:0x431:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x431 lmm_fid: [0x200000404:0x431:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x628:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834993 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-29 16:36:22.000000000 -0400 Modify: 2026-05-29 16:36:22.000000000 -0400 Change: 2026-05-29 16:36:23.000000000 -0400 Birth: 2026-05-29 16:36:23.000000000 -0400 PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 16:36:25 (1780086985) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 16:36:26 (1780086986) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.168332 s, 243 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0224062 s, 183 kB/s size after second write -rw-r--r-- 1 root root 4096 May 29 16:36 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x433:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 May 29 16:36 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 16:36:29 (1780086989) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (3s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 16:36:32 (1780086992) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (3s) == sanity-flr test complete, duration 921 sec ============ 16:36:35 (1780086995) === sanity-flr: start cleanup 16:36:36 (1780086996) === === sanity-flr: finish cleanup 16:36:38 (1780086998) ===