-----============= acceptance-small: sanity-flr ============----- Wed Sep 16 18:13:36 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg337-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 18:13:57 (1789596837) === oleg337-client.virtnet: executing check_config_client /mnt/lustre oleg337-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg337-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8957c4e95800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8957c4e95800.idle_timeout=debug disable quota as required oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 18:14:32 (1789596872) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7663] == sanity-flr test 0a: lfs mirror create with -N option == 18:14:34 (1789596874) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 18:14:50 (1789596890) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 18:14:53 (1789596893) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 18:14:56 (1789596896) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 18:15:22 (1789596922) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 18:15:26 (1789596926) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 18:15:30 (1789596930) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8957c4e95800.stats=clear osc.lustre-OST0001-osc-ffff8957c4e95800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 18:15:37 (1789596937) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 18:15:48 (1789596948) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 18:15:56 (1789596956) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 18:15:59 (1789596959) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 18:16:06 (1789596966) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 18:16:16 (1789596976) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 18:16:22 (1789596982) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 18:16:26 (1789596986) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 18:16:33 (1789596993) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 18:16:43 (1789597003) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 18:16:51 (1789597011) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 1.49648 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165148 s, 6.3 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 18:17:02 (1789597022) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.132805 s, 39.5 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 18:17:09 (1789597029) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 18:17:18 (1789597038) Stopping /mnt/lustre-ost1 (opts:) on oleg337-server oleg337-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8957c4e95800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8957c4e95800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg337-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg337-server oleg337-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8957c4e95800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8957c4e95800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg337-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (75s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 18:18:34 (1789597114) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.3812 s, 16.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg337-server oleg337-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8957c4e95800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8957c4e95800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg337-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 18:19:17 (1789597157) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg337-server oleg337-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8957c4e95800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8957c4e95800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg337-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg337-server oleg337-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8957c4e95800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8957c4e95800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg337-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (85s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 18:20:42 (1789597242) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.7227 s, 11.1 MB/s -rw-r--r-- 1 root root 52428800 Sep 16 18:20 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.244s user 0m0.000s sys 0m0.177s Stopping /mnt/lustre-ost1 (opts:) on oleg337-server oleg337-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8957c4e95800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8957c4e95800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.152s user 0m0.000s sys 0m0.184s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg337-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg337-server oleg337-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8957c4e95800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8957c4e95800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.572s user 0m0.000s sys 0m0.185s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg337-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg337-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8957c4e95800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8957c4e95800.ost_server_uuid in FULL state after 0 sec PASS 33b (97s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 18:22:20 (1789597340) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 18:22:24 (1789597344) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 18:22:27 (1789597347) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 18:22:32 (1789597352) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 18:22:41 (1789597361) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.561495 s, 22.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 18:22:54 (1789597374) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684283 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401123 s, 26.1 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 18:23:03 (1789597383) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 18:23:11 (1789597391) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00198253 s, 517 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00186834 s, 548 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000913319 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000832219 s, 1.2 MB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 18:23:20 (1789597400) create test files with size 15747184 .. 745cdf3e5eabdb0f0ce108177a0ff6b3 - 7225f933c956f6731ec63d92c8283497 - 5d00755319f0ca84c8fd6f3bdea2308f - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg337-client.virtnet /mnt/lustre (opts:) Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 18:23:44 (1789597424) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 18:23:59 (1789597439) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 18:24:06 (1789597446) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157029 s, 6.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562463 s, 18.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 18:24:14 (1789597454) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.115513 s, 18.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.327749 s, 12.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.181249 s, 11.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 18:24:31 (1789597471) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 18:24:35 (1789597475) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 18:24:39 (1789597479) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416224 s, 25.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 18:24:48 (1789597488) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 18:24:52 (1789597492) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 18:25:04 (1789597504) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.387984 s, 27.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg337-client.virtnet /mnt/lustre (opts:) Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 18:25:18 (1789597518) llite.lustre-ffff8957c903a800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.252048 s, 41.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg337-client.virtnet /mnt/lustre (opts:) Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8957c6035000.xattr_cache=1 PASS 44e (18s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 18:25:36 (1789597536) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (43s) == sanity-flr test 46: Verify setstripe --copy option ==== 18:26:19 (1789597579) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (32s) == sanity-flr test 47: Verify mirror obj alloc =========== 18:26:51 (1789597611) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 18:26:55 (1789597615) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.190334 s, 16.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.213868 s, 14.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xa4383d0 (nosync,stale) Mirror 2: 0xa4383d0 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 39363d0a926b71f66e2209d4875de898 - mirror 1 checksum: a4f3015d5766efc3d9d81b64c73fa7bc - mirror 2 checksum: 39363d0a926b71f66e2209d4875de898 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-16T18:26:58-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-16T18:26:58-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 39363d0a926b71f66e2209d4875de898 - mirror 1 checksum: a4f3015d5766efc3d9d81b64c73fa7bc - mirror 2 checksum: a4f3015d5766efc3d9d81b64c73fa7bc - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-16T18:26:58-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-16T18:26:58-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 18:27:07 (1789597627) Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server 18:27:13 (1789597633) shut down facet: mds1 facet_host: oleg337-server facet_failover_host: oleg337-server Failover mds1 to oleg337-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 18:27:37 (1789597657) targets are mounted 18:27:37 (1789597657) facet_failover done oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529952 s, 19.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xee:0x0] } PASS 50A (44s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 18:27:52 (1789597672) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 18:27:55 (1789597675) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 18:27:59 (1789597679) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0138599 s, 1.2 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 18:28:07 (1789597687) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 18:28:11 (1789597691) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0130615 s, 314 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0644074 s, 63.6 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 18:28:19 (1789597699) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 18:28:41 (1789597721) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 18:29:11 (1789597751) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 18:29:35 (1789597775) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 18:29:59 (1789597799) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 18:30:07 (1789597807) mirror create pid 54362 mirror split pid 54363 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54362 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54363 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 18:31:15 (1789597875) Seed set to 7877 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero length punch hole skipping zero size read truncating to largest ever: 0x2cb1e 000004 1789597877.570374 trunc from 00000000 to 0x02cb1d (0x2cb1e bytes) truncating to largest ever: 0x34450 000009 1789597877.590572 trunc from 0x02cb1e to 0x03444f (0x7932 bytes) 000010 1789597877.606619 mapwrite 0x00b00d thru 0x00e76d (0x3761 bytes) 000011 1789597877.729787 mapread 0x013b51 thru 0x013d12 (0x1c2 bytes) 000012 1789597877.730603 read 0x0011b8 thru 0x0045dd (0x3426 bytes) 000013 1789597877.740602 read 0x0158ae thru 0x025387 (0xfada bytes) 000014 1789597877.759302 mapwrite 0x03a05b thru 0x03ffff (0x5fa5 bytes) 000017 1789597877.852352 mapwrite 0x01c219 thru 0x01f7cb (0x35b3 bytes) 000018 1789597877.895650 trunc from 0x040000 to 0x0285f3 (0x17a0c bytes) 000020 1789597877.933108 read 0x016a64 thru 0x01fc09 (0x91a6 bytes) 000022 1789597877.934166 mapread 0x01dc6b thru 0x01e66e (0xa04 bytes) 000025 1789597877.938441 read 0x025561 thru 0x0285f3 (0x3093 bytes) 000026 1789597877.939238 mapread 0x008018 thru 0x008bb9 (0xba2 bytes) 000028 1789597877.949674 mapread 0x01b15d thru 0x01d3de (0x2282 bytes) 000029 1789597877.950467 write 0x00acd5 thru 0x016e6e (0xc19a bytes) 000030 1789597877.963501 mapread 0x002372 thru 0x0072b7 (0x4f46 bytes) 000031 1789597877.964983 mapwrite 0x011189 thru 0x0118f6 (0x76e bytes) 000033 1789597878.040651 mapread 0x0115ac thru 0x01dc5d (0xc6b2 bytes) 000034 1789597878.042288 read 0x02295a thru 0x0285f3 (0x5c9a bytes) 000035 1789597878.043697 read 0x0187e6 thru 0x0238e8 (0xb103 bytes) 000036 1789597878.046137 punch from 0x01a259 to 0x024a36 (0xa7de bytes) 000037 1789597878.056043 punch from 0x018a96 to 0x0259cb (0xcf36 bytes) 000038 1789597878.066411 MIRROR_EXTEND to 2 mirrors 000041 1789597879.180660 write 0x03493d thru 0x035119 (0x7dd bytes) 000043 1789597879.313926 punch from 0x030ca8 to 0x035119 (0x4472 bytes) 000044 1789597879.353927 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) 000047 1789597879.544174 trunc from 0x03511a to 0x00fb38 (0x255e1 bytes) 000048 1789597879.560227 mapread 0x003936 thru 0x00fb38 (0xc203 bytes) 000049 1789597879.575893 mapwrite 0x02a311 thru 0x02edf7 (0x4ae7 bytes) 000051 1789597879.642927 mapwrite 0x02db9e thru 0x03ae1a (0xd27d bytes) 000052 1789597879.714109 punch from 0x039ad4 to 0x03ae1a (0x1347 bytes) 000054 1789597879.722772 read 0x01bc0b thru 0x01d5b9 (0x19af bytes) 000055 1789597879.729666 mapwrite 0x022749 thru 0x02cc39 (0xa4f1 bytes) 000056 1789597879.791919 trunc from 0x03ae1b to 0x01f7a0 (0x1b67a bytes) 000058 1789597879.808264 write 0x02aefa thru 0x02dcaa (0x2db1 bytes) 000060 1789597879.810559 mapread 0x001257 thru 0x010048 (0xedf2 bytes) 000061 1789597879.819629 mapwrite 0x013538 thru 0x021a33 (0xe4fc bytes) 000062 1789597879.868042 read 0x0039ca thru 0x00c129 (0x8760 bytes) 000063 1789597879.868656 mapwrite 0x01dac0 thru 0x0251e9 (0x772a bytes) 000065 1789597879.932550 read 0x028c78 thru 0x02dcaa (0x5033 bytes) 000067 1789597879.934310 mapread 0x024427 thru 0x029df1 (0x59cb bytes) 000069 1789597879.936479 write 0x02ea66 thru 0x0357fb (0x6d96 bytes) 000070 1789597879.937727 write 0x03f7a0 thru 0x03ffff (0x860 bytes) 000074 1789597879.941435 write 0x00c48f thru 0x012583 (0x60f5 bytes) 000075 1789597879.942801 read 0x036e59 thru 0x03cc9d (0x5e45 bytes) 000076 1789597879.949551 trunc from 0x040000 to 0x02d5e0 (0x12a1f bytes) 000078 1789597879.975269 MIRROR_EXTEND to 2 mirrors 000079 1789597880.654541 punch from 0x005406 to 0x00b967 (0x6562 bytes) 000081 1789597880.720922 read 0x00da7a thru 0x00f78f (0x1d16 bytes) 000082 1789597880.728423 punch from 0x02bf6c to 0x02d5e0 (0x1675 bytes) 000083 1789597880.736224 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2d5e1): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d5e1): Mirror 2: 0x9216c77 (stale) CRC-32 checksum value for chunk [0, 0x2d5e1): Mirror 1: 0xb1c063d6 Mirror 3: 0xb1c063d6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d5e1: skipped 000084 1789597882.022223 MIRROR_RESYNC 3 mirrors 000085 1789597883.206189 write 0x00d728 thru 0x016a17 (0x92f0 bytes) 000087 1789597883.299709 mapwrite 0x0394c6 thru 0x03ffff (0x6b3a bytes) 000088 1789597883.385620 write 0x03ba40 thru 0x03e184 (0x2745 bytes) 000089 1789597883.387532 mapread 0x00894f thru 0x00f1ee (0x68a0 bytes) 000091 1789597883.397591 write 0x018620 thru 0x01c616 (0x3ff7 bytes) 000093 1789597883.412120 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x2e880228 (stale) Mirror 3: 0x2e880228 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8fea4059 Mirror 4: 0x8fea4059 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000096 1789597884.770003 punch from 0x006810 to 0x008675 (0x1e66 bytes) 000097 1789597884.883521 mapwrite 0x0253f5 thru 0x029864 (0x4470 bytes) 000098 1789597884.954224 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x2e880228 (stale) Mirror 4: 0x8fea4059 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x2e880228 (stale) Mirror 4: 0x8fea4059 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000099 1789597885.365879 mapwrite 0x002e84 thru 0x004447 (0x15c4 bytes) 000101 1789597885.413440 write 0x02cfc3 thru 0x03344d (0x648b bytes) 000102 1789597885.436993 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8fea4059 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8fea4059 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000103 1789597885.830352 mapread 0x00f2bc thru 0x01ba2c (0xc771 bytes) 000104 1789597885.870541 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) 000106 1789597885.983255 mapread 0x0117bb thru 0x017df3 (0x6639 bytes) 000108 1789597886.004477 read 0x01b352 thru 0x01d87c (0x252b bytes) 000109 1789597886.009414 trunc from 0x040000 to 0x021a83 (0x1e57c bytes) 000111 1789597886.028482 read 0x0002dc thru 0x00cde0 (0xcb05 bytes) 000112 1789597886.036193 write 0x01b569 thru 0x01ccc6 (0x175e bytes) 000113 1789597886.037162 punch from 0x00776f to 0x00c122 (0x49b4 bytes) 000114 1789597886.045315 mapwrite 0x00eea6 thru 0x011c1e (0x2d79 bytes) 000115 1789597886.084028 mapwrite 0x034416 thru 0x03c47f (0x806a bytes) 000117 1789597886.152573 read 0x0234c4 thru 0x033106 (0xfc43 bytes) 000118 1789597886.154239 write 0x029daf thru 0x031be0 (0x7e32 bytes) 000119 1789597886.160073 MIRROR_EXTEND to 2 mirrors 000120 1789597887.094312 punch from 0x025adf to 0x0353d3 (0xf8f5 bytes) 000121 1789597887.234681 mapwrite 0x03aaed thru 0x03bb94 (0x10a8 bytes) 000124 1789597887.437466 read 0x005e1e thru 0x00b78a (0x596d bytes) 000127 1789597887.471996 punch from 0x02616b to 0x033dfb (0xdc91 bytes) 000128 1789597887.489309 read 0x00440f thru 0x00ae50 (0x6a42 bytes) 000130 1789597887.530561 mapread 0x038353 thru 0x03c47f (0x412d bytes) 000131 1789597887.531612 punch from 0x024ac6 to 0x02e00f (0x954a bytes) 000132 1789597887.545523 mapread 0x015b47 thru 0x01cedf (0x7399 bytes) 000133 1789597887.551508 trunc from 0x03c480 to 0x024483 (0x17ffc bytes) 000134 1789597887.649627 write 0x034480 thru 0x03ffff (0xbb80 bytes) 000137 1789597887.652699 write 0x014779 thru 0x020eea (0xc772 bytes) 000138 1789597887.655744 write 0x02c7e3 thru 0x02fd4b (0x3569 bytes) 000142 1789597887.675803 read 0x02409d thru 0x0338b5 (0xf819 bytes) 000143 1789597887.711188 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe4f669a2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb79ba9fd Mirror 3: 0xb79ba9fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000144 1789597888.778193 trunc from 0x040000 to 0x00b56f (0x34a90 bytes) 000145 1789597888.879225 punch from 0x002bb8 to 0x006682 (0x3acb bytes) 000146 1789597888.890053 mapread 0x002cd6 thru 0x007718 (0x4a43 bytes) 000147 1789597888.923303 read 0x000965 thru 0x00b56f (0xac0b bytes) 000148 1789597888.929359 mapwrite 0x01d81e thru 0x0240ef (0x68d2 bytes) 000150 1789597889.025554 mapread 0x00053d thru 0x0027f8 (0x22bc bytes) 000152 1789597889.027758 write 0x00d9e8 thru 0x011d1b (0x4334 bytes) 000153 1789597889.038790 punch from 0x014152 to 0x01c473 (0x8322 bytes) 000154 1789597889.050189 mapread 0x006223 thru 0x013291 (0xd06f bytes) 000155 1789597889.058067 read 0x0116e7 thru 0x01b7c7 (0xa0e1 bytes) 000157 1789597889.069469 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x240f0): Mirror 3: 0x4956302c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x240f0): Mirror 3: 0x4956302c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x240f0: skipped 000158 1789597889.408300 mapread 0x005851 thru 0x0067cd (0xf7d bytes) 000159 1789597889.456502 MIRROR_RESYNC 2 mirrors 000160 1789597890.520649 trunc from 0x0240f0 to 0x01c68c (0x7a63 bytes) 000161 1789597890.603886 write 0x02ed21 thru 0x0389b7 (0x9c97 bytes) 000163 1789597890.616638 read 0x01969e thru 0x0234be (0x9e21 bytes) 000165 1789597890.624787 read 0x0141be thru 0x021232 (0xd075 bytes) 000166 1789597890.635251 mapwrite 0x02b763 thru 0x0309fc (0x529a bytes) 000167 1789597890.689139 punch from 0x00f011 to 0x017288 (0x8278 bytes) 000168 1789597890.693388 write 0x001816 thru 0x00fa2e (0xe219 bytes) 000171 1789597890.699660 mapwrite 0x00e12c thru 0x019790 (0xb665 bytes) 000172 1789597890.779409 mapwrite 0x036b8b thru 0x03ccb6 (0x612c bytes) 000173 1789597890.897348 write 0x00361f thru 0x010d62 (0xd744 bytes) 000174 1789597890.905419 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) 000176 1789597891.122756 mapwrite 0x02b36d thru 0x02c480 (0x1114 bytes) 000177 1789597891.235545 trunc from 0x03ccb7 to 0x026972 (0x16344 bytes) 000179 1789597891.263471 mapread 0x019d53 thru 0x024b43 (0xadf1 bytes) 000182 1789597891.289210 write 0x02ef77 thru 0x03e82f (0xf8b9 bytes) 000183 1789597891.291227 mapwrite 0x027321 thru 0x031a2b (0xa70b bytes) 000186 1789597891.366646 punch from 0x027625 to 0x036cde (0xf6ba bytes) 000187 1789597891.373285 read 0x022538 thru 0x02c22d (0x9cf6 bytes) 000188 1789597891.386624 mapread 0x02c698 thru 0x031807 (0x5170 bytes) 000189 1789597891.394137 write 0x016236 thru 0x021fe1 (0xbdac bytes) 000191 1789597891.398672 read 0x01120d thru 0x019807 (0x85fb bytes) 000194 1789597891.411251 mapread 0x0255e2 thru 0x03363d (0xe05c bytes) 000195 1789597891.412224 write 0x02abac thru 0x0308c5 (0x5d1a bytes) 000196 1789597891.416661 write 0x02aa5e thru 0x039fd1 (0xf574 bytes) 000197 1789597891.422595 read 0x011789 thru 0x01c54b (0xadc3 bytes) 000198 1789597891.423800 mapwrite 0x01bf93 thru 0x02a91b (0xe989 bytes) 000199 1789597891.484365 read 0x03e09c thru 0x03e60c (0x571 bytes) 000203 1789597891.486378 punch from 0x035045 to 0x03e656 (0x9612 bytes) 000205 1789597891.493940 read 0x033f6c thru 0x03dab9 (0x9b4e bytes) 000206 1789597891.501336 mapread 0x003ec0 thru 0x0077de (0x391f bytes) 000207 1789597891.514996 MIRROR_EXTEND to 2 mirrors 000208 1789597892.356496 mapwrite 0x005c81 thru 0x006791 (0xb11 bytes) 000209 1789597892.580295 write 0x011171 thru 0x01612c (0x4fbc bytes) 000211 1789597892.589491 write 0x03a5c1 thru 0x03ffff (0x5a3f bytes) 000212 1789597892.601992 MIRROR_RESYNC 2 mirrors 000214 1789597893.704162 MIRROR_EXTEND to 3 mirrors 000215 1789597894.427609 mapwrite 0x032dad thru 0x03d70d (0xa961 bytes) truncating to largest ever: 0x37caf 000217 1789597894.645182 trunc from 0x040000 to 0x037cae (0x8351 bytes) 000218 1789597894.676057 read 0x0275f5 thru 0x02817e (0xb8a bytes) 000222 1789597894.687758 mapread 0x022cdc thru 0x02bbc9 (0x8eee bytes) 000223 1789597894.708625 mapwrite 0x012601 thru 0x015f54 (0x3954 bytes) 000224 1789597894.820967 MIRROR_RESYNC 3 mirrors 000225 1789597895.845276 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) 000226 1789597896.242698 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) 000227 1789597896.379739 write 0x02d9c8 thru 0x036f5a (0x9593 bytes) 000229 1789597896.402495 mapwrite 0x03bed0 thru 0x03ee12 (0x2f43 bytes) 000230 1789597896.570430 mapread 0x024a8c thru 0x033992 (0xef07 bytes) 000231 1789597896.590162 read 0x035360 thru 0x03ee12 (0x9ab3 bytes) 000232 1789597896.590897 trunc from 0x03ee13 to 0x00c5e9 (0x32829 bytes) 000233 1789597896.606129 write 0x01e78d thru 0x01f474 (0xce8 bytes) 000236 1789597896.609106 punch from 0x006bc5 to 0x006e7b (0x2b7 bytes) 000238 1789597896.620047 mapwrite 0x033154 thru 0x038afb (0x59a8 bytes) 000241 1789597896.686490 read 0x0113b0 thru 0x02013e (0xed8f bytes) 000245 1789597896.703451 punch from 0x02f174 to 0x038afb (0x9988 bytes) 000247 1789597896.710482 read 0x017812 thru 0x021a1c (0xa20b bytes) 000248 1789597896.711253 mapread 0x0267ea thru 0x0354fe (0xed15 bytes) 000249 1789597896.721611 write 0x02e22b thru 0x02f0f4 (0xeca bytes) 000250 1789597896.723191 write 0x0365c1 thru 0x03cbe9 (0x6629 bytes) 000251 1789597896.724806 trunc from 0x03cbea to 0x0167f0 (0x263f9 bytes) 000252 1789597896.742552 read 0x012b1c thru 0x0167f0 (0x3cd5 bytes) 000253 1789597896.744086 trunc from 0x0167f1 to 0x008108 (0xe6e8 bytes) 000255 1789597896.760474 punch from 0x0070ed to 0x008108 (0x101c bytes) 000256 1789597896.765610 mapwrite 0x00097c thru 0x00a60e (0x9c93 bytes) 000257 1789597896.849245 trunc from 0x00a60f to 0x001953 (0x8cbb bytes) 000258 1789597896.869332 write 0x02fe29 thru 0x0363d7 (0x65af bytes) 000259 1789597896.871487 read 0x01421c thru 0x018209 (0x3fee bytes) 000260 1789597896.883691 trunc from 0x0363d8 to 0x02ea94 (0x7943 bytes) 000261 1789597896.896931 punch from 0x01bb1e to 0x0206f6 (0x4bd9 bytes) 000262 1789597896.902088 mapwrite 0x005d6c thru 0x00dfcf (0x8264 bytes) 000265 1789597896.942953 punch from 0x00c61f to 0x016efb (0xa8dd bytes) 000266 1789597896.949427 read 0x008170 thru 0x010923 (0x87b4 bytes) 000267 1789597896.960467 read 0x025058 thru 0x02b223 (0x61cc bytes) 000271 1789597896.965297 mapwrite 0x036820 thru 0x03ffff (0x97e0 bytes) 000273 1789597897.085478 write 0x029dc8 thru 0x02b962 (0x1b9b bytes) 000277 1789597897.091474 trunc from 0x040000 to 0x00040c (0x3fbf3 bytes) 000278 1789597897.118036 punch from 0x000308 to 0x00040c (0x105 bytes) 000280 1789597897.125853 write 0x028b33 thru 0x033c96 (0xb164 bytes) 000281 1789597897.131434 punch from 0x031351 to 0x032105 (0xdb5 bytes) 000282 1789597897.155183 read 0x031845 thru 0x033c96 (0x2452 bytes) 000283 1789597897.166281 write 0x0288cd thru 0x031ebb (0x95ef bytes) 000285 1789597897.168053 write 0x00bb6e thru 0x015bbc (0xa04f bytes) 000286 1789597897.170658 read 0x007141 thru 0x013887 (0xc747 bytes) 000288 1789597897.172987 punch from 0x02630e to 0x02912e (0x2e21 bytes) 000290 1789597897.194266 mapread 0x0339a2 thru 0x033c96 (0x2f5 bytes) 000294 1789597897.198166 mapwrite 0x02baa4 thru 0x02da95 (0x1ff2 bytes) 000295 1789597897.251008 trunc from 0x033c97 to 0x029e7f (0x9e17 bytes) 000296 1789597897.265018 read 0x01dc4e thru 0x01eb1a (0xecd bytes) 000298 1789597897.266027 write 0x023014 thru 0x02b1a5 (0x8192 bytes) 000299 1789597897.267143 mapwrite 0x005f0a thru 0x007e11 (0x1f08 bytes) 000300 1789597897.312710 write 0x0078b8 thru 0x013ed3 (0xc61c bytes) 000302 1789597897.314859 mapwrite 0x01abfc thru 0x01b668 (0xa6d bytes) 000304 1789597897.358461 MIRROR_EXTEND to 2 mirrors 000305 1789597898.222638 MIRROR_EXTEND to 3 mirrors 000306 1789597898.932741 mapwrite 0x023019 thru 0x02d531 (0xa519 bytes) 000308 1789597899.099583 mapwrite 0x000898 thru 0x00fccd (0xf436 bytes) 000310 1789597899.160173 punch from 0x027c28 to 0x02d531 (0x590a bytes) 000311 1789597899.169789 mapwrite 0x007ee6 thru 0x012ae0 (0xabfb bytes) 000312 1789597899.273763 mapwrite 0x01774d thru 0x0264ba (0xed6e bytes) 000314 1789597899.383085 write 0x00800a thru 0x00823a (0x231 bytes) 000315 1789597899.387441 write 0x03ab32 thru 0x03ffff (0x54ce bytes) 000317 1789597899.394943 read 0x02c36c thru 0x03a950 (0xe5e5 bytes) 000319 1789597899.409053 mapwrite 0x0090d9 thru 0x015569 (0xc491 bytes) 000320 1789597899.493569 punch from 0x0062e2 to 0x007f52 (0x1c71 bytes) 000321 1789597899.501080 mapwrite 0x034053 thru 0x035357 (0x1305 bytes) 000322 1789597899.563753 mapread 0x025748 thru 0x02c8c2 (0x717b bytes) 000323 1789597899.575602 read 0x0200f1 thru 0x02d624 (0xd534 bytes) 000326 1789597899.577190 write 0x029133 thru 0x036fa0 (0xde6e bytes) 000327 1789597899.579636 trunc from 0x040000 to 0x025ea5 (0x1a15a bytes) 000328 1789597899.601491 mapread 0x010eb8 thru 0x015553 (0x469c bytes) 000329 1789597899.606217 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x25ea6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25ea6): Mirror 2: 0x8997aafb (stale) Mirror 3: 0x8997aafb (stale) CRC-32 checksum value for chunk [0, 0x25ea6): Mirror 1: 0xb572ec2a Mirror 4: 0xb572ec2a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25ea6: skipped 000330 1789597901.058746 mapread 0x00e396 thru 0x00f9ac (0x1617 bytes) 000331 1789597901.082596 write 0x03490d thru 0x039cf0 (0x53e4 bytes) 000332 1789597901.164392 mapread 0x037c72 thru 0x039cf0 (0x207f bytes) 000333 1789597901.165968 mapread 0x025490 thru 0x02ac81 (0x57f2 bytes) 000334 1789597901.193981 MIRROR_RESYNC 4 mirrors 000335 1789597902.652705 write 0x020797 thru 0x0246ef (0x3f59 bytes) 000336 1789597902.762312 punch from 0x039ca0 to 0x039cf0 (0x51 bytes) 000337 1789597902.788500 punch from 0x038ad3 to 0x039cf0 (0x121e bytes) 000339 1789597902.795381 punch from 0x004791 to 0x00b550 (0x6dc0 bytes) 000340 1789597902.809086 write 0x01d4ce thru 0x01f4da (0x200d bytes) 000341 1789597902.821959 mapread 0x01c6d9 thru 0x0260c3 (0x99eb bytes) 000343 1789597902.855375 trunc from 0x039cf1 to 0x033800 (0x64f0 bytes) 000345 1789597902.895022 mapwrite 0x015a5a thru 0x015e14 (0x3bb bytes) 000346 1789597903.012968 punch from 0x00af5d to 0x0102ac (0x5350 bytes) 000347 1789597903.021710 punch from 0x01706e to 0x0199bb (0x294e bytes) 000348 1789597903.030510 read 0x006190 thru 0x00fccd (0x9b3e bytes) 000349 1789597903.046091 punch from 0x012ec4 to 0x01febc (0xcff9 bytes) 000350 1789597903.052500 trunc from 0x033801 to 0x009f60 (0x298a0 bytes) 000351 1789597903.090146 mapwrite 0x002abe thru 0x00de93 (0xb3d6 bytes) 000353 1789597903.202179 mapwrite 0x03fa7b thru 0x03ffff (0x585 bytes) 000354 1789597903.299685 trunc from 0x040000 to 0x01bf8c (0x24073 bytes) 000357 1789597903.323260 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1bf8d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bf8d): Mirror 2: 0x2ac51769 (stale) Mirror 3: 0x2ac51769 (stale) Mirror 4: 0x2ac51769 (stale) CRC-32 checksum value for chunk [0, 0x1bf8d): Mirror 1: 0x92209d72 Mirror 5: 0x92209d72 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bf8d: skipped 000360 1789597904.313282 mapread 0x01728a thru 0x01bf8c (0x4d03 bytes) 000364 1789597904.343795 punch from 0x01b351 to 0x01bf8c (0xc3c bytes) 000365 1789597904.438258 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1bf8d): Mirror 2: 0x2ac51769 (stale) Mirror 3: 0x2ac51769 (stale) Mirror 5: 0x92209d72 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bf8d): Mirror 2: 0x2ac51769 (stale) Mirror 3: 0x2ac51769 (stale) Mirror 5: 0x92209d72 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bf8d: skipped 000370 1789597904.771562 mapwrite 0x028ba4 thru 0x02d5f5 (0x4a52 bytes) 000372 1789597904.903696 write 0x037dea thru 0x03ffff (0x8216 bytes) 000374 1789597904.906277 trunc from 0x040000 to 0x01e013 (0x21fec bytes) 000376 1789597904.932584 write 0x03ca20 thru 0x03ffff (0x35e0 bytes) 000377 1789597904.934569 trunc from 0x040000 to 0x008bdb (0x37424 bytes) 000378 1789597904.951878 write 0x0372ed thru 0x03ffff (0x8d13 bytes) 000379 1789597904.953189 punch from 0x012c80 to 0x016ef5 (0x4276 bytes) 000380 1789597904.971079 read 0x035df9 thru 0x03c1fd (0x6405 bytes) 000381 1789597904.979504 punch from 0x03e099 to 0x03ffff (0x1f67 bytes) 000382 1789597904.984689 mapwrite 0x018857 thru 0x01ff43 (0x76ed bytes) 000384 1789597905.049070 punch from 0x0226e6 to 0x0288b1 (0x61cc bytes) 000385 1789597905.067337 mapwrite 0x031038 thru 0x03ad9c (0x9d65 bytes) 000386 1789597905.144321 mapwrite 0x01fbc5 thru 0x02d726 (0xdb62 bytes) 000388 1789597905.213953 trunc from 0x040000 to 0x0135c3 (0x2ca3c bytes) 000389 1789597905.227115 write 0x013a89 thru 0x01ba14 (0x7f8c bytes) 000391 1789597905.238363 MIRROR_RESYNC 4 mirrors 000392 1789597906.501943 punch from 0x00648e to 0x008bfb (0x276e bytes) 000395 1789597906.647858 write 0x025c56 thru 0x02cd3e (0x70e9 bytes) 000397 1789597906.649554 read 0x020c4d thru 0x0221d2 (0x1586 bytes) 000398 1789597906.663456 read 0x029f25 thru 0x02cd3e (0x2e1a bytes) 000400 1789597906.668170 mapwrite 0x00badb thru 0x01b7c7 (0xfced bytes) 000406 1789597906.766531 punch from 0x01fb59 to 0x02cd3e (0xd1e6 bytes) 000409 1789597906.779261 trunc from 0x02cd3f to 0x02378d (0x95b1 bytes) 000410 1789597906.820078 write 0x03c9db thru 0x03ffff (0x3625 bytes) 000413 1789597906.824143 trunc from 0x040000 to 0x0126ac (0x2d953 bytes) 000414 1789597906.855917 trunc from 0x0126ad to 0x037ab9 (0x2540d bytes) 000415 1789597906.882011 trunc from 0x037aba to 0x02f4ce (0x85eb bytes) 000416 1789597906.906264 punch from 0x027176 to 0x02f4ce (0x8359 bytes) 000417 1789597906.920065 read 0x0070c8 thru 0x00ccf1 (0x5c2a bytes) 000418 1789597906.920682 write 0x00ba0f thru 0x012949 (0x6f3b bytes) 000419 1789597906.921720 mapread 0x02a9bb thru 0x02f4ce (0x4b14 bytes) 000420 1789597906.939477 punch from 0x028434 to 0x02f4ce (0x709b bytes) 000422 1789597906.959333 punch from 0x008e6c to 0x00feba (0x704f bytes) 000423 1789597906.970780 trunc from 0x02f4cf to 0x007eac (0x27622 bytes) 000425 1789597907.030689 read 0x003129 thru 0x007eac (0x4d84 bytes) 000427 1789597907.040242 punch from 0x007a9d to 0x007eac (0x410 bytes) 000429 1789597907.045525 mapwrite 0x01cc39 thru 0x022458 (0x5820 bytes) 000430 1789597907.162440 punch from 0x00eafd to 0x0147fb (0x5cff bytes) 000431 1789597907.169347 mapread 0x00abf3 thru 0x011fd6 (0x73e4 bytes) 000433 1789597907.204948 mapread 0x01cf5c thru 0x020885 (0x392a bytes) 000434 1789597907.205854 read 0x001403 thru 0x00ae27 (0x9a25 bytes) 000435 1789597907.206397 punch from 0x019f32 to 0x01a3f6 (0x4c5 bytes) 000436 1789597907.213930 trunc from 0x022459 to 0x02a898 (0x8440 bytes) 000438 1789597907.255731 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2a899): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a899): Mirror 2: 0x61a54a5f (stale) Mirror 3: 0x61a54a5f (stale) Mirror 5: 0x61a54a5f (stale) CRC-32 checksum value for chunk [0, 0x2a899): Mirror 1: 0xdcb69aaa Mirror 6: 0xdcb69aaa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a899: skipped 000440 1789597908.636730 mapread 0x00c9cd thru 0x014145 (0x7779 bytes) 000441 1789597908.687977 mapwrite 0x0074b1 thru 0x016485 (0xefd5 bytes) 000442 1789597908.884579 read 0x01e80f thru 0x024911 (0x6103 bytes) 000443 1789597908.892584 mapread 0x010a12 thru 0x01ba5c (0xb04b bytes) 000446 1789597908.900577 read 0x01b951 thru 0x024b5c (0x920c bytes) 000447 1789597908.901266 mapread 0x026146 thru 0x02944d (0x3308 bytes) 000448 1789597908.909091 punch from 0x01238a to 0x01ab01 (0x8778 bytes) 000449 1789597908.917566 MIRROR_RESYNC 5 mirrors 000450 1789597910.275871 MIRROR_RESYNC 5 mirrors 000452 1789597910.781402 mapwrite 0x0276e1 thru 0x02e2d7 (0x6bf7 bytes) 000453 1789597910.950638 read 0x002751 thru 0x00d6dc (0xaf8c bytes) 000454 1789597910.958086 punch from 0x00a11a to 0x01837c (0xe263 bytes) 000456 1789597910.966484 write 0x003f0e thru 0x00d126 (0x9219 bytes) 000457 1789597910.973478 mapread 0x0192b1 thru 0x0203d8 (0x7128 bytes) 000458 1789597910.987219 mapread 0x029f5d thru 0x02bb0e (0x1bb2 bytes) 000459 1789597910.988687 punch from 0x00337f to 0x00a53f (0x71c1 bytes) 000460 1789597911.005432 write 0x005f15 thru 0x0105c1 (0xa6ad bytes) 000461 1789597911.020207 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e2d8): Mirror 2: 0xf8c24657 (stale) Mirror 3: 0xf8c24657 (stale) Mirror 6: 0xf8c24657 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e2d8): Mirror 2: 0xf8c24657 (stale) Mirror 3: 0xf8c24657 (stale) Mirror 6: 0xf8c24657 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e2d8: skipped 000462 1789597911.364898 MIRROR_RESYNC 4 mirrors 000463 1789597912.916993 punch from 0x01df27 to 0x02dec5 (0xff9f bytes) 000465 1789597913.001920 punch from 0x021b78 to 0x02a585 (0x8a0e bytes) 000466 1789597913.006910 trunc from 0x02e2d8 to 0x02533c (0x8f9b bytes) 000467 1789597913.027169 read 0x0183d8 thru 0x01f991 (0x75ba bytes) 000468 1789597913.036476 mapread 0x00334a thru 0x0121ca (0xee81 bytes) 000469 1789597913.080470 punch from 0x007f0c to 0x014028 (0xc11d bytes) 000471 1789597913.089278 mapread 0x018378 thru 0x020da8 (0x8a31 bytes) 000472 1789597913.098156 trunc from 0x02533d to 0x0319d4 (0xc698 bytes) 000473 1789597913.120787 punch from 0x02085a to 0x02a7d3 (0x9f7a bytes) 000474 1789597913.127437 mapwrite 0x007129 thru 0x00f204 (0x80dc bytes) 000477 1789597913.212554 write 0x009a3c thru 0x00aa4c (0x1011 bytes) 000478 1789597913.213489 punch from 0x00ff2c to 0x012eec (0x2fc1 bytes) 000480 1789597913.239068 punch from 0x028852 to 0x02bb0c (0x32bb bytes) 000485 1789597913.262777 MIRROR_RESYNC 4 mirrors 000486 1789597915.252787 punch from 0x00432c to 0x00c57d (0x8252 bytes) 000487 1789597915.322679 write 0x035d77 thru 0x03c5d1 (0x685b bytes) 000488 1789597915.323988 mapread 0x004e25 thru 0x00c4df (0x76bb bytes) 000489 1789597915.361240 MIRROR_RESYNC 4 mirrors 000490 1789597917.414275 mapwrite 0x03f26e thru 0x03ffff (0xd92 bytes) 000491 1789597917.566319 trunc from 0x040000 to 0x013329 (0x2ccd6 bytes) 000492 1789597917.599135 write 0x03086d thru 0x03b5dd (0xad71 bytes) 000494 1789597917.606307 mapread 0x014509 thru 0x0243ad (0xfea5 bytes) 000497 1789597917.630129 write 0x03246d thru 0x03866b (0x61ff bytes) 000498 1789597917.631797 read 0x0094a8 thru 0x00d547 (0x40a0 bytes) 000499 1789597917.639241 mapread 0x000a24 thru 0x005582 (0x4b5f bytes) 000500 1789597917.651365 mapread 0x013ca2 thru 0x01b6df (0x7a3e bytes) 000501 1789597917.653355 trunc from 0x03b5de to 0x028c8b (0x12952 bytes) 000503 1789597917.670616 trunc from 0x028c8c to 0x008e89 (0x1fe02 bytes) 000504 1789597917.696040 punch from 0x000ebf to 0x008e89 (0x7fcb bytes) 000506 1789597917.709113 mapwrite 0x01df22 thru 0x02a3af (0xc48e bytes) 000507 1789597917.785421 trunc from 0x02a3b0 to 0x037546 (0xd197 bytes) 000509 1789597917.807025 mapwrite 0x03d7a4 thru 0x03ffff (0x285c bytes) 000510 1789597917.915545 punch from 0x0192ef to 0x028c46 (0xf958 bytes) 000512 1789597917.921499 mapread 0x00eb7c thru 0x019713 (0xab98 bytes) 000514 1789597917.941277 read 0x007f11 thru 0x00cc09 (0x4cf9 bytes) 000515 1789597917.942111 punch from 0x018b77 to 0x02495c (0xbde6 bytes) 000516 1789597917.948451 mapread 0x02b55f thru 0x031656 (0x60f8 bytes) 000517 1789597917.961078 punch from 0x032bdb to 0x03d1dd (0xa603 bytes) 000518 1789597917.966137 write 0x02e955 thru 0x02ee96 (0x542 bytes) 000520 1789597917.973749 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x2733f657 (stale) Mirror 3: 0x2733f657 (stale) Mirror 6: 0x2733f657 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcd7ab088 Mirror 7: 0xcd7ab088 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000521 1789597919.057531 punch from 0x024e97 to 0x032122 (0xd28c bytes) 000523 1789597919.135441 trunc from 0x040000 to 0x003b45 (0x3c4ba bytes) 000524 1789597919.167982 mapread 0x0009f0 thru 0x00316b (0x277c bytes) 000525 1789597919.186562 mapread 0x002814 thru 0x003b45 (0x1332 bytes) 000526 1789597919.187410 mapread 0x0032d0 thru 0x003b45 (0x876 bytes) 000527 1789597919.188175 mapread 0x0004e8 thru 0x003b45 (0x365e bytes) 000528 1789597919.194164 MIRROR_RESYNC 5 mirrors 000529 1789597920.460132 write 0x0101fb thru 0x01f2e3 (0xf0e9 bytes) 000530 1789597920.537286 mapread 0x0141cc thru 0x01df86 (0x9dbb bytes) 000531 1789597920.538673 read 0x00658d thru 0x009554 (0x2fc8 bytes) 000532 1789597920.551538 write 0x015725 thru 0x017017 (0x18f3 bytes) 000534 1789597920.557738 mapwrite 0x016ca5 thru 0x020f3d (0xa299 bytes) 000536 1789597920.641500 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20f3e): Mirror 2: 0xcad56285 (stale) Mirror 3: 0xcad56285 (stale) Mirror 6: 0xcad56285 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20f3e): Mirror 2: 0xcad56285 (stale) Mirror 3: 0xcad56285 (stale) Mirror 6: 0xcad56285 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20f3e: skipped 000537 1789597920.955243 mapread 0x013fc7 thru 0x01d0d3 (0x910d bytes) 000538 1789597920.988661 trunc from 0x020f3e to 0x0096b5 (0x17888 bytes) 000539 1789597921.030585 MIRROR_RESYNC 4 mirrors 000540 1789597922.120843 mapread 0x004cc3 thru 0x0096b5 (0x49f3 bytes) 000542 1789597922.143604 write 0x021686 thru 0x029116 (0x7a91 bytes) 000544 1789597922.226233 punch from 0x00fa9e to 0x0173f9 (0x795c bytes) 000545 1789597922.243038 read 0x02124a thru 0x02788d (0x6644 bytes) 000546 1789597922.246928 punch from 0x01ed21 to 0x029116 (0xa3f6 bytes) 000549 1789597922.259955 punch from 0x00578a to 0x00ff56 (0xa7cd bytes) 000550 1789597922.265281 write 0x0222ed thru 0x032142 (0xfe56 bytes) 000552 1789597922.285347 mapwrite 0x01e1a9 thru 0x022828 (0x4680 bytes) 000553 1789597922.339981 punch from 0x0214f8 to 0x027e83 (0x698c bytes) 000555 1789597922.345420 punch from 0x01c977 to 0x02637e (0x9a08 bytes) 000557 1789597922.351553 read 0x004f8c thru 0x008807 (0x387c bytes) 000558 1789597922.369201 MIRROR_RESYNC 4 mirrors 000559 1789597923.916520 mapread 0x0079c4 thru 0x00f14e (0x778b bytes) 000561 1789597923.934557 mapread 0x01734e thru 0x0186fb (0x13ae bytes) 000563 1789597923.942771 mapwrite 0x00e618 thru 0x014b16 (0x64ff bytes) 000564 1789597924.112976 punch from 0x028cce to 0x02c118 (0x344b bytes) 000566 1789597924.121795 trunc from 0x032143 to 0x01e2e5 (0x13e5d bytes) 000567 1789597924.149586 mapwrite 0x01071c thru 0x01a93f (0xa224 bytes) 000568 1789597924.230283 write 0x0179e8 thru 0x01f8ca (0x7ee3 bytes) 000570 1789597924.233223 write 0x02cf09 thru 0x034b82 (0x7c7a bytes) 000572 1789597924.235327 trunc from 0x034b83 to 0x00362c (0x31556 bytes) 000573 1789597924.270769 write 0x017e15 thru 0x018bef (0xddb bytes) 000578 1789597924.276067 mapread 0x00940a thru 0x0136cf (0xa2c6 bytes) 000579 1789597924.300070 read 0x0001c4 thru 0x00ff8b (0xfdc8 bytes) 000580 1789597924.306881 write 0x009196 thru 0x016aa8 (0xd913 bytes) 000582 1789597924.308780 write 0x0039db thru 0x00b6a5 (0x7ccb bytes) 000583 1789597924.311170 mapwrite 0x00f730 thru 0x0174ba (0x7d8b bytes) 000585 1789597924.401306 trunc from 0x018bf0 to 0x0243e0 (0xb7f1 bytes) 000589 1789597924.420253 mapwrite 0x03cb6f thru 0x03ffff (0x3491 bytes) 000590 1789597924.505631 read 0x0105da thru 0x01ca01 (0xc428 bytes) 000592 1789597924.532299 mapwrite 0x0151f5 thru 0x017bf2 (0x29fe bytes) 000593 1789597924.582093 punch from 0x0333fb to 0x0386a2 (0x52a8 bytes) 000595 1789597924.588305 punch from 0x0314d6 to 0x03896a (0x7495 bytes) 000596 1789597924.592802 write 0x027209 thru 0x03144d (0xa245 bytes) 000597 1789597924.599279 mapread 0x00667e thru 0x006719 (0x9c bytes) 000598 1789597924.600226 mapread 0x01ed18 thru 0x02525d (0x6546 bytes) 000599 1789597924.602061 trunc from 0x040000 to 0x01aa6d (0x25592 bytes) 000600 1789597924.621867 punch from 0x00cf78 to 0x00e8ee (0x1977 bytes) 000601 1789597924.627132 mapwrite 0x00170f thru 0x0024bb (0xdad bytes) 000602 1789597924.682720 write 0x0211de thru 0x02e6a5 (0xd4c8 bytes) 000603 1789597924.689406 mapwrite 0x00e820 thru 0x01a05e (0xb83f bytes) 000604 1789597924.789952 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2e6a6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e6a6): Mirror 2: 0x208cb630 (stale) Mirror 3: 0x208cb630 (stale) Mirror 6: 0x208cb630 (stale) CRC-32 checksum value for chunk [0, 0x2e6a6): Mirror 1: 0x7739953d Mirror 7: 0x7739953d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e6a6: skipped 000606 1789597926.178983 MIRROR_RESYNC 5 mirrors 000607 1789597927.742383 write 0x0127c7 thru 0x02221d (0xfa57 bytes) 000608 1789597927.842224 mapwrite 0x01cb16 thru 0x028ca9 (0xc194 bytes) 000610 1789597927.934425 read 0x002ff4 thru 0x00ac00 (0x7c0d bytes) 000612 1789597927.953555 mapread 0x01df30 thru 0x026c6b (0x8d3c bytes) 000613 1789597927.954531 mapread 0x0282ad thru 0x02e6a5 (0x63f9 bytes) 000614 1789597927.976355 punch from 0x02574c to 0x028f71 (0x3826 bytes) 000616 1789597927.991876 mapwrite 0x02725e thru 0x032b06 (0xb8a9 bytes) 000619 1789597928.155381 trunc from 0x032b07 to 0x025a05 (0xd101 bytes) 000620 1789597928.188318 write 0x0058d5 thru 0x010714 (0xae40 bytes) 000621 1789597928.210652 write 0x0080d4 thru 0x00dd28 (0x5c55 bytes) 000622 1789597928.212384 mapread 0x01b4be thru 0x01dff6 (0x2b39 bytes) 000623 1789597928.222997 MIRROR_RESYNC 5 mirrors 000624 1789597929.637178 MIRROR_EXTEND to 6 mirrors 000626 1789597930.216998 read 0x0186f3 thru 0x01d86c (0x517a bytes) 000628 1789597930.239988 write 0x03b5c7 thru 0x03ffff (0x4a39 bytes) 000629 1789597930.299116 write 0x01cd6c thru 0x025708 (0x899d bytes) 000630 1789597930.308082 trunc from 0x040000 to 0x00992a (0x366d5 bytes) 000631 1789597930.331098 write 0x004bb5 thru 0x0056ae (0xafa bytes) 000633 1789597930.343510 read 0x0001f1 thru 0x00992a (0x973a bytes) 000634 1789597930.351170 punch from 0x006fb8 to 0x00992a (0x2973 bytes) 000636 1789597930.364514 MIRROR_RESYNC 6 mirrors 000637 1789597931.789280 punch from 0x001051 to 0x00992a (0x88da bytes) 000641 1789597931.865337 mapread 0x008b07 thru 0x00992a (0xe24 bytes) 000642 1789597931.875926 mapwrite 0x03fd71 thru 0x03ffff (0x28f bytes) 000645 1789597931.961017 mapwrite 0x00cd2c thru 0x01bf10 (0xf1e5 bytes) 000646 1789597932.048291 mapwrite 0x032d2d thru 0x03e343 (0xb617 bytes) 000647 1789597932.121213 write 0x019b63 thru 0x01bec8 (0x2366 bytes) 000649 1789597932.124109 read 0x03b9ec thru 0x03f172 (0x3787 bytes) 000650 1789597932.125720 read 0x014b54 thru 0x01e42c (0x98d9 bytes) 000652 1789597932.143039 MIRROR_RESYNC 6 mirrors 000653 1789597934.021475 trunc from 0x040000 to 0x003bee (0x3c411 bytes) 000654 1789597934.121915 mapwrite 0x026353 thru 0x02feb5 (0x9b63 bytes) 000655 1789597934.250259 punch from 0x028b69 to 0x02d395 (0x482d bytes) 000656 1789597934.259298 read 0x01386a thru 0x01994a (0x60e1 bytes) 000658 1789597934.268130 read 0x009f07 thru 0x009f86 (0x80 bytes) 000659 1789597934.279208 mapread 0x010754 thru 0x0188ab (0x8158 bytes) 000660 1789597934.280175 write 0x00d115 thru 0x016a33 (0x991f bytes) 000662 1789597934.283191 write 0x0338ff thru 0x03ffff (0xc701 bytes) 000665 1789597934.295231 write 0x011424 thru 0x018577 (0x7154 bytes) 000666 1789597934.300985 MIRROR_RESYNC 6 mirrors 000667 1789597937.874730 trunc from 0x040000 to 0x00ec94 (0x3136b bytes) 000668 1789597938.031962 punch from 0x0072dc to 0x00758d (0x2b2 bytes) 000669 1789597938.044553 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xec95): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xec95): Mirror 2: 0x15c0ff01 (stale) Mirror 3: 0x15c0ff01 (stale) Mirror 6: 0x15c0ff01 (stale) Mirror 7: 0x15c0ff01 (stale) Mirror 8: 0x15c0ff01 (stale) CRC-32 checksum value for chunk [0, 0xec95): Mirror 1: 0x15c0ff01 Mirror 9: 0x15c0ff01 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xec95: skipped 000670 1789597939.715437 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xec95): Mirror 3: 0x15c0ff01 (stale) Mirror 6: 0x15c0ff01 (stale) Mirror 7: 0x15c0ff01 (stale) Mirror 8: 0x15c0ff01 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xec95): Mirror 3: 0x15c0ff01 (stale) Mirror 6: 0x15c0ff01 (stale) Mirror 7: 0x15c0ff01 (stale) Mirror 8: 0x15c0ff01 (stale) CRC-32 checksum value for chunk [0, 0xec95): Mirror 1: 0x15c0ff01 Mirror 9: 0x15c0ff01 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xec95: skipped 000671 1789597940.513426 mapread 0x00bb52 thru 0x00ec94 (0x3143 bytes) 000673 1789597940.579434 punch from 0x00299e to 0x00a2fa (0x795d bytes) 000674 1789597940.704773 read 0x00c11e thru 0x00ec94 (0x2b77 bytes) 000675 1789597940.711039 trunc from 0x00ec95 to 0x0059ac (0x92e8 bytes) truncating to largest ever: 0x3ea65 000676 1789597940.753015 trunc from 0x0059ad to 0x03ea64 (0x390b8 bytes) 000677 1789597940.785091 mapread 0x0022f8 thru 0x011b27 (0xf830 bytes) 000678 1789597940.813981 read 0x023d9f thru 0x0319c0 (0xdc22 bytes) 000679 1789597940.830660 read 0x02c626 thru 0x030e6a (0x4845 bytes) 000681 1789597940.832316 mapread 0x02e297 thru 0x030866 (0x25d0 bytes) 000682 1789597940.833276 punch from 0x0202c2 to 0x02a736 (0xa475 bytes) 000683 1789597940.851971 punch from 0x03229c to 0x03a46c (0x81d1 bytes) 000686 1789597940.869674 write 0x028698 thru 0x02eaca (0x6433 bytes) 000687 1789597940.880274 mapread 0x00d0fc thru 0x01c606 (0xf50b bytes) 000688 1789597940.881256 read 0x018524 thru 0x01cbc2 (0x469f bytes) 000689 1789597940.881732 trunc from 0x03ea65 to 0x03bb47 (0x2f1d bytes) 000690 1789597940.920090 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3bb48): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb48): Mirror 3: 0xf719f580 (stale) Mirror 6: 0xf719f580 (stale) Mirror 7: 0xf719f580 (stale) Mirror 8: 0xf719f580 (stale) Mirror 9: 0x2883f3bc (stale) CRC-32 checksum value for chunk [0, 0x3bb48): Mirror 1: 0xb7054d07 Mirror 10: 0xb7054d07 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb48: skipped 000692 1789597942.735725 trunc from 0x03bb48 to 0x039989 (0x21be bytes) 000693 1789597942.843244 mapwrite 0x01a66f thru 0x01e3d7 (0x3d69 bytes) 000694 1789597942.895301 punch from 0x006594 to 0x012ab3 (0xc520 bytes) 000695 1789597942.899684 punch from 0x003158 to 0x0115e7 (0xe490 bytes) 000696 1789597942.907522 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3998a): Mirror 3: 0xcf237417 (stale) Mirror 6: 0xcf237417 (stale) Mirror 7: 0xcf237417 (stale) Mirror 9: 0x104336c2 (stale) Mirror 10: 0x59ea4973 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3998a): Mirror 3: 0xcf237417 (stale) Mirror 6: 0xcf237417 (stale) Mirror 7: 0xcf237417 (stale) Mirror 9: 0x104336c2 (stale) Mirror 10: 0x59ea4973 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3998a: skipped 000697 1789597943.519144 mapwrite 0x01afa3 thru 0x01ff2b (0x4f89 bytes) 000698 1789597943.651519 mapread 0x0251b0 thru 0x02d041 (0x7e92 bytes) 000699 1789597943.663963 read 0x00b8f3 thru 0x00c294 (0x9a2 bytes) 000700 1789597943.677415 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3998a): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3998a): Mirror 3: 0xcf237417 (stale) Mirror 6: 0xcf237417 (stale) Mirror 7: 0xcf237417 (stale) Mirror 9: 0x104336c2 (stale) Mirror 10: 0x59ea4973 (stale) CRC-32 checksum value for chunk [0, 0x3998a): Mirror 1: 0xdbf4dc88 Mirror 11: 0xdbf4dc88 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3998a: skipped 000701 1789597945.738578 mapwrite 0x0355dd thru 0x03ffff (0xaa23 bytes) 000702 1789597946.122998 mapwrite 0x032587 thru 0x03ffff (0xda79 bytes) 000704 1789597946.220313 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1eb98641 (stale) Mirror 6: 0x1eb98641 (stale) Mirror 9: 0x7352b111 (stale) Mirror 10: 0x9f70c845 (stale) Mirror 11: 0xe2e2419f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1eb98641 (stale) Mirror 6: 0x1eb98641 (stale) Mirror 9: 0x7352b111 (stale) Mirror 10: 0x9f70c845 (stale) Mirror 11: 0xe2e2419f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000705 1789597946.856992 read 0x017343 thru 0x020915 (0x95d3 bytes) 000706 1789597946.867960 read 0x03f630 thru 0x03ffff (0x9d0 bytes) 000709 1789597946.886287 punch from 0x00eaaa to 0x018796 (0x9ced bytes) 000712 1789597946.893394 write 0x031946 thru 0x03b5ee (0x9ca9 bytes) 000713 1789597946.895274 mapread 0x028726 thru 0x02d048 (0x4923 bytes) 000714 1789597946.896378 punch from 0x01308b to 0x022592 (0xf508 bytes) 000716 1789597946.910925 write 0x01efe4 thru 0x025e88 (0x6ea5 bytes) 000717 1789597946.920073 mapread 0x01d391 thru 0x01e1c0 (0xe30 bytes) 000718 1789597946.932858 trunc from 0x040000 to 0x03deb4 (0x214b bytes) 000719 1789597946.956745 punch from 0x0389bf to 0x03deb4 (0x54f6 bytes) 000721 1789597946.979702 punch from 0x004fdc to 0x00ca62 (0x7a87 bytes) 000722 1789597946.985930 mapread 0x026d1d thru 0x029811 (0x2af5 bytes) 000724 1789597946.996028 write 0x01b0cd thru 0x01b8e3 (0x817 bytes) 000725 1789597947.021068 mapread 0x037eae thru 0x03c301 (0x4454 bytes) 000726 1789597947.059098 mapwrite 0x00afb0 thru 0x010f54 (0x5fa5 bytes) 000727 1789597947.240801 mapwrite 0x016cbd thru 0x02219e (0xb4e2 bytes) 000728 1789597947.340171 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3deb5): Mirror 3: 0x2c1c3a82 (stale) Mirror 9: 0x66f6afa4 (stale) Mirror 10: 0x173b4f4 (stale) Mirror 11: 0x5efc3930 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3deb5): Mirror 3: 0x2c1c3a82 (stale) Mirror 9: 0x66f6afa4 (stale) Mirror 10: 0x173b4f4 (stale) Mirror 11: 0x5efc3930 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3deb5: skipped 000729 1789597947.943371 mapread 0x03b9b3 thru 0x03deb4 (0x2502 bytes) 000732 1789597947.971454 read 0x0105f6 thru 0x01c958 (0xc363 bytes) 000733 1789597947.987022 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3deb5): Mirror 9: 0x66f6afa4 (stale) Mirror 10: 0x173b4f4 (stale) Mirror 11: 0x5efc3930 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3deb5): Mirror 9: 0x66f6afa4 (stale) Mirror 10: 0x173b4f4 (stale) Mirror 11: 0x5efc3930 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3deb5: skipped 000734 1789597948.423700 mapwrite 0x03d546 thru 0x03ffff (0x2aba bytes) 000735 1789597948.502089 read 0x033afd thru 0x03548c (0x1990 bytes) 000737 1789597948.509089 write 0x006d86 thru 0x00b36b (0x45e6 bytes) 000738 1789597948.518022 mapwrite 0x030cc0 thru 0x034c69 (0x3faa bytes) 000739 1789597948.568215 mapwrite 0x0179f1 thru 0x024075 (0xc685 bytes) 000740 1789597948.629462 read 0x01b7f3 thru 0x0290e2 (0xd8f0 bytes) 000741 1789597948.630249 mapwrite 0x001c36 thru 0x004e48 (0x3213 bytes) 000742 1789597948.679902 mapread 0x01fd73 thru 0x02d6bd (0xd94b bytes) 000745 1789597948.684983 read 0x003810 thru 0x0137cd (0xffbe bytes) 000746 1789597948.686080 trunc from 0x040000 to 0x032f60 (0xd09f bytes) 000747 1789597948.708091 read 0x00367f thru 0x00faed (0xc46f bytes) 000748 1789597948.708972 mapwrite 0x02e180 thru 0x03a8a8 (0xc729 bytes) 000749 1789597948.781288 mapread 0x01a252 thru 0x01ea9d (0x484c bytes) 000751 1789597948.783299 punch from 0x02e969 to 0x0353a1 (0x6a39 bytes) 000752 1789597948.789405 mapwrite 0x020b1e thru 0x021e58 (0x133b bytes) 000754 1789597948.849006 MIRROR_RESYNC 4 mirrors 000755 1789597951.591543 read 0x03221c thru 0x034135 (0x1f1a bytes) 000757 1789597951.601881 trunc from 0x03a8a9 to 0x0248c7 (0x15fe1 bytes) 000758 1789597951.725995 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x248c8): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x248c8): Mirror 9: 0x37b9a9c4 (stale) Mirror 10: 0x37b9a9c4 (stale) Mirror 11: 0x37b9a9c4 (stale) CRC-32 checksum value for chunk [0, 0x248c8): Mirror 1: 0x37b9a9c4 Mirror 12: 0x37b9a9c4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x248c8: skipped 000759 1789597953.249566 mapread 0x0180e9 thru 0x01dcbd (0x5bd5 bytes) 000760 1789597953.294209 read 0x023c17 thru 0x0248c7 (0xcb1 bytes) 000761 1789597953.303323 punch from 0x022d23 to 0x0248c7 (0x1ba5 bytes) 000762 1789597953.450464 punch from 0x002bb3 to 0x0114da (0xe928 bytes) 000763 1789597953.461344 read 0x0122a6 thru 0x01e754 (0xc4af bytes) 000764 1789597953.486370 mapwrite 0x01ec08 thru 0x027612 (0x8a0b bytes) 000766 1789597953.659724 write 0x021414 thru 0x023c5f (0x284c bytes) 000768 1789597953.665116 trunc from 0x027613 to 0x01420b (0x13407 bytes) 000770 1789597953.718191 write 0x00c8d7 thru 0x0125ed (0x5d17 bytes) 000774 1789597953.743947 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1420c): Mirror 9: 0xacc71eb6 (stale) Mirror 11: 0xacc71eb6 (stale) Mirror 12: 0xacc71eb6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1420c): Mirror 9: 0xacc71eb6 (stale) Mirror 11: 0xacc71eb6 (stale) Mirror 12: 0xacc71eb6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1420c: skipped 000775 1789597954.361012 read 0x008605 thru 0x00aa55 (0x2451 bytes) 000776 1789597954.381499 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1420c): Mirror 9: 0xacc71eb6 (stale) Mirror 12: 0xacc71eb6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1420c): Mirror 9: 0xacc71eb6 (stale) Mirror 12: 0xacc71eb6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1420c: skipped 000777 1789597954.906960 read 0x00dca2 thru 0x012302 (0x4661 bytes) 000778 1789597954.930352 mapread 0x00b462 thru 0x011cd7 (0x6876 bytes) 000779 1789597954.940980 mapread 0x0077a8 thru 0x0139cd (0xc226 bytes) 000781 1789597954.966603 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1420c): Mirror 9: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1420c): Mirror 9: 0xacc71eb6 (stale) Mirror 12: 0xacc71eb6 (stale) CRC-32 checksum value for chunk [0, 0x1420c): Mirror 1: 0x7dde09b6 Mirror 13: 0x7dde09b6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1420c: skipped 000783 1789597957.157372 mapwrite 0x00af53 thru 0x00bc5b (0xd09 bytes) 000785 1789597957.436296 mapwrite 0x036cf3 thru 0x03ffff (0x930d bytes) 000786 1789597957.605150 mapread 0x003244 thru 0x0070e2 (0x3e9f bytes) 000787 1789597957.619688 read 0x016630 thru 0x01afbf (0x4990 bytes) 000788 1789597957.629363 mapwrite 0x016f4e thru 0x01d56d (0x6620 bytes) 000791 1789597957.766888 write 0x021d7c thru 0x02e40a (0xc68f bytes) 000793 1789597957.790647 read 0x00ef47 thru 0x01c6ca (0xd784 bytes) 000795 1789597957.791745 trunc from 0x040000 to 0x0204d2 (0x1fb2d bytes) 000796 1789597957.839208 mapread 0x0187f0 thru 0x01e51d (0x5d2e bytes) 000797 1789597957.846881 read 0x01f78c thru 0x0204d2 (0xd47 bytes) 000798 1789597957.852317 mapwrite 0x00a30d thru 0x018329 (0xe01d bytes) 000800 1789597958.098379 punch from 0x013d51 to 0x01788f (0x3b3f bytes) 000802 1789597958.110010 read 0x012d13 thru 0x0185b1 (0x589f bytes) 000803 1789597958.126538 read 0x007379 thru 0x00fd4d (0x89d5 bytes) 000804 1789597958.128801 punch from 0x011f15 to 0x0190cd (0x71b9 bytes) 000805 1789597958.144481 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x204d3): Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x204d3): Mirror 9: 0xb0f84635 (stale) Mirror 12: 0xb0f84635 (stale) Mirror 13: 0x6021f5a9 (stale) CRC-32 checksum value for chunk [0, 0x204d3): Mirror 1: 0x538b0e4e Mirror 14: 0x538b0e4e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x204d3: skipped 000808 1789597960.547244 write 0x027123 thru 0x034764 (0xd642 bytes) 000810 1789597960.711294 mapwrite 0x0087c3 thru 0x00f2a4 (0x6ae2 bytes) 000812 1789597960.901151 trunc from 0x034765 to 0x00b489 (0x292db bytes) 000814 1789597960.983966 mapwrite 0x027647 thru 0x02f1b6 (0x7b70 bytes) 000816 1789597961.165169 read 0x019a65 thru 0x025873 (0xbe0f bytes) 000817 1789597961.181058 read 0x0238ee thru 0x02f1b6 (0xb8c9 bytes) 000818 1789597961.181723 trunc from 0x02f1b7 to 0x02dcf8 (0x14be bytes) 000819 1789597961.218846 read 0x0100e0 thru 0x01235e (0x227f bytes) 000821 1789597961.256792 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2dcf9): Mirror 9: 0xa7bd7d95 (stale) Mirror 12: 0xb9456a12 (stale) Mirror 13: 0x2a80adbc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dcf9): Mirror 9: 0xa7bd7d95 (stale) Mirror 12: 0xb9456a12 (stale) Mirror 13: 0x2a80adbc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dcf9: skipped 000822 1789597962.018044 write 0x00cbfb thru 0x015bbe (0x8fc4 bytes) 000823 1789597962.040241 write 0x028ed2 thru 0x034f1a (0xc049 bytes) 000824 1789597962.048076 trunc from 0x034f1b to 0x001ff4 (0x32f26 bytes) 000825 1789597962.072639 read 0x000a7b thru 0x001ff4 (0x157a bytes) 000826 1789597962.078885 read 0x0003d4 thru 0x001ff4 (0x1c21 bytes) 000827 1789597962.080762 mapread 0x000cf4 thru 0x001ff4 (0x1301 bytes) 000828 1789597962.082217 mapread 0x001857 thru 0x001ff4 (0x79e bytes) 000830 1789597962.090008 trunc from 0x001ff5 to 0x010897 (0xe8a3 bytes) 000831 1789597962.116863 mapread 0x001857 thru 0x009caf (0x8459 bytes) 000832 1789597962.130980 mapwrite 0x03fbf3 thru 0x03ffff (0x40d bytes) 000833 1789597962.261691 punch from 0x01b69c to 0x029fe3 (0xe948 bytes) 000834 1789597962.275513 mapread 0x01fced thru 0x01ff00 (0x214 bytes) 000835 1789597962.286617 mapwrite 0x03f52b thru 0x03ffff (0xad5 bytes) 000836 1789597962.390198 write 0x015811 thru 0x017494 (0x1c84 bytes) 000837 1789597962.391950 trunc from 0x040000 to 0x0304ea (0xfb15 bytes) 000838 1789597962.437075 trunc from 0x0304eb to 0x003538 (0x2cfb2 bytes) 000839 1789597962.463652 mapwrite 0x019de9 thru 0x01f926 (0x5b3e bytes) 000842 1789597962.732147 mapread 0x00a992 thru 0x0133a7 (0x8a16 bytes) 000843 1789597962.855520 MIRROR_RESYNC 4 mirrors 000844 1789597964.829229 write 0x01c64e thru 0x02c051 (0xfa04 bytes) 000846 1789597964.946097 mapwrite 0x039013 thru 0x03ffff (0x6fed bytes) 000847 1789597965.145063 mapwrite 0x005fb4 thru 0x0065da (0x627 bytes) 000848 1789597965.250364 mapwrite 0x0337cb thru 0x039b23 (0x6359 bytes) 000849 1789597965.444200 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x89a4be50 (stale) Mirror 13: 0x89a4be50 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x89a4be50 (stale) Mirror 13: 0x89a4be50 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000850 1789597966.125210 read 0x0190f0 thru 0x01d5b2 (0x44c3 bytes) 000851 1789597966.143294 read 0x01eccb thru 0x025f88 (0x72be bytes) 000852 1789597966.155987 read 0x035809 thru 0x03ffff (0xa7f7 bytes) 000853 1789597966.156613 read 0x01379f thru 0x0165cf (0x2e31 bytes) 000854 1789597966.157157 trunc from 0x040000 to 0x014aa8 (0x2b557 bytes) 000855 1789597966.172999 punch from 0x00a132 to 0x014aa8 (0xa977 bytes) 000856 1789597966.185865 trunc from 0x014aa9 to 0x01c32b (0x7883 bytes) 000862 1789597966.236129 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c32c): Mirror 13: 0xef293f3a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c32c): Mirror 13: 0xef293f3a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c32c: skipped 000864 1789597966.744235 MIRROR_RESYNC 2 mirrors 000865 1789597967.759255 MIRROR_EXTEND to 3 mirrors 000866 1789597968.960235 write 0x0061f8 thru 0x008f00 (0x2d09 bytes) 000867 1789597969.133609 mapwrite 0x02d748 thru 0x03b1f6 (0xdaaf bytes) 000868 1789597969.315401 write 0x01ec3d thru 0x0249ed (0x5db1 bytes) 000869 1789597969.338186 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b1f7): Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b1f7): Mirror 13: 0x1075b19f (stale) Mirror 14: 0x1075b19f (stale) CRC-32 checksum value for chunk [0, 0x3b1f7): Mirror 1: 0xc502a92d Mirror 15: 0xc502a92d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b1f7: skipped 000870 1789597971.032553 mapwrite 0x013190 thru 0x015feb (0x2e5c bytes) 000871 1789597971.306740 punch from 0x009056 to 0x018baa (0xfb55 bytes) 000872 1789597971.314114 trunc from 0x03b1f7 to 0x009633 (0x31bc3 bytes) 000874 1789597971.334784 write 0x01d81f thru 0x0280f0 (0xa8d2 bytes) 000875 1789597971.342313 MIRROR_RESYNC 4 mirrors 000876 1789597972.707714 read 0x009e8b thru 0x01223d (0x83b3 bytes) 000877 1789597972.721164 mapread 0x027e5e thru 0x0280f0 (0x293 bytes) 000879 1789597972.731986 trunc from 0x0280f1 to 0x02de21 (0x5d31 bytes) 000881 1789597972.850510 write 0x011dd3 thru 0x018310 (0x653e bytes) 000882 1789597972.881506 MIRROR_RESYNC 4 mirrors 000885 1789597974.836755 mapread 0x003df0 thru 0x010d8a (0xcf9b bytes) 000886 1789597974.867195 punch from 0x01018b to 0x019a20 (0x9896 bytes) 000888 1789597975.008241 trunc from 0x02de22 to 0x0328bd (0x4a9c bytes) 000889 1789597975.046914 punch from 0x02af98 to 0x0328bd (0x7926 bytes) 000890 1789597975.062139 write 0x036c0d thru 0x0378db (0xccf bytes) 000893 1789597975.076562 trunc from 0x0378dc to 0x0268d1 (0x1100a bytes) 000894 1789597975.136524 mapwrite 0x01c8e4 thru 0x02a006 (0xd723 bytes) 000895 1789597975.392048 mapread 0x00c704 thru 0x00e2a5 (0x1ba2 bytes) 000896 1789597975.409171 write 0x004e97 thru 0x0133f8 (0xe562 bytes) 000897 1789597975.448948 punch from 0x025e82 to 0x0265e3 (0x762 bytes) 000900 1789597975.495931 mapwrite 0x015a97 thru 0x021047 (0xb5b1 bytes) 000901 1789597975.592951 punch from 0x00dda6 to 0x01994e (0xbba9 bytes) 000902 1789597975.605970 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2a007): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a007): Mirror 13: 0x57cd661d (stale) Mirror 14: 0x57cd661d (stale) Mirror 15: 0x57cd661d (stale) CRC-32 checksum value for chunk [0, 0x2a007): Mirror 1: 0xb04a5800 Mirror 16: 0xb04a5800 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a007: skipped 000903 1789597977.464983 mapread 0x00d2e9 thru 0x0118da (0x45f2 bytes) 000904 1789597977.501610 trunc from 0x02a007 to 0x00788c (0x2277a bytes) 000905 1789597977.699872 mapwrite 0x001325 thru 0x00496d (0x3649 bytes) 000906 1789597977.802838 write 0x009d85 thru 0x00eca9 (0x4f25 bytes) 000909 1789597977.821622 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xecaa): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xecaa): Mirror 13: 0x7a2d2111 (stale) Mirror 14: 0x7a2d2111 (stale) Mirror 15: 0x7a2d2111 (stale) Mirror 16: 0xbbd56dc7 (stale) CRC-32 checksum value for chunk [0, 0xecaa): Mirror 1: 0xd87ce5f Mirror 17: 0xd87ce5f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xecaa: skipped 000910 1789597979.367992 read 0x003a73 thru 0x00eca9 (0xb237 bytes) 000911 1789597979.396013 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xecaa): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xecaa): Mirror 13: 0x7a2d2111 (stale) Mirror 14: 0x7a2d2111 (stale) Mirror 15: 0x7a2d2111 (stale) Mirror 16: 0xbbd56dc7 (stale) CRC-32 checksum value for chunk [0, 0xecaa): Mirror 1: 0xd87ce5f Mirror 17: 0xd87ce5f Mirror 18: 0xd87ce5f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xecaa: skipped 000913 1789597980.529513 punch from 0x008e24 to 0x00d9b7 (0x4b94 bytes) 000914 1789597980.650424 mapwrite 0x035aff thru 0x03a5ce (0x4ad0 bytes) 000915 1789597980.779875 read 0x011709 thru 0x017a8a (0x6382 bytes) 000916 1789597980.789996 punch from 0x035125 to 0x038efa (0x3dd6 bytes) 000917 1789597980.800536 punch from 0x00218c to 0x0105a3 (0xe418 bytes) 000918 1789597980.814794 punch from 0x01652b to 0x01bb11 (0x55e7 bytes) 000920 1789597980.823207 mapread 0x008589 thru 0x012c3a (0xa6b2 bytes) 000922 1789597980.847993 read 0x031cc5 thru 0x036270 (0x45ac bytes) 000923 1789597980.870560 mapread 0x039a83 thru 0x03a5ce (0xb4c bytes) 000924 1789597980.872860 mapread 0x003a9e thru 0x0082ec (0x484f bytes) 000925 1789597980.875201 read 0x0187e3 thru 0x019271 (0xa8f bytes) 000926 1789597980.877221 read 0x0330a3 thru 0x03a5ce (0x752c bytes) 000927 1789597980.888488 mapread 0x000c41 thru 0x00769d (0x6a5d bytes) 000929 1789597980.895562 read 0x02ad4a thru 0x02d5b6 (0x286d bytes) 000930 1789597980.900720 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3a5cf): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a5cf): Mirror 13: 0xa5e328d6 (stale) Mirror 14: 0xa5e328d6 (stale) Mirror 15: 0xa5e328d6 (stale) Mirror 16: 0x5e03ab42 (stale) Mirror 17: 0x2db3dd3f (stale) Mirror 18: 0x2db3dd3f (stale) CRC-32 checksum value for chunk [0, 0x3a5cf): Mirror 1: 0xfb79d8ba Mirror 19: 0xfb79d8ba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a5cf: skipped 000932 1789597982.837635 mapwrite 0x03591f thru 0x036e26 (0x1508 bytes) 000933 1789597983.043518 MIRROR_RESYNC 8 mirrors 000934 1789597985.744582 mapwrite 0x012f7f thru 0x01ac54 (0x7cd6 bytes) 000935 1789597985.969793 trunc from 0x03a5cf to 0x028d79 (0x11855 bytes) 000936 1789597986.018391 punch from 0x01e130 to 0x022956 (0x4827 bytes) 000937 1789597986.023148 punch from 0x00e639 to 0x010e1f (0x27e7 bytes) 000938 1789597986.038922 mapread 0x0064e5 thru 0x00c401 (0x5f1d bytes) 000940 1789597986.054075 write 0x006179 thru 0x0082de (0x2166 bytes) 000942 1789597986.066758 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x28d7a): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28d7a): Mirror 13: 0x7ab7e05 (stale) Mirror 14: 0x7ab7e05 (stale) Mirror 15: 0x7ab7e05 (stale) Mirror 16: 0x7ab7e05 (stale) Mirror 17: 0x7ab7e05 (stale) Mirror 18: 0x7ab7e05 (stale) Mirror 19: 0x7ab7e05 (stale) CRC-32 checksum value for chunk [0, 0x28d7a): Mirror 1: 0xe9fab307 Mirror 20: 0xe9fab307 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28d7a: skipped 000944 1789597987.515631 mapwrite 0x036e0a thru 0x03bfb3 (0x51aa bytes) 000945 1789597987.875523 write 0x013c0f thru 0x022a09 (0xedfb bytes) 000947 1789597987.897465 read 0x03962f thru 0x039972 (0x344 bytes) 000948 1789597987.899291 mapwrite 0x00bfa8 thru 0x01a2ed (0xe346 bytes) 000949 1789597988.047658 mapwrite 0x03058d thru 0x03598e (0x5402 bytes) 000950 1789597988.184101 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bfb4): Mirror 14: 0xf4752cda (stale) Mirror 15: 0xf4752cda (stale) Mirror 16: 0xf4752cda (stale) Mirror 17: 0xf4752cda (stale) Mirror 18: 0xf4752cda (stale) Mirror 19: 0xf4752cda (stale) Mirror 20: 0xbecbc81d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bfb4): Mirror 14: 0xf4752cda (stale) Mirror 15: 0xf4752cda (stale) Mirror 16: 0xf4752cda (stale) Mirror 17: 0xf4752cda (stale) Mirror 18: 0xf4752cda (stale) Mirror 19: 0xf4752cda (stale) Mirror 20: 0xbecbc81d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bfb4: skipped 000951 1789597988.740026 mapwrite 0x00bd1a thru 0x01759f (0xb886 bytes) 000953 1789597988.824844 mapwrite 0x005a91 thru 0x01435e (0xe8ce bytes) 000954 1789597988.913705 read 0x02b112 thru 0x039a7c (0xe96b bytes) 000955 1789597988.928454 mapread 0x01e4b7 thru 0x023793 (0x52dd bytes) 000958 1789597988.944714 trunc from 0x03bfb4 to 0x00a162 (0x31e51 bytes) 000960 1789597988.963651 mapread 0x002bba thru 0x00a162 (0x75a9 bytes) 000961 1789597988.965490 mapread 0x0066d6 thru 0x00a162 (0x3a8d bytes) 000962 1789597988.966446 punch from 0x005c86 to 0x00a162 (0x44dd bytes) 000963 1789597988.971932 punch from 0x006aad to 0x0078b6 (0xe0a bytes) 000965 1789597988.989572 mapwrite 0x014749 thru 0x0171f2 (0x2aaa bytes) 000967 1789597989.059301 read 0x0057ea thru 0x00ec9d (0x94b4 bytes) 000968 1789597989.067133 trunc from 0x0171f3 to 0x020f3f (0x9d4d bytes) 000969 1789597989.085302 trunc from 0x020f40 to 0x015fe8 (0xaf57 bytes) 000970 1789597989.104019 mapwrite 0x037add thru 0x03ffff (0x8523 bytes) 000971 1789597989.186050 trunc from 0x040000 to 0x0268bb (0x19744 bytes) 000974 1789597989.209083 mapwrite 0x01bf86 thru 0x02bbed (0xfc68 bytes) 000975 1789597989.283644 mapwrite 0x038618 thru 0x03c569 (0x3f52 bytes) 000976 1789597989.361595 trunc from 0x03c56a to 0x01dc5e (0x1e90b bytes) 000977 1789597989.382935 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1dc5f): Mirror 14: 0xf47464c3 (stale) Mirror 15: 0xf47464c3 (stale) Mirror 16: 0xf47464c3 (stale) Mirror 18: 0xf47464c3 (stale) Mirror 19: 0xf47464c3 (stale) Mirror 20: 0x791a4e26 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dc5f): Mirror 14: 0xf47464c3 (stale) Mirror 15: 0xf47464c3 (stale) Mirror 16: 0xf47464c3 (stale) Mirror 18: 0xf47464c3 (stale) Mirror 19: 0xf47464c3 (stale) Mirror 20: 0x791a4e26 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dc5f: skipped 000978 1789597989.800908 mapread 0x00904c thru 0x00b5c5 (0x257a bytes) 000979 1789597989.820753 punch from 0x008f4b to 0x00d782 (0x4838 bytes) 000980 1789597989.828236 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1dc5f): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dc5f): Mirror 14: 0xf47464c3 (stale) Mirror 15: 0xf47464c3 (stale) Mirror 16: 0xf47464c3 (stale) Mirror 18: 0xf47464c3 (stale) Mirror 19: 0xf47464c3 (stale) Mirror 20: 0x791a4e26 (stale) CRC-32 checksum value for chunk [0, 0x1dc5f): Mirror 1: 0x1f9c5479 Mirror 21: 0x1f9c5479 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dc5f: skipped 000981 1789597991.227386 mapwrite 0x03bd6b thru 0x03ffff (0x4295 bytes) 000982 1789597991.357603 read 0x001e32 thru 0x0032c1 (0x1490 bytes) 000984 1789597991.363929 read 0x02c36d thru 0x02f3f5 (0x3089 bytes) 000985 1789597991.374767 read 0x037089 thru 0x03ffff (0x8f77 bytes) 000986 1789597991.375441 mapwrite 0x004a16 thru 0x00bd37 (0x7322 bytes) 000987 1789597991.422323 punch from 0x007bc9 to 0x00ecac (0x70e4 bytes) 000988 1789597991.426864 trunc from 0x040000 to 0x02d665 (0x1299a bytes) 000989 1789597991.443266 trunc from 0x02d666 to 0x028a97 (0x4bce bytes) 000992 1789597991.459507 mapread 0x00d6db thru 0x01bd56 (0xe67c bytes) 000993 1789597991.467642 mapread 0x018fae thru 0x025656 (0xc6a9 bytes) 000994 1789597991.468759 mapread 0x007e72 thru 0x015541 (0xd6d0 bytes) 000995 1789597991.470339 mapread 0x00aedd thru 0x01ad1a (0xfe3e bytes) 000996 1789597991.471695 read 0x0237a7 thru 0x028a97 (0x52f1 bytes) 000997 1789597991.472508 punch from 0x0216ff to 0x028a97 (0x7399 bytes) 000999 1789597991.477261 punch from 0x016700 to 0x023f66 (0xd867 bytes) 001000 1789597991.481495 trunc from 0x028a98 to 0x020b9d (0x7efa bytes) All operations completed A-OK! PASS 70a (121s) == sanity-flr test 71: check mirror extend parent fid ==== 18:33:17 (1789597997) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 18:33:21 (1789598001) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.271091 s, 7.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279862 s, 3.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f8:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 18:33:30 (1789598010) Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre2 Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 3419776 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1548352 .. Extending file size to 6246720 .. Extending file size to 7054624 .. Extending file size to 5612128 .. Extending file size to 8107392 .. Extending file size to 8639712 .. Extending file size to 7358368 .. Extending file size to 7853408 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8391936 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9031104 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57491 57493 57495 57496 57500 done 192.168.203.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg337-client.virtnet /mnt/lustre2 (opts:) 192.168.203.137@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg337-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 18:34:53 (1789598093) Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre2 Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6448992 .. Extending file size to 3983104 .. done Extending file size to 7289280 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7913696 .. Extending file size to 7497408 .. Extending file size to 8810976 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8194400 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8617600 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9288896 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8842912 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9303520 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 58877 58878 58881 58882 58884 192.168.203.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg337-client.virtnet /mnt/lustre2 (opts:) 192.168.203.137@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg337-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 18:36:11 (1789598171) Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.142681 s, 14.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.381141 s, 5.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x1ccc87dc (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg337-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 18:36:30 (1789598190) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0913719 s, 23.0 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 18:36:37 (1789598197) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x211:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x30d:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 18:36:45 (1789598205) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.186118 s, 11.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 18:36:55 (1789598215) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.294182 s, 7.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x310:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x310:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x311:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 18:37:04 (1789598224) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.545561 s, 3.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x312:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x312:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x313:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 18:37:40 (1789598260) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg337-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145085 s, 7.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x315:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x314:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x315:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x314:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } PASS 204d (61s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 18:38:41 (1789598321) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg337-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.218811 s, 9.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x316:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x316:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (58s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 18:39:39 (1789598379) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg337-server mds-ost sync done. Creating new pool test_204f oleg337-server: Pool lustre.test_204f created Adding targets to pool oleg337-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg337-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.198005 s, 10.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x317:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x317:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg337-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg337-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg337-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 18:40:59 (1789598459) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 18:41:09 (1789598469) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 18:41:16 (1789598476) oleg337-server: Pool lustre.test_206 created Adding targets to pool oleg337-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg337-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg337-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg337-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg337-server: Pool lustre.test_206 destroyed PASS 206 (22s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 18:41:38 (1789598498) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691229 s, 15.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462995 s, 22.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x51f30eee Mirror 3: 0x51f30eee PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 18:41:49 (1789598509) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 18:41:52 (1789598512) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 18:41:55 (1789598515) Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.143836 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x17f0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1d10000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1d10000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1d10000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x17b0000 Read this (4 bytes): BBBB 192.168.203.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg337-client.virtnet /mnt/lustre2 (opts:) PASS 209a (35s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 18:42:30 (1789598550) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0197021 s, 416 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 18:42:38 (1789598558) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950249 s, 11.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x236 lmm_fid: [0x200000404:0x236:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x236 lmm_fid: [0x200000404:0x236:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x322:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834486 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 18:42:40.000000000 -0400 Modify: 2026-09-16 18:42:40.000000000 -0400 Change: 2026-09-16 18:42:42.000000000 -0400 Birth: 2026-09-16 18:42:40.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 18:42:48 (1789598568) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 18:42:52 (1789598572) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.562741 s, 72.8 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.06513 s, 62.9 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 16 18:42 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x238:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 16 18:42 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 18:43:01 (1789598581) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 18:43:10 (1789598590) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1782 sec =========== 18:43:19 (1789598599) === sanity-flr: start cleanup 18:43:21 (1789598601) === === sanity-flr: finish cleanup 18:43:28 (1789598608) ===