-----============= acceptance-small: sanity-flr ============----- Mon Jul 13 14:57:09 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg322-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 14:57:26 (1783969046) === oleg322-client.virtnet: executing check_config_client /mnt/lustre oleg322-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg322-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff985051088000.idle_timeout=debug osc.lustre-OST0001-osc-ffff985051088000.idle_timeout=debug disable quota as required oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 14:57:59 (1783969079) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7594] == sanity-flr test 0a: lfs mirror create with -N option == 14:58:02 (1783969082) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 14:58:18 (1783969098) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 14:58:21 (1783969101) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 14:58:25 (1783969105) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 14:58:50 (1783969130) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 14:58:54 (1783969134) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 14:58:58 (1783969138) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff985051088000.stats=clear osc.lustre-OST0001-osc-ffff985051088000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 14:59:06 (1783969146) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 14:59:18 (1783969158) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 14:59:28 (1783969168) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 14:59:31 (1783969171) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 14:59:41 (1783969181) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 14:59:53 (1783969193) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:00:01 (1783969201) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:00:04 (1783969204) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:00:13 (1783969213) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (9s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 15:00:26 (1783969226) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:00:35 (1783969235) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 1.24796 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231051 s, 4.5 MB/s PASS 21 (15s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:00:50 (1783969250) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 0.854967 s, 22.1 MB/s PASS 22 (11s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 15:01:01 (1783969261) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:01:13 (1783969273) Stopping /mnt/lustre-ost1 (opts:) on oleg322-server oleg322-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff985051088000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff985051088000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg322-server: oleg322-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg322-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg322-server oleg322-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff985051088000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff985051088000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg322-server: oleg322-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg322-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (71s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 15:02:24 (1783969344) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.35177 s, 17.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg322-server oleg322-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff985051088000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff985051088000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg322-server: oleg322-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg322-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (43s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:03:07 (1783969387) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg322-server oleg322-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff985051088000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff985051088000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg322-server: oleg322-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg322-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg322-server oleg322-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff985051088000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff985051088000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg322-server: oleg322-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg322-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (82s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:04:29 (1783969469) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.18337 s, 10.1 MB/s -rw-r--r-- 1 root root 52428800 Jul 13 15:04 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.222s user 0m0.002s sys 0m0.156s Stopping /mnt/lustre-ost1 (opts:) on oleg322-server oleg322-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff985051088000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff985051088000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.201s user 0m0.001s sys 0m0.199s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg322-server: oleg322-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg322-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg322-server oleg322-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff985051088000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff985051088000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.451s user 0m0.001s sys 0m0.191s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg322-server: oleg322-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg322-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg322-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff985051088000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff985051088000.ost_server_uuid in FULL state after 0 sec PASS 33b (94s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:06:03 (1783969563) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:06:07 (1783969567) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:06:11 (1783969571) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 15:06:14 (1783969574) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 15:06:22 (1783969582) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.730342 s, 17.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:06:34 (1783969594) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374338 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419639 s, 25.0 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:06:42 (1783969602) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:06:50 (1783969610) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00135783 s, 754 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0010667 s, 960 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00425546 s, 241 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00577037 s, 177 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 15:07:01 (1783969621) create test files with size 15736117 .. d2a7aa9fbd58987351708aec973b8325 - d6f19b77f69d17c4da2d58e37cafe856 - 2ca0f25c3025197ba5a14a5cdc881a1f - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg322-client.virtnet /mnt/lustre (opts:) Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 15:07:24 (1783969644) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (23s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 15:07:47 (1783969667) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 15:07:56 (1783969676) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119915 s, 8.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606462 s, 17.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 15:08:05 (1783969685) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0940788 s, 22.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.307318 s, 13.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.337217 s, 6.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 15:08:23 (1783969703) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 15:08:27 (1783969707) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 15:08:31 (1783969711) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353946 s, 29.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 15:08:40 (1783969720) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 15:08:44 (1783969724) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:08:53 (1783969733) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.390509 s, 26.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg322-client.virtnet /mnt/lustre (opts:) Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:09:05 (1783969745) llite.lustre-ffff985042f0e800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.376915 s, 27.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg322-client.virtnet /mnt/lustre (opts:) Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff985047eaa000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 15:09:21 (1783969761) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (43s) == sanity-flr test 46: Verify setstripe --copy option ==== 15:10:03 (1783969803) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (36s) == sanity-flr test 47: Verify mirror obj alloc =========== 15:10:39 (1783969839) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 15:10:42 (1783969842) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.231103 s, 13.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.173426 s, 18.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xe8bd1ad2 (nosync,stale) Mirror 2: 0xe8bd1ad2 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 004bd4589c11878412ce7b49a2967e87 - mirror 1 checksum: 858767c2b8d019e9ddfdf3436d308940 - mirror 2 checksum: 004bd4589c11878412ce7b49a2967e87 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-13T15:10:46-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-13T15:10:46-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 004bd4589c11878412ce7b49a2967e87 - mirror 1 checksum: 858767c2b8d019e9ddfdf3436d308940 - mirror 2 checksum: 858767c2b8d019e9ddfdf3436d308940 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-13T15:10:46-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-13T15:10:46-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 15:10:55 (1783969855) Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 15:11:03 (1783969863) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 15:11:25 (1783969885) targets are mounted 15:11:25 (1783969885) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286351 s, 36.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } PASS 50A (49s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 15:11:44 (1783969904) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 15:11:48 (1783969908) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 15:11:51 (1783969911) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0179504 s, 913 kB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 15:12:01 (1783969921) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 15:12:04 (1783969924) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 15:12:08 (1783969928) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (28s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 15:12:36 (1783969956) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 15:13:06 (1783969986) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 15:13:31 (1783970011) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 15:13:57 (1783970037) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 15:14:04 (1783970044) mirror create pid 53961 mirror split pid 53962 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 53961 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 53962 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 15:15:12 (1783970112) Seed set to 114 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read skipping zero size read truncating to largest ever: 0x2eec4 000013 1783970114.479381 trunc from 00000000 to 0x02eec3 (0x2eec4 bytes) 000014 1783970114.502419 mapwrite 0x013674 thru 0x016785 (0x3112 bytes) 000016 1783970114.636821 read 0x000f34 thru 0x004d24 (0x3df1 bytes) 000018 1783970114.664370 mapread 0x00a1ba thru 0x012fc5 (0x8e0c bytes) 000019 1783970114.692005 read 0x028ea9 thru 0x02eec3 (0x601b bytes) 000022 1783970114.708256 mapread 0x018a66 thru 0x01cae4 (0x407f bytes) 000027 1783970114.724216 mapwrite 0x00f77d thru 0x0197b6 (0xa03a bytes) 000028 1783970114.822282 mapread 0x00b98e thru 0x00c95c (0xfcf bytes) 000030 1783970114.823787 read 0x025cd7 thru 0x02eec3 (0x91ed bytes) 000031 1783970114.825545 mapwrite 0x01d3b1 thru 0x02075b (0x33ab bytes) 000032 1783970114.899827 mapread 0x01a5c0 thru 0x01c328 (0x1d69 bytes) 000037 1783970114.906429 write 0x014f84 thru 0x01a083 (0x5100 bytes) 000038 1783970114.911281 mapwrite 0x03d3fb thru 0x03d463 (0x69 bytes) 000040 1783970115.080223 MIRROR_EXTEND to 2 mirrors 000041 1783970115.946400 read 0x02f0ca thru 0x03729a (0x81d1 bytes) 000042 1783970115.961206 mapwrite 0x0101a9 thru 0x01a39e (0xa1f6 bytes) 000043 1783970116.228286 mapread 0x007631 thru 0x009550 (0x1f20 bytes) 000044 1783970116.246503 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x178:0x0]: Invalid argument (22) 000045 1783970116.443207 trunc from 0x03d464 to 0x00f5fc (0x2de67 bytes) 000046 1783970116.462207 read 0x000dd5 thru 0x00b6af (0xa8db bytes) truncating to largest ever: 0x33d81 000047 1783970116.470354 trunc from 0x00f5fd to 0x033d80 (0x24784 bytes) 000048 1783970116.486054 trunc from 0x033d81 to 0x0080a6 (0x2bcda bytes) 000052 1783970116.503605 mapread 0x002d77 thru 0x0080a6 (0x5330 bytes) 000053 1783970116.504789 read 0x00200a thru 0x0080a6 (0x609d bytes) 000055 1783970116.506423 mapread 0x005ec6 thru 0x0080a6 (0x21e1 bytes) 000056 1783970116.507485 read 0x006ab8 thru 0x0080a6 (0x15ef bytes) 000057 1783970116.508510 write 0x0116f3 thru 0x0157c1 (0x40cf bytes) 000058 1783970116.510158 mapwrite 0x000b88 thru 0x00fe42 (0xf2bb bytes) 000059 1783970116.568563 mapwrite 0x03526e thru 0x037e51 (0x2be4 bytes) 000061 1783970116.640493 trunc from 0x037e52 to 0x029ede (0xdf73 bytes) 000062 1783970116.663495 mapwrite 0x02c5bc thru 0x039fcd (0xda12 bytes) 000064 1783970116.809648 mapwrite 0x022079 thru 0x022f22 (0xeaa bytes) 000068 1783970116.881990 mapread 0x01e8a4 thru 0x025486 (0x6be3 bytes) 000069 1783970116.886059 trunc from 0x039fce to 0x0026ec (0x378e1 bytes) 000070 1783970116.924258 trunc from 0x0026ed to 0x01eb18 (0x1c42c bytes) 000072 1783970116.956512 trunc from 0x01eb19 to 0x02aee2 (0xc3ca bytes) 000076 1783970117.019565 mapread 0x00894d thru 0x00c890 (0x3f44 bytes) 000080 1783970117.063906 mapwrite 0x036d2e thru 0x03ffff (0x92d2 bytes) 000081 1783970117.189963 trunc from 0x040000 to 0x024d56 (0x1b2a9 bytes) 000083 1783970117.227906 write 0x030214 thru 0x031d56 (0x1b43 bytes) 000092 1783970117.248165 mapwrite 0x023712 thru 0x02c6cf (0x8fbe bytes) 000094 1783970117.384759 mapread 0x02699e thru 0x02a85f (0x3ec2 bytes) 000095 1783970117.387440 mapwrite 0x03d657 thru 0x03ffff (0x29a9 bytes) 000097 1783970117.510142 trunc from 0x040000 to 0x023651 (0x1c9ae bytes) 000099 1783970117.531672 write 0x02ab56 thru 0x034ecc (0xa377 bytes) 000101 1783970117.535556 mapwrite 0x0365d8 thru 0x03b750 (0x5179 bytes) 000103 1783970117.673866 MIRROR_EXTEND to 2 mirrors 000107 1783970118.574284 read 0x0245f5 thru 0x0323ce (0xddda bytes) 000108 1783970118.592157 mapread 0x010a78 thru 0x015303 (0x488c bytes) 000111 1783970118.634220 read 0x011b8d thru 0x0187af (0x6c23 bytes) 000112 1783970118.639672 MIRROR_EXTEND to 3 mirrors 000113 1783970119.333975 mapwrite 0x01f82f thru 0x02825b (0x8a2d bytes) 000116 1783970119.618006 write 0x01c91d thru 0x01edbf (0x24a3 bytes) 000119 1783970119.627517 trunc from 0x03b751 to 0x02431a (0x17436 bytes) 000120 1783970119.656126 write 0x00888e thru 0x012ff4 (0xa767 bytes) 000122 1783970119.675646 write 0x004ad7 thru 0x01277e (0xdca8 bytes) 000123 1783970119.687275 MIRROR_RESYNC 3 mirrors 000125 1783970120.562575 mapwrite 0x0175e6 thru 0x0259ba (0xe3d5 bytes) 000130 1783970120.941465 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x259bb): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x259bb): Mirror 2: 0xd4e87fc (stale) Mirror 3: 0xd4e87fc (stale) CRC-32 checksum value for chunk [0, 0x259bb): Mirror 1: 0x512e5a37 Mirror 4: 0x512e5a37 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x259bb: skipped 000131 1783970122.064567 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x259bb): Mirror 2: 0xd4e87fc (stale) Mirror 3: 0xd4e87fc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x259bb): Mirror 2: 0xd4e87fc (stale) Mirror 3: 0xd4e87fc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x259bb: skipped 000133 1783970122.464704 mapread 0x023e6e thru 0x0259ba (0x1b4d bytes) 000134 1783970122.488775 mapwrite 0x027ecf thru 0x033200 (0xb332 bytes) 000135 1783970122.637665 mapread 0x00e679 thru 0x00f50c (0xe94 bytes) 000136 1783970122.660973 read 0x032039 thru 0x033200 (0x11c8 bytes) 000137 1783970122.661714 read 0x008f74 thru 0x00a3b5 (0x1442 bytes) 000138 1783970122.670075 MIRROR_RESYNC 3 mirrors 000139 1783970123.634630 mapread 0x00a65a thru 0x010326 (0x5ccd bytes) 000141 1783970123.672740 mapwrite 0x001c04 thru 0x001eaa (0x2a7 bytes) 000142 1783970123.819405 mapwrite 0x03753f thru 0x03ffff (0x8ac1 bytes) 000143 1783970123.944247 mapread 0x03a46d thru 0x03ae8c (0xa20 bytes) 000145 1783970123.954214 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x39faea0a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x39faea0a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000146 1783970124.377781 trunc from 0x040000 to 0x005bcb (0x3a434 bytes) 000149 1783970124.432980 mapwrite 0x029319 thru 0x0369f1 (0xd6d9 bytes) 000150 1783970124.642400 write 0x010f2d thru 0x012d5f (0x1e33 bytes) 000156 1783970124.672779 trunc from 0x0369f2 to 0x02efb9 (0x7a38 bytes) 000160 1783970124.693455 mapwrite 0x031bc1 thru 0x03b2d9 (0x9719 bytes) 000161 1783970124.809150 read 0x02cbe3 thru 0x02ee10 (0x222e bytes) 000163 1783970124.811663 write 0x00c476 thru 0x01bb6b (0xf6f6 bytes) 000165 1783970124.826738 trunc from 0x03b2da to 0x014c5e (0x2667b bytes) 000167 1783970124.848395 trunc from 0x014c5f to 0x01f211 (0xa5b3 bytes) 000168 1783970124.870522 MIRROR_RESYNC 2 mirrors 000169 1783970125.416697 mapread 0x00c230 thru 0x011124 (0x4ef5 bytes) 000171 1783970125.449085 trunc from 0x01f212 to 0x00ff5e (0xf2b3 bytes) 000173 1783970125.585439 read 0x00eb67 thru 0x00ff5e (0x13f8 bytes) 000175 1783970125.594501 mapread 0x009980 thru 0x00ccf1 (0x3372 bytes) 000176 1783970125.610653 mapread 0x00dddb thru 0x00ff5e (0x2184 bytes) 000178 1783970125.616826 MIRROR_RESYNC 2 mirrors 000179 1783970126.256977 MIRROR_EXTEND to 3 mirrors 000180 1783970126.826101 mapwrite 0x02a29b thru 0x0388b1 (0xe617 bytes) 000184 1783970127.053624 mapwrite 0x02f135 thru 0x037746 (0x8612 bytes) 000187 1783970127.119958 mapread 0x027bfc thru 0x02ba43 (0x3e48 bytes) 000191 1783970127.124108 mapread 0x0045b3 thru 0x0112e6 (0xcd34 bytes) 000192 1783970127.143962 trunc from 0x0388b2 to 0x00f79c (0x29115 bytes) 000194 1783970127.168051 read 0x0095bb thru 0x00f79c (0x61e2 bytes) 000195 1783970127.168849 mapread 0x002e85 thru 0x006836 (0x39b2 bytes) 000196 1783970127.174573 MIRROR_RESYNC 3 mirrors 000199 1783970127.757467 mapread 0x00c59e thru 0x00d693 (0x10f6 bytes) 000200 1783970127.766715 mapwrite 0x01473c thru 0x01ddf3 (0x96b8 bytes) 000201 1783970128.086541 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1ddf4): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ddf4): Mirror 3: 0x8f60eb17 (stale) Mirror 4: 0x8f60eb17 (stale) CRC-32 checksum value for chunk [0, 0x1ddf4): Mirror 1: 0x530875ee Mirror 5: 0x530875ee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ddf4: skipped 000203 1783970129.065713 write 0x016251 thru 0x01f4ce (0x927e bytes) 000205 1783970129.188214 trunc from 0x01f4cf to 0x025e44 (0x6976 bytes) 000206 1783970129.227798 mapread 0x025454 thru 0x025e44 (0x9f1 bytes) 000207 1783970129.243113 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x25e45): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25e45): Mirror 3: 0x3f8a9329 (stale) Mirror 4: 0x3f8a9329 (stale) Mirror 5: 0xf6674022 (stale) CRC-32 checksum value for chunk [0, 0x25e45): Mirror 1: 0xfc85b322 Mirror 6: 0xfc85b322 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25e45: skipped 000209 1783970130.158612 read 0x01ce7d thru 0x021d55 (0x4ed9 bytes) 000210 1783970130.173380 write 0x016037 thru 0x01a87c (0x4846 bytes) 000212 1783970130.273129 mapwrite 0x01ac12 thru 0x01f8ae (0x4c9d bytes) 000213 1783970130.337564 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25e45): Mirror 3: 0x3f8a9329 (stale) Mirror 4: 0x3f8a9329 (stale) Mirror 6: 0xfc85b322 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25e45): Mirror 3: 0x3f8a9329 (stale) Mirror 4: 0x3f8a9329 (stale) Mirror 6: 0xfc85b322 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25e45: skipped 000215 1783970130.725697 trunc from 0x025e45 to 0x02d0d0 (0x728c bytes) 000217 1783970130.786717 read 0x02be16 thru 0x02d0d0 (0x12bb bytes) 000218 1783970130.804328 MIRROR_RESYNC 4 mirrors 000219 1783970131.917818 read 0x01e3d1 thru 0x027680 (0x92b0 bytes) 000221 1783970131.936419 write 0x02c4f4 thru 0x031a41 (0x554e bytes) 000222 1783970132.043962 write 0x034535 thru 0x03f780 (0xb24c bytes) 000224 1783970132.053205 MIRROR_RESYNC 4 mirrors 000226 1783970132.843613 trunc from 0x03f781 to 0x008fec (0x36794 bytes) 000227 1783970132.927343 trunc from 0x008fed to 0x029204 (0x20218 bytes) 000228 1783970132.959523 read 0x005306 thru 0x014fb0 (0xfcab bytes) 000230 1783970132.971803 trunc from 0x029205 to 0x00120b (0x27ff9 bytes) 000232 1783970133.000587 mapread 0x000387 thru 0x00120b (0xe85 bytes) 000235 1783970133.017539 read 0x000712 thru 0x00120b (0xafa bytes) 000239 1783970133.022057 mapread 0x00063a thru 0x00120b (0xbd2 bytes) 000240 1783970133.023469 read 0x0007e0 thru 0x00120b (0xa2c bytes) 000241 1783970133.025056 mapwrite 0x02e25b thru 0x038a69 (0xa80f bytes) 000242 1783970133.147291 trunc from 0x038a6a to 0x00d40e (0x2b65b bytes) 000247 1783970133.179362 write 0x009b54 thru 0x0156da (0xbb87 bytes) 000249 1783970133.185752 write 0x01b963 thru 0x027c4a (0xc2e8 bytes) 000250 1783970133.189804 mapread 0x01ed26 thru 0x027c4a (0x8f25 bytes) 000251 1783970133.207236 MIRROR_RESYNC 4 mirrors 000252 1783970134.228231 mapread 0x00683c thru 0x010c74 (0xa439 bytes) 000253 1783970134.248069 write 0x03cf02 thru 0x03ffff (0x30fe bytes) 000255 1783970134.310190 mapread 0x011376 thru 0x012c63 (0x18ee bytes) 000256 1783970134.319781 mapread 0x023047 thru 0x0254e7 (0x24a1 bytes) 000257 1783970134.328956 trunc from 0x040000 to 0x02fac6 (0x10539 bytes) truncating to largest ever: 0x3a763 000259 1783970134.364731 trunc from 0x02fac7 to 0x03a762 (0xac9c bytes) 000260 1783970134.385447 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a763): Mirror 4: 0xa4006a0 (stale) Mirror 6: 0xa4006a0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a763): Mirror 4: 0xa4006a0 (stale) Mirror 6: 0xa4006a0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a763: skipped 000261 1783970134.794557 write 0x001c65 thru 0x001e82 (0x21e bytes) 000263 1783970134.801819 write 0x0393aa thru 0x03a6ae (0x1305 bytes) 000266 1783970134.818466 trunc from 0x03a763 to 0x014b3e (0x25c24 bytes) 000267 1783970134.836077 write 0x03e352 thru 0x03ffff (0x1cae bytes) 000268 1783970134.838336 write 0x03debc thru 0x03ffff (0x2144 bytes) 000269 1783970134.843980 write 0x024b2b thru 0x034b23 (0xfff9 bytes) 000270 1783970134.865932 mapwrite 0x0328e1 thru 0x03df03 (0xb623 bytes) 000272 1783970135.016660 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x57ce671 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x57ce671 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000273 1783970135.381819 trunc from 0x040000 to 0x03323e (0xcdc1 bytes) 000274 1783970135.415242 read 0x011886 thru 0x01f86d (0xdfe8 bytes) 000277 1783970135.443751 mapwrite 0x01cfd5 thru 0x0289ae (0xb9da bytes) 000278 1783970135.578814 mapread 0x03247b thru 0x03323e (0xdc4 bytes) 000279 1783970135.609941 write 0x0086f2 thru 0x011aae (0x93bd bytes) 000283 1783970135.638654 write 0x019fd3 thru 0x01e4be (0x44ec bytes) 000284 1783970135.639852 mapread 0x011d15 thru 0x014455 (0x2741 bytes) 000285 1783970135.640929 mapread 0x00705d thru 0x00a0ef (0x3093 bytes) 000286 1783970135.668562 write 0x032c4c thru 0x034452 (0x1807 bytes) 000287 1783970135.680328 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x178:0x0]: Invalid argument (22) 000288 1783970135.849602 mapwrite 0x030dfa thru 0x035aaa (0x4cb1 bytes) 000289 1783970135.952205 read 0x034206 thru 0x035aaa (0x18a5 bytes) 000291 1783970135.953731 write 0x01d84d thru 0x027259 (0x9a0d bytes) 000294 1783970135.968005 mapread 0x002208 thru 0x008b95 (0x698e bytes) 000296 1783970135.978719 mapwrite 0x021c34 thru 0x02ce86 (0xb253 bytes) 000297 1783970136.060498 trunc from 0x035aab to 0x0094b4 (0x2c5f6 bytes) 000300 1783970136.083643 mapwrite 0x00341d thru 0x00bf7e (0x8b62 bytes) 000302 1783970136.190521 mapwrite 0x027929 thru 0x033f9a (0xc672 bytes) 000304 1783970136.339444 read 0x0294fa thru 0x030379 (0x6e80 bytes) 000305 1783970136.343270 write 0x0334e9 thru 0x03ffff (0xcb17 bytes) 000306 1783970136.350939 mapread 0x03c220 thru 0x03ffff (0x3de0 bytes) 000307 1783970136.352943 read 0x0191c6 thru 0x026d10 (0xdb4b bytes) 000311 1783970136.387878 trunc from 0x040000 to 0x011eb3 (0x2e14c bytes) 000312 1783970136.413550 trunc from 0x011eb4 to 0x009808 (0x86ab bytes) 000313 1783970136.435992 mapread 0x005a1a thru 0x009808 (0x3def bytes) 000315 1783970136.442811 mapread 0x002d0b thru 0x009808 (0x6afe bytes) 000319 1783970136.448181 read 0x008b30 thru 0x009808 (0xcd9 bytes) 000320 1783970136.448888 mapread 0x00073d thru 0x0094e4 (0x8da8 bytes) 000323 1783970136.456904 mapwrite 0x02f843 thru 0x039189 (0x9947 bytes) 000324 1783970136.606890 trunc from 0x03918a to 0x0334a2 (0x5ce7 bytes) 000326 1783970136.628991 write 0x03de95 thru 0x03ffff (0x216b bytes) 000328 1783970136.633952 trunc from 0x040000 to 0x004c8c (0x3b373 bytes) 000332 1783970136.658324 mapread 0x003497 thru 0x004c8c (0x17f6 bytes) 000333 1783970136.664954 mapwrite 0x032e3e thru 0x03d313 (0xa4d6 bytes) 000334 1783970136.833396 mapread 0x037109 thru 0x03d313 (0x620b bytes) 000338 1783970136.841353 mapwrite 0x000c30 thru 0x0045d2 (0x39a3 bytes) 000339 1783970136.908975 read 0x02e286 thru 0x0353ea (0x7165 bytes) 000342 1783970136.926319 trunc from 0x03d314 to 0x033770 (0x9ba3 bytes) 000343 1783970136.952501 mapwrite 0x016262 thru 0x02615e (0xfefd bytes) 000344 1783970137.001631 mapread 0x0268a3 thru 0x033770 (0xcece bytes) 000348 1783970137.007385 write 0x00acb0 thru 0x013b29 (0x8e7a bytes) 000349 1783970137.013480 read 0x01a7d0 thru 0x022f61 (0x8792 bytes) 000350 1783970137.014651 read 0x007a52 thru 0x0102cc (0x887b bytes) 000360 1783970137.025594 write 0x02deef thru 0x03584c (0x795e bytes) 000361 1783970137.028106 mapread 0x0301d3 thru 0x03584c (0x567a bytes) 000362 1783970137.033564 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x3ea17 000363 1783970137.797545 trunc from 0x03584d to 0x03ea16 (0x91ca bytes) 000364 1783970137.952462 mapread 0x005965 thru 0x009a1b (0x40b7 bytes) 000366 1783970137.977513 trunc from 0x03ea17 to 0x007717 (0x372ff bytes) 000367 1783970138.020397 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x7718): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7718): Mirror 2: 0x6e965400 (stale) CRC-32 checksum value for chunk [0, 0x7718): Mirror 1: 0x6e965400 Mirror 3: 0x6e965400 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7718: skipped 000369 1783970139.054613 trunc from 0x007718 to 0x01d390 (0x15c79 bytes) 000370 1783970139.216520 trunc from 0x01d391 to 0x033698 (0x16308 bytes) 000372 1783970139.268731 mapread 0x025b87 thru 0x02a3d4 (0x484e bytes) 000375 1783970139.309228 mapread 0x0052ff thru 0x00bec1 (0x6bc3 bytes) 000376 1783970139.336260 mapread 0x016166 thru 0x01d418 (0x72b3 bytes) 000378 1783970139.374602 write 0x021947 thru 0x0275a0 (0x5c5a bytes) 000379 1783970139.379051 mapread 0x0163a8 thru 0x019f02 (0x3b5b bytes) 000380 1783970139.382629 read 0x017324 thru 0x01aefe (0x3bdb bytes) 000381 1783970139.390076 trunc from 0x033699 to 0x03a9f2 (0x735a bytes) 000382 1783970139.428749 mapwrite 0x01c6bf thru 0x02916b (0xcaad bytes) 000383 1783970139.523437 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a9f3): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a9f3): Mirror 2: 0x3c6fd39c (stale) Mirror 3: 0xc414f93e (stale) CRC-32 checksum value for chunk [0, 0x3a9f3): Mirror 1: 0xc814199c Mirror 4: 0xc814199c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a9f3: skipped 000385 1783970140.593487 read 0x01d5c2 thru 0x022d1a (0x5759 bytes) 000386 1783970140.609035 trunc from 0x03a9f3 to 0x01b0fb (0x1f8f7 bytes) 000387 1783970140.746280 write 0x018989 thru 0x01f4c1 (0x6b39 bytes) 000388 1783970140.755622 read 0x016436 thru 0x01f4c1 (0x908c bytes) 000389 1783970140.772466 mapwrite 0x03e1a0 thru 0x03f845 (0x16a6 bytes) 000390 1783970140.853048 trunc from 0x03f846 to 0x034dc7 (0xaa7e bytes) 000391 1783970140.876152 MIRROR_RESYNC 4 mirrors 000392 1783970142.571606 trunc from 0x034dc8 to 0x0285c7 (0xc800 bytes) 000393 1783970142.705144 write 0x0125b0 thru 0x019503 (0x6f54 bytes) 000395 1783970142.756643 mapwrite 0x00565c thru 0x01202a (0xc9cf bytes) 000397 1783970142.936428 mapread 0x0283af thru 0x0285c7 (0x219 bytes) 000399 1783970142.959725 mapwrite 0x01a289 thru 0x01e01b (0x3d93 bytes) 000402 1783970143.103743 mapwrite 0x0129be thru 0x01f439 (0xca7c bytes) 000404 1783970143.158571 read 0x00bb69 thru 0x011d13 (0x61ab bytes) 000406 1783970143.160619 trunc from 0x0285c8 to 0x018c2c (0xf99b bytes) 000408 1783970143.184459 read 0x010bfa thru 0x012395 (0x179c bytes) 000409 1783970143.185778 read 0x017a85 thru 0x018c2c (0x11a8 bytes) 000410 1783970143.186575 mapread 0x018165 thru 0x018c2c (0xac8 bytes) 000412 1783970143.189138 mapread 0x005747 thru 0x012a43 (0xd2fd bytes) 000413 1783970143.190683 trunc from 0x018c2d to 0x01ebc7 (0x5f9b bytes) 000414 1783970143.212935 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ebc8): Mirror 2: 0xb8242bff (stale) Mirror 4: 0xb8242bff (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ebc8): Mirror 2: 0xb8242bff (stale) Mirror 4: 0xb8242bff (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ebc8: skipped 000415 1783970143.531791 read 0x01631b thru 0x0175e4 (0x12ca bytes) 000417 1783970143.541361 trunc from 0x01ebc8 to 0x03a4b4 (0x1b8ed bytes) 000420 1783970143.564401 mapread 0x01e295 thru 0x02b326 (0xd092 bytes) 000421 1783970143.596386 mapread 0x007f69 thru 0x00a671 (0x2709 bytes) 000422 1783970143.606524 trunc from 0x03a4b5 to 0x03591e (0x4b96 bytes) 000423 1783970143.623373 trunc from 0x03591f to 0x01502b (0x208f3 bytes) 000425 1783970143.657490 mapwrite 0x026901 thru 0x030767 (0x9e67 bytes) 000427 1783970143.805292 trunc from 0x030768 to 0x0265bd (0xa1aa bytes) 000428 1783970143.825376 write 0x00d825 thru 0x01d370 (0xfb4c bytes) 000429 1783970143.834390 write 0x033d69 thru 0x03cfdc (0x9274 bytes) 000430 1783970143.844832 mapwrite 0x03d258 thru 0x03ffff (0x2da8 bytes) 000431 1783970144.002240 mapwrite 0x0351b9 thru 0x03f492 (0xa2da bytes) 000433 1783970144.075327 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xaae343c6 (stale) Mirror 4: 0xaae343c6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4d1dcf08 Mirror 5: 0x4d1dcf08 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000434 1783970144.950351 write 0x0130ee thru 0x022ebc (0xfdcf bytes) 000437 1783970145.153403 mapwrite 0x029a78 thru 0x02ed71 (0x52fa bytes) 000438 1783970145.296753 mapwrite 0x01f9f0 thru 0x027bd9 (0x81ea bytes) 000439 1783970145.411945 trunc from 0x040000 to 0x016ad8 (0x29527 bytes) 000440 1783970145.486250 read 0x00fd9d thru 0x0126c7 (0x292b bytes) 000441 1783970145.495302 write 0x03d987 thru 0x03ffff (0x2679 bytes) 000442 1783970145.496513 mapread 0x023008 thru 0x02893f (0x5938 bytes) 000444 1783970145.518312 read 0x027e9e thru 0x033400 (0xb563 bytes) 000446 1783970145.563279 MIRROR_RESYNC 4 mirrors 000447 1783970146.608658 read 0x015140 thru 0x01d5e2 (0x84a3 bytes) 000448 1783970146.632303 mapwrite 0x004079 thru 0x0076f8 (0x3680 bytes) 000451 1783970146.834109 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3e6836c0 (stale) Mirror 4: 0x3e6836c0 (stale) Mirror 5: 0x3e6836c0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc4afc45b Mirror 6: 0xc4afc45b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000453 1783970147.769686 read 0x027308 thru 0x0336f1 (0xc3ea bytes) 000455 1783970147.800651 mapread 0x0017e6 thru 0x003f1e (0x2739 bytes) 000456 1783970147.836503 read 0x02f11e thru 0x03d24c (0xe12f bytes) 000457 1783970147.861727 mapread 0x0293c6 thru 0x035b4f (0xc78a bytes) 000458 1783970147.885687 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3e6836c0 (stale) Mirror 4: 0x3e6836c0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3e6836c0 (stale) Mirror 4: 0x3e6836c0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc4afc45b Mirror 6: 0xc4afc45b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000459 1783970148.520635 mapread 0x005216 thru 0x011295 (0xc080 bytes) 000460 1783970148.599921 trunc from 0x040000 to 0x00a7a0 (0x3585f bytes) 000461 1783970148.714720 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xa7a1): Mirror 4: 0xc778fe95 (stale) Mirror 6: 0xca24b4b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa7a1): Mirror 4: 0xc778fe95 (stale) Mirror 6: 0xca24b4b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa7a1: skipped 000462 1783970149.043457 read 0x00a281 thru 0x00a7a0 (0x520 bytes) 000463 1783970149.049337 mapwrite 0x01fbb9 thru 0x02a324 (0xa76c bytes) 000464 1783970149.156279 read 0x00c70b thru 0x01660f (0x9f05 bytes) 000468 1783970149.177103 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a325): Mirror 4: 0x56bf976b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a325): Mirror 4: 0x56bf976b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a325: skipped 000469 1783970149.577598 mapwrite 0x02c2f0 thru 0x0338e9 (0x75fa bytes) 000470 1783970149.696148 mapwrite 0x0269f5 thru 0x031c35 (0xb241 bytes) 000472 1783970149.799836 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x178:0x0]: Invalid argument (22) 000473 1783970149.999158 mapwrite 0x031081 thru 0x0354f4 (0x4474 bytes) 000476 1783970150.129156 mapread 0x03225b thru 0x0354f4 (0x329a bytes) 000477 1783970150.131848 mapwrite 0x01636c thru 0x01851f (0x21b4 bytes) 000479 1783970150.192469 mapread 0x02b903 thru 0x0354f4 (0x9bf2 bytes) 000484 1783970150.219066 read 0x011ae9 thru 0x0179f1 (0x5f09 bytes) 000486 1783970150.220921 trunc from 0x0354f5 to 0x012352 (0x231a2 bytes) 000487 1783970150.246932 mapwrite 0x0170cc thru 0x0269d2 (0xf907 bytes) 000488 1783970150.368508 write 0x020591 thru 0x0291e9 (0x8c59 bytes) 000490 1783970150.373808 mapwrite 0x00c85e thru 0x010a91 (0x4234 bytes) 000492 1783970150.443797 mapread 0x017330 thru 0x01ab09 (0x37da bytes) 000493 1783970150.444825 mapread 0x009d10 thru 0x01972c (0xfa1d bytes) 000494 1783970150.446064 mapread 0x00ae18 thru 0x0126fc (0x78e5 bytes) 000498 1783970150.452705 trunc from 0x0291ea to 0x018031 (0x111b8 bytes) 000499 1783970150.468735 read 0x009fb3 thru 0x016fbe (0xd00c bytes) 000501 1783970150.472050 write 0x02bf42 thru 0x02d188 (0x1247 bytes) 000502 1783970150.474539 mapwrite 0x02f529 thru 0x03bbb1 (0xc689 bytes) 000503 1783970150.575955 trunc from 0x03bbb2 to 0x0373a5 (0x480c bytes) 000507 1783970150.604092 read 0x00173f thru 0x007207 (0x5ac9 bytes) 000508 1783970150.605186 mapwrite 0x039369 thru 0x03ffff (0x6c97 bytes) 000509 1783970150.690324 mapwrite 0x00e390 thru 0x016f6b (0x8bdc bytes) 000510 1783970150.734900 trunc from 0x040000 to 0x027a14 (0x185eb bytes) 000511 1783970150.767335 MIRROR_EXTEND to 2 mirrors 000514 1783970151.490991 MIRROR_RESYNC 2 mirrors 000516 1783970151.842907 write 0x0048be thru 0x013802 (0xef45 bytes) 000517 1783970151.999987 trunc from 0x027a15 to 0x031a18 (0xa004 bytes) 000520 1783970152.050897 MIRROR_RESYNC 2 mirrors 000522 1783970152.807494 MIRROR_EXTEND to 3 mirrors 000524 1783970153.577124 write 0x02c313 thru 0x02c8d8 (0x5c6 bytes) 000525 1783970153.682397 write 0x015e74 thru 0x025e5a (0xffe7 bytes) 000529 1783970153.719831 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31a19): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31a19): Mirror 2: 0x2c076b91 (stale) Mirror 3: 0x2c076b91 (stale) CRC-32 checksum value for chunk [0, 0x31a19): Mirror 1: 0xf756ea28 Mirror 4: 0xf756ea28 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31a19: skipped 000530 1783970154.617494 read 0x00c66e thru 0x01bc12 (0xf5a5 bytes) 000532 1783970154.649515 trunc from 0x031a19 to 0x01da53 (0x13fc5 bytes) 000535 1783970154.784637 write 0x0397d8 thru 0x03ffff (0x6828 bytes) 000536 1783970154.788512 read 0x03f780 thru 0x03ffff (0x880 bytes) 000538 1783970154.796450 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x19a9bdfc (stale) Mirror 4: 0x88cf5d1e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x19a9bdfc (stale) Mirror 4: 0x88cf5d1e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000539 1783970155.221626 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x88cf5d1e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x88cf5d1e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000540 1783970155.653119 mapwrite 0x01e1a4 thru 0x02c56b (0xe3c8 bytes) 000541 1783970155.833175 mapread 0x01cf09 thru 0x0275fd (0xa6f5 bytes) 000543 1783970155.834892 write 0x003b8a thru 0x0074a1 (0x3918 bytes) 000545 1783970155.862375 trunc from 0x040000 to 0x02c121 (0x13ede bytes) 000546 1783970155.889994 trunc from 0x02c122 to 0x017036 (0x150eb bytes) 000547 1783970155.945392 trunc from 0x017037 to 0x033364 (0x1c32e bytes) 000548 1783970155.959458 read 0x0117fd thru 0x018c59 (0x745d bytes) 000550 1783970155.973384 write 0x0294c3 thru 0x02ba15 (0x2553 bytes) 000551 1783970155.989528 MIRROR_RESYNC 2 mirrors 000552 1783970156.756405 mapwrite 0x025cc0 thru 0x028a1e (0x2d5f bytes) 000555 1783970157.049708 write 0x02f152 thru 0x037c50 (0x8aff bytes) 000556 1783970157.055265 write 0x00dd04 thru 0x013600 (0x58fd bytes) 000558 1783970157.100486 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x178:0x0]: Invalid argument (22) 000559 1783970157.332690 mapwrite 0x002735 thru 0x00edd2 (0xc69e bytes) 000561 1783970157.437144 read 0x02b3b9 thru 0x0300d9 (0x4d21 bytes) 000563 1783970157.454159 write 0x02e534 thru 0x03df04 (0xf9d1 bytes) 000565 1783970157.457939 mapread 0x03d979 thru 0x03df04 (0x58c bytes) 000566 1783970157.458853 mapwrite 0x024732 thru 0x02d094 (0x8963 bytes) 000567 1783970157.546521 mapread 0x00c5a6 thru 0x015890 (0x92eb bytes) 000569 1783970157.562354 trunc from 0x03df05 to 0x00c1f2 (0x31d12 bytes) 000570 1783970157.628875 trunc from 0x00c1f3 to 0x019dc3 (0xdbd1 bytes) 000572 1783970157.700740 mapread 0x0145f6 thru 0x019dc3 (0x57ce bytes) 000575 1783970157.719512 mapwrite 0x03d753 thru 0x03ffff (0x28ad bytes) 000578 1783970157.787003 read 0x00b9c8 thru 0x00fc2b (0x4264 bytes) 000580 1783970157.792438 trunc from 0x040000 to 0x019552 (0x26aad bytes) 000582 1783970157.816756 mapread 0x00e063 thru 0x019552 (0xb4f0 bytes) 000583 1783970157.825873 trunc from 0x019553 to 0x00f22f (0xa323 bytes) 000584 1783970157.864636 read 0x00d6d2 thru 0x00f22f (0x1b5e bytes) 000586 1783970157.866255 mapread 0x002acc thru 0x00f22f (0xc764 bytes) 000588 1783970157.876339 read 0x00e2b3 thru 0x00f22f (0xf7d bytes) 000590 1783970157.878374 trunc from 0x00f230 to 0x002ee3 (0xc34c bytes) 000591 1783970157.921247 read 0x0020fa thru 0x002ee3 (0xdea bytes) 000594 1783970157.927876 mapread 0x0021ed thru 0x002ee3 (0xcf7 bytes) 000598 1783970157.936920 read 0x001205 thru 0x002ee3 (0x1cdf bytes) 000601 1783970157.942509 trunc from 0x002ee4 to 0x013302 (0x1041f bytes) 000602 1783970158.008900 write 0x008600 thru 0x0133b4 (0xadb5 bytes) 000604 1783970158.028766 write 0x01684a thru 0x02346d (0xcc24 bytes) 000606 1783970158.032552 mapwrite 0x036e28 thru 0x03ffff (0x91d8 bytes) 000610 1783970158.224274 write 0x034c49 thru 0x038296 (0x364e bytes) 000611 1783970158.226133 write 0x034994 thru 0x03d99d (0x900a bytes) 000612 1783970158.228817 write 0x01aca9 thru 0x01db50 (0x2ea8 bytes) 000613 1783970158.230265 mapread 0x009ea1 thru 0x0164c0 (0xc620 bytes) 000617 1783970158.236835 trunc from 0x040000 to 0x006269 (0x39d96 bytes) 000618 1783970158.279999 mapread 0x00031e thru 0x006269 (0x5f4c bytes) 000620 1783970158.288476 trunc from 0x00626a to 0x0316e7 (0x2b47e bytes) 000624 1783970158.325631 write 0x00b33d thru 0x012ef8 (0x7bbc bytes) 000625 1783970158.337331 trunc from 0x0316e8 to 0x03e8ed (0xd206 bytes) 000627 1783970158.365458 mapwrite 0x024967 thru 0x02dcce (0x9368 bytes) 000630 1783970158.574358 MIRROR_EXTEND to 2 mirrors 000631 1783970159.482748 mapread 0x01c1e9 thru 0x02a8b4 (0xe6cc bytes) 000632 1783970159.544969 write 0x02d9c0 thru 0x033569 (0x5baa bytes) 000634 1783970159.743203 read 0x024503 thru 0x024d80 (0x87e bytes) 000635 1783970159.767883 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x178:0x0]: Invalid argument (22) 000636 1783970160.159809 trunc from 0x03e8ee to 0x025e87 (0x18a66 bytes) 000638 1783970160.196804 mapwrite 0x0366ff thru 0x03ffff (0x9901 bytes) 000641 1783970160.406969 mapread 0x02ad71 thru 0x02bd77 (0x1007 bytes) 000642 1783970160.476969 MIRROR_EXTEND to 2 mirrors 000643 1783970161.222319 trunc from 0x040000 to 0x023aed (0x1c512 bytes) 000645 1783970161.315918 mapread 0x014170 thru 0x02136b (0xd1fc bytes) 000646 1783970161.340297 mapread 0x018a9b thru 0x01ef57 (0x64bd bytes) 000647 1783970161.341482 write 0x03a0b8 thru 0x03ffff (0x5f48 bytes) 000648 1783970161.343745 mapwrite 0x019d06 thru 0x0208a7 (0x6ba2 bytes) 000649 1783970161.394742 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xcd37b78c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xde799d26 Mirror 3: 0xde799d26 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000650 1783970162.399856 mapread 0x02f01b thru 0x039097 (0xa07d bytes) 000651 1783970162.482457 write 0x0168fd thru 0x022bdb (0xc2df bytes) 000652 1783970162.608266 read 0x02a17a thru 0x02be41 (0x1cc8 bytes) 000653 1783970162.624856 write 0x00cf33 thru 0x0176ab (0xa779 bytes) 000658 1783970162.659381 MIRROR_RESYNC 3 mirrors 000662 1783970163.442518 mapread 0x03d658 thru 0x03ffff (0x29a8 bytes) 000663 1783970163.464230 trunc from 0x040000 to 0x03a44d (0x5bb2 bytes) 000664 1783970163.522757 read 0x00c0a2 thru 0x013d8e (0x7ced bytes) 000665 1783970163.534212 mapwrite 0x03ae37 thru 0x03ffff (0x51c9 bytes) 000667 1783970163.613357 mapwrite 0x0140a4 thru 0x022a39 (0xe996 bytes) 000669 1783970163.676514 mapwrite 0x004893 thru 0x00eabb (0xa229 bytes) 000672 1783970163.778714 trunc from 0x040000 to 0x02fa79 (0x10586 bytes) 000673 1783970163.836292 trunc from 0x02fa7a to 0x0006ca (0x2f3af bytes) 000676 1783970163.866800 write 0x03b11a thru 0x03d2b7 (0x219e bytes) 000677 1783970163.873412 read 0x020e42 thru 0x02f094 (0xe253 bytes) 000678 1783970163.888787 read 0x03aa1b thru 0x03d2b7 (0x289d bytes) 000680 1783970163.890291 write 0x01257f thru 0x018688 (0x610a bytes) 000681 1783970163.897933 MIRROR_RESYNC 3 mirrors 000682 1783970164.846870 mapwrite 0x02ba9b thru 0x02f672 (0x3bd8 bytes) 000684 1783970165.052356 mapread 0x01ee62 thru 0x02c430 (0xd5cf bytes) 000686 1783970165.070487 write 0x034a84 thru 0x035216 (0x793 bytes) 000692 1783970165.086706 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3d2b8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d2b8): Mirror 2: 0x7773136e (stale) Mirror 3: 0x7773136e (stale) CRC-32 checksum value for chunk [0, 0x3d2b8): Mirror 1: 0x2d2267de Mirror 4: 0x2d2267de lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d2b8: skipped 000695 1783970165.979204 mapread 0x00caa9 thru 0x00e193 (0x16eb bytes) 000698 1783970166.035438 mapwrite 0x020758 thru 0x02a40c (0x9cb5 bytes) 000699 1783970166.355080 MIRROR_RESYNC 4 mirrors 000700 1783970167.443455 trunc from 0x03d2b8 to 0x03d515 (0x25e bytes) 000701 1783970167.515550 trunc from 0x03d516 to 0x0349e5 (0x8b30 bytes) 000702 1783970167.582264 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x349e6): Mirror 3: 0x42f94520 (stale) Mirror 4: 0x42f94520 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x349e6): Mirror 3: 0x42f94520 (stale) Mirror 4: 0x42f94520 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x349e6: skipped 000704 1783970167.918376 mapread 0x02be60 thru 0x0349e5 (0x8b86 bytes) 000706 1783970167.941162 mapwrite 0x01f9df thru 0x021cd1 (0x22f3 bytes) 000707 1783970167.993782 write 0x00c727 thru 0x014a6f (0x8349 bytes) 000708 1783970168.015602 MIRROR_RESYNC 3 mirrors 000709 1783970169.068109 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x178:0x0]: Invalid argument (22) 000710 1783970169.496233 write 0x00d1cd thru 0x0183d5 (0xb209 bytes) 000711 1783970169.608151 trunc from 0x0349e6 to 0x00d9aa (0x2703b bytes) 000713 1783970169.635591 trunc from 0x00d9ab to 0x0216ac (0x13d02 bytes) 000714 1783970169.688453 mapread 0x020018 thru 0x0216ac (0x1695 bytes) 000715 1783970169.718947 read 0x00611b thru 0x0103c3 (0xa2a9 bytes) 000721 1783970169.745748 mapread 0x0077db thru 0x010826 (0x904c bytes) 000724 1783970169.753873 write 0x024a9c thru 0x027773 (0x2cd8 bytes) 000725 1783970169.756481 trunc from 0x027774 to 0x026ce3 (0xa90 bytes) 000726 1783970169.783106 trunc from 0x026ce4 to 0x027c51 (0xf6e bytes) 000727 1783970169.821746 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x27c52): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27c52): Mirror 4: 0xa126ecda (stale) CRC-32 checksum value for chunk [0, 0x27c52): Mirror 1: 0xcffe1a7b Mirror 5: 0xcffe1a7b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27c52: skipped 000729 1783970170.834787 write 0x00ea70 thru 0x014bc7 (0x6158 bytes) 000730 1783970170.970714 mapwrite 0x016a3a thru 0x02059b (0x9b62 bytes) 000731 1783970171.170981 mapread 0x0155bb thru 0x019c95 (0x46db bytes) 000736 1783970171.181811 read 0x007ea1 thru 0x010831 (0x8991 bytes) 000738 1783970171.216802 MIRROR_RESYNC 3 mirrors 000739 1783970172.120071 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x178:0x0]: Invalid argument (22) 000740 1783970172.705157 read 0x008371 thru 0x01767d (0xf30d bytes) 000741 1783970172.715415 trunc from 0x027c52 to 0x00e4d1 (0x19780 bytes) 000742 1783970172.919214 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xe4d2): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe4d2): Mirror 5: 0xcea4e2a4 (stale) CRC-32 checksum value for chunk [0, 0xe4d2): Mirror 1: 0xcea4e2a4 Mirror 6: 0xcea4e2a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe4d2: skipped 000743 1783970174.289665 mapread 0x0065d4 thru 0x00e4d1 (0x7efe bytes) 000744 1783970174.355403 trunc from 0x00e4d2 to 0x0078ab (0x6c26 bytes) 000745 1783970174.495185 write 0x01045b thru 0x01b793 (0xb339 bytes) 000746 1783970174.503414 read 0x006a4c thru 0x00abe9 (0x419e bytes) 000747 1783970174.532298 write 0x03fe7a thru 0x03ffff (0x186 bytes) 000748 1783970174.533579 mapread 0x03f018 thru 0x03ffff (0xfe8 bytes) 000751 1783970174.541931 mapread 0x03ebe9 thru 0x03ffff (0x1417 bytes) 000752 1783970174.562178 mapread 0x03cab4 thru 0x03ffff (0x354c bytes) 000753 1783970174.573118 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x2197e51a (stale) Mirror 6: 0x973c2a92 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd9b94c52 Mirror 7: 0xd9b94c52 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000755 1783970175.649571 mapread 0x0160ff thru 0x01ff84 (0x9e86 bytes) 000756 1783970175.707287 mapread 0x016510 thru 0x02005a (0x9b4b bytes) 000757 1783970175.720239 mapread 0x0201bf thru 0x025627 (0x5469 bytes) 000758 1783970175.740977 trunc from 0x040000 to 0x038ee8 (0x7117 bytes) 000760 1783970175.880349 write 0x00232c thru 0x00288e (0x563 bytes) 000762 1783970175.890261 read 0x029aaf thru 0x034592 (0xaae4 bytes) 000765 1783970175.949897 read 0x02efd6 thru 0x038ee8 (0x9f13 bytes) 000769 1783970175.986636 read 0x004839 thru 0x011a7c (0xd244 bytes) 000770 1783970175.989550 read 0x026fd0 thru 0x02be40 (0x4e71 bytes) 000771 1783970175.990889 mapwrite 0x01022d thru 0x01d962 (0xd736 bytes) 000772 1783970176.047679 read 0x0027e2 thru 0x00c050 (0x986f bytes) 000776 1783970176.050376 mapread 0x016711 thru 0x01872d (0x201d bytes) 000781 1783970176.054550 trunc from 0x038ee9 to 0x028f5c (0xff8c bytes) 000783 1783970176.082416 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28f5d): Mirror 6: 0xc1a65f51 (stale) Mirror 7: 0xe830de87 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28f5d): Mirror 6: 0xc1a65f51 (stale) Mirror 7: 0xe830de87 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28f5d: skipped 000785 1783970176.634774 mapwrite 0x014a52 thru 0x01868a (0x3c39 bytes) 000786 1783970176.725328 write 0x03d08d thru 0x03ffff (0x2f73 bytes) 000788 1783970176.731375 read 0x00daf4 thru 0x011d36 (0x4243 bytes) 000789 1783970176.743913 read 0x03c5e9 thru 0x03ffff (0x3a17 bytes) 000791 1783970176.757658 trunc from 0x040000 to 0x022eec (0x1d113 bytes) 000792 1783970176.795474 MIRROR_RESYNC 3 mirrors 000793 1783970177.780380 write 0x022698 thru 0x02e639 (0xbfa2 bytes) 000794 1783970177.938562 mapwrite 0x001360 thru 0x00b324 (0x9fc5 bytes) 000796 1783970178.045661 read 0x028109 thru 0x02e639 (0x6531 bytes) 000797 1783970178.051785 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e63a): Mirror 7: 0x9fb754c2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e63a): Mirror 7: 0x9fb754c2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e63a: skipped 000798 1783970178.438366 mapwrite 0x01e863 thru 0x029611 (0xadaf bytes) 000802 1783970178.600175 mapread 0x02de02 thru 0x02e639 (0x838 bytes) 000803 1783970178.625379 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x178:0x0]: Invalid argument (22) 000806 1783970178.791058 read 0x01d45a thru 0x02b104 (0xdcab bytes) 000807 1783970178.800000 mapread 0x01d7ba thru 0x0205ab (0x2df2 bytes) 000808 1783970178.801940 trunc from 0x02e63a to 0x01fb92 (0xeaa7 bytes) 000809 1783970178.825546 mapwrite 0x00cfba thru 0x017f13 (0xaf5a bytes) 000810 1783970178.910763 trunc from 0x01fb93 to 0x0116a6 (0xe4ec bytes) 000811 1783970178.942562 read 0x0091c7 thru 0x00e8d9 (0x5713 bytes) 000812 1783970178.945571 read 0x008a3e thru 0x00b7ac (0x2d6f bytes) 000813 1783970178.946486 mapread 0x010590 thru 0x0116a6 (0x1117 bytes) 000814 1783970178.947526 trunc from 0x0116a7 to 0x02f60b (0x1df65 bytes) 000815 1783970178.983885 read 0x0075e6 thru 0x011b26 (0xa541 bytes) 000816 1783970178.985728 mapwrite 0x02980c thru 0x0356b0 (0xbea5 bytes) 000818 1783970179.107359 read 0x02698b thru 0x035183 (0xe7f9 bytes) 000823 1783970179.112684 mapread 0x02ab1c thru 0x0308a1 (0x5d86 bytes) 000826 1783970179.123209 MIRROR_EXTEND to 2 mirrors 000827 1783970179.957350 MIRROR_EXTEND to 3 mirrors 000828 1783970180.813942 mapread 0x017132 thru 0x01ecff (0x7bce bytes) 000829 1783970180.895652 mapwrite 0x001430 thru 0x005b91 (0x4762 bytes) 000831 1783970181.212891 trunc from 0x0356b1 to 0x02e6a1 (0x700f bytes) 000833 1783970181.242918 MIRROR_RESYNC 3 mirrors 000834 1783970182.634453 mapwrite 0x03edbe thru 0x03ffff (0x1242 bytes) 000835 1783970182.918754 mapread 0x02fb14 thru 0x03f798 (0xfc85 bytes) 000837 1783970182.972435 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe4d9d1cb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe4d9d1cb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000839 1783970183.358234 mapread 0x012e06 thru 0x01775b (0x4956 bytes) 000840 1783970183.403206 read 0x0345e6 thru 0x03ffff (0xba1a bytes) 000845 1783970183.423429 trunc from 0x040000 to 0x0091b4 (0x36e4b bytes) 000847 1783970183.454260 trunc from 0x0091b5 to 0x006505 (0x2caf bytes) 000848 1783970183.482700 write 0x00d4fe thru 0x014ef8 (0x79fb bytes) 000850 1783970183.501308 trunc from 0x014ef9 to 0x0137fd (0x16fb bytes) 000851 1783970183.554828 mapwrite 0x016d52 thru 0x01d04d (0x62fc bytes) 000853 1783970183.665191 write 0x02d8c4 thru 0x033e7f (0x65bc bytes) 000854 1783970183.667610 mapread 0x0140bd thru 0x01cde6 (0x8d2a bytes) 000855 1783970183.678702 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x178:0x0]: Invalid argument (22) 000856 1783970183.909952 write 0x0208fe thru 0x02ae09 (0xa50c bytes) 000857 1783970183.974633 mapread 0x01ef87 thru 0x022b63 (0x3bdd bytes) 000861 1783970184.037823 mapwrite 0x01a3dd thru 0x0276f3 (0xd317 bytes) 000862 1783970184.208076 write 0x021b45 thru 0x02c02a (0xa4e6 bytes) 000864 1783970184.222358 mapwrite 0x010725 thru 0x01766a (0x6f46 bytes) 000865 1783970184.318734 trunc from 0x033e80 to 0x0119c9 (0x224b6 bytes) 000866 1783970184.352518 write 0x02309a thru 0x024b01 (0x1a68 bytes) 000869 1783970184.356136 mapwrite 0x0399fb thru 0x03ffff (0x6605 bytes) 000870 1783970184.587919 mapread 0x027e93 thru 0x031536 (0x96a4 bytes) 000871 1783970184.622284 trunc from 0x040000 to 0x028bb8 (0x17447 bytes) 000874 1783970184.672990 trunc from 0x028bb9 to 0x03d27b (0x146c3 bytes) 000875 1783970184.714906 trunc from 0x03d27c to 0x01cda6 (0x204d5 bytes) 000877 1783970184.756509 mapwrite 0x00fa1c thru 0x0155b4 (0x5b99 bytes) 000879 1783970184.907603 read 0x009135 thru 0x010ab9 (0x7985 bytes) 000880 1783970184.910524 mapread 0x013b0b thru 0x014d8b (0x1281 bytes) 000881 1783970184.928478 MIRROR_EXTEND to 2 mirrors 000883 1783970186.057393 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x178:0x0]: Invalid argument (22) 000886 1783970186.355857 write 0x016b7d thru 0x017863 (0xce7 bytes) 000889 1783970186.414778 write 0x032646 thru 0x038551 (0x5f0c bytes) 000891 1783970186.425276 read 0x037f55 thru 0x038551 (0x5fd bytes) 000893 1783970186.428488 trunc from 0x038552 to 0x006cd8 (0x31879 bytes) 000895 1783970186.465470 write 0x009572 thru 0x017c14 (0xe6a3 bytes) 000896 1783970186.472475 trunc from 0x017c15 to 0x011d0e (0x5f06 bytes) 000897 1783970186.513959 trunc from 0x011d0f to 0x031586 (0x1f878 bytes) 000898 1783970186.561067 write 0x01bab5 thru 0x020aa0 (0x4fec bytes) 000900 1783970186.581415 mapwrite 0x028a9b thru 0x0338fd (0xae63 bytes) 000903 1783970186.713129 mapread 0x026fde thru 0x030990 (0x99b3 bytes) 000905 1783970186.717574 mapread 0x01f79f thru 0x028de7 (0x9649 bytes) 000907 1783970186.728993 write 0x037618 thru 0x03cbb9 (0x55a2 bytes) 000910 1783970186.732586 write 0x038834 thru 0x03ffff (0x77cc bytes) 000911 1783970186.734420 mapwrite 0x0125c2 thru 0x0172df (0x4d1e bytes) 000912 1783970186.804116 mapread 0x027ce4 thru 0x02ad89 (0x30a6 bytes) 000913 1783970186.809092 mapwrite 0x01f4ff thru 0x01f7c8 (0x2ca bytes) 000915 1783970186.869984 mapread 0x01fe4c thru 0x02a8c8 (0xaa7d bytes) 000916 1783970186.871781 mapread 0x03354d thru 0x034895 (0x1349 bytes) 000920 1783970186.906861 write 0x0063ff thru 0x015ec8 (0xfaca bytes) 000924 1783970186.927162 trunc from 0x040000 to 0x015235 (0x2adca bytes) 000927 1783970186.977518 write 0x016868 thru 0x018500 (0x1c99 bytes) 000928 1783970186.990083 MIRROR_EXTEND to 2 mirrors 000930 1783970187.866456 mapread 0x00cd17 thru 0x015d12 (0x8ffc bytes) 000931 1783970187.914595 trunc from 0x018501 to 0x034fb3 (0x1cab3 bytes) 000932 1783970188.116734 trunc from 0x034fb4 to 0x00e69c (0x26917 bytes) 000934 1783970188.167725 mapwrite 0x02a117 thru 0x031b9f (0x7a89 bytes) 000935 1783970188.445389 read 0x030b45 thru 0x031b9f (0x105b bytes) 000936 1783970188.446261 mapwrite 0x037160 thru 0x03ffff (0x8ea0 bytes) 000937 1783970188.643483 mapwrite 0x036507 thru 0x03ffff (0x9af9 bytes) 000939 1783970188.791468 mapwrite 0x037a93 thru 0x03ffff (0x856d bytes) 000940 1783970189.052534 mapwrite 0x01fb8f thru 0x02c84b (0xccbd bytes) 000941 1783970189.270210 mapwrite 0x023f95 thru 0x032ee1 (0xef4d bytes) 000942 1783970189.393686 mapread 0x004fe1 thru 0x013c01 (0xec21 bytes) 000943 1783970189.457927 mapread 0x019756 thru 0x01cc2a (0x34d5 bytes) 000946 1783970189.480645 mapwrite 0x03059b thru 0x03d6e8 (0xd14e bytes) 000948 1783970189.632178 mapwrite 0x03562f thru 0x03ffff (0xa9d1 bytes) 000950 1783970189.832163 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x178:0x0]: Invalid argument (22) 000951 1783970190.031341 write 0x02ce8d thru 0x039518 (0xc68c bytes) 000953 1783970190.055423 mapread 0x03a108 thru 0x03ffff (0x5ef8 bytes) 000954 1783970190.093465 trunc from 0x040000 to 0x036401 (0x9bfe bytes) 000955 1783970190.116640 mapread 0x006f47 thru 0x01353e (0xc5f8 bytes) 000956 1783970190.146448 trunc from 0x036402 to 0x00c8f2 (0x29b0f bytes) 000957 1783970190.166803 trunc from 0x00c8f3 to 0x00c891 (0x61 bytes) 000959 1783970190.192258 mapread 0x001d13 thru 0x00c891 (0xab7f bytes) 000960 1783970190.195819 mapread 0x00acdb thru 0x00c891 (0x1bb7 bytes) 000961 1783970190.197697 read 0x007d18 thru 0x00c891 (0x4b7a bytes) 000963 1783970190.202061 mapwrite 0x012bcf thru 0x01f7db (0xcc0d bytes) 000965 1783970190.387161 trunc from 0x01f7dc to 0x01ab09 (0x4cd2 bytes) 000966 1783970190.403325 mapwrite 0x00f09c thru 0x01241e (0x3383 bytes) 000967 1783970190.493915 mapwrite 0x008bb8 thru 0x013922 (0xad6b bytes) 000971 1783970190.645536 mapwrite 0x028b23 thru 0x030030 (0x750e bytes) 000972 1783970190.828744 MIRROR_EXTEND to 2 mirrors 000975 1783970191.739677 write 0x0090be thru 0x00b93c (0x287f bytes) 000978 1783970191.892384 MIRROR_RESYNC 2 mirrors 000979 1783970192.747549 mapwrite 0x0058cb thru 0x006689 (0xdbf bytes) 000981 1783970192.918985 mapwrite 0x02d71f thru 0x037704 (0x9fe6 bytes) 000982 1783970193.081950 read 0x006647 thru 0x00d0f6 (0x6ab0 bytes) 000983 1783970193.089515 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x178:0x0]: Invalid argument (22) 000984 1783970193.284671 read 0x022b42 thru 0x02fdc2 (0xd281 bytes) 000990 1783970193.311470 read 0x0243de thru 0x032184 (0xdda7 bytes) 000991 1783970193.344417 mapwrite 0x014635 thru 0x01ea10 (0xa3dc bytes) 000992 1783970193.429256 mapwrite 0x0067be thru 0x00d5b2 (0x6df5 bytes) 000995 1783970193.478450 read 0x00ea1e thru 0x01bb49 (0xd12c bytes) 000996 1783970193.480543 mapwrite 0x03d5bd thru 0x03ffff (0x2a43 bytes) 000997 1783970193.581823 trunc from 0x040000 to 0x01358a (0x2ca75 bytes) 000998 1783970193.600923 trunc from 0x01358b to 0x031eef (0x1e965 bytes) 000999 1783970193.617153 mapread 0x003df8 thru 0x00e5e1 (0xa7ea bytes) All operations completed A-OK! PASS 70a (87s) == sanity-flr test 71: check mirror extend parent fid ==== 15:16:39 (1783970199) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 15:16:43 (1783970203) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2be:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.33988 s, 6.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2be:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.430033 s, 2.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2be:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a3:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2be:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a3:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c0:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 15:16:54 (1783970214) Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre2 Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4473344 .. Extending file size to 1571744 .. done Extending file size to 7024448 .. Extending file size to 4845984 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6848320 .. Extending file size to 8134912 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6865568 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8170144 .. failed Extending file size to 8562912 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8725760 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8641056 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8908192 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8793440 .. Extending file size to 8827776 .. Extending file size to 9063712 .. Extending file size to 9115168 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57007 57008 57010 57011 57012 failed 192.168.203.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg322-client.virtnet /mnt/lustre2 (opts:) 192.168.203.122@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg322-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:18:17 (1783970297) Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre2 Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3373632 .. Extending file size to 6348480 .. Extending file size to 4206560 .. Extending file size to 7105088 .. Extending file size to 7746208 .. Extending file size to 8719200 .. Extending file size to 8813600 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8839520 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8918528 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 58572 58573 58575 58577 58578 192.168.203.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg322-client.virtnet /mnt/lustre2 (opts:) 192.168.203.122@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg322-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 15:19:38 (1783970378) Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.108498 s, 19.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.463328 s, 4.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x61442a79 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg322-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 15:19:58 (1783970398) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.119991 s, 17.5 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 15:20:05 (1783970405) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ba:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 15:20:13 (1783970413) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.240586 s, 8.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 15:20:21 (1783970421) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.28756 s, 7.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d9:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 15:20:30 (1783970430) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.177728 s, 11.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2da:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2da:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2db:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:21:02 (1783970462) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg322-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120563 s, 8.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bf:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bf:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } PASS 204d (63s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:22:05 (1783970525) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg322-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2de:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.209864 s, 10.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2de:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2de:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (65s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:23:11 (1783970591) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg322-server mds-ost sync done. Creating new pool test_204f oleg322-server: Pool lustre.test_204f created Adding targets to pool oleg322-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg322-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.207138 s, 10.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg322-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg322-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg322-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 15:24:32 (1783970672) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 15:24:40 (1783970680) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 15:24:49 (1783970689) oleg322-server: Pool lustre.test_206 created Adding targets to pool oleg322-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg322-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg322-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg322-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg322-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 15:25:15 (1783970715) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125803 s, 8.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587278 s, 17.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xeb32534e Mirror 3: 0xeb32534e PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 15:25:27 (1783970727) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 15:25:31 (1783970731) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 15:25:34 (1783970734) Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.12711 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2010000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2560000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2560000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2560000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1d20000 Read this (4 bytes): BBBB 192.168.203.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg322-client.virtnet /mnt/lustre2 (opts:) PASS 209a (40s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 15:26:14 (1783970774) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0363118 s, 226 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 15:26:21 (1783970781) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708754 s, 14.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1e4 lmm_fid: [0x200000404:0x1e4:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1e4 lmm_fid: [0x200000404:0x1e4:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834404 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-13 15:26:23.000000000 -0400 Modify: 2026-07-13 15:26:23.000000000 -0400 Change: 2026-07-13 15:26:24.000000000 -0400 Birth: 2026-07-13 15:26:23.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 15:26:30 (1783970790) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 15:26:33 (1783970793) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.822197 s, 49.8 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0851227 s, 48.1 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 13 15:26 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e6:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 13 15:26 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 15:26:41 (1783970801) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 15:26:50 (1783970810) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1789 sec =========== 15:26:59 (1783970819) === sanity-flr: start cleanup 15:27:01 (1783970821) === === sanity-flr: finish cleanup 15:27:09 (1783970829) ===