-----============= acceptance-small: sanity-flr ============----- Fri Jul 3 23:15:05 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg230-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 23:15:15 (1783134915) === oleg230-client.virtnet: executing check_config_client /mnt/lustre oleg230-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg230-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff966e59ace000.idle_timeout=debug osc.lustre-OST0001-osc-ffff966e59ace000.idle_timeout=debug disable quota as required oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 23:15:36 (1783134936) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7594] == sanity-flr test 0a: lfs mirror create with -N option == 23:15:37 (1783134937) PASS 0a (11s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 23:15:48 (1783134948) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 23:15:50 (1783134950) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 23:15:52 (1783134952) PASS 0d (15s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 23:16:07 (1783134967) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 23:16:09 (1783134969) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 23:16:10 (1783134970) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff966e59ace000.stats=clear osc.lustre-OST0001-osc-ffff966e59ace000.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 23:16:14 (1783134974) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (6s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 23:16:20 (1783134980) PASS 0j (5s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 23:16:25 (1783134985) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 23:16:27 (1783134987) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (5s) == sanity-flr test 1: create components with setstripe options ========================================================== 23:16:32 (1783134992) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (5s) == sanity-flr test 2: create components from existing files ========================================================== 23:16:37 (1783134997) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 23:16:41 (1783135001) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 23:16:43 (1783135003) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (5s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 23:16:48 (1783135008) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (4s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 23:16:55 (1783135015) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 23:17:00 (1783135020) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 1.3663 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1009 s, 10.4 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 23:17:08 (1783135028) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.053026 s, 79.1 MB/s PASS 22 (4s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 23:17:12 (1783135032) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (7s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 23:17:19 (1783135039) Stopping /mnt/lustre-ost1 (opts:) on oleg230-server oleg230-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff966e59ace000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff966e59ace000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg230-server: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg230-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg230-server oleg230-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff966e59ace000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff966e59ace000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg230-server: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg230-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (66s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 23:18:25 (1783135105) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 1.56915 s, 7.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg230-server oleg230-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff966e59ace000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff966e59ace000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg230-server: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg230-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (61s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 23:19:26 (1783135166) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg230-server oleg230-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff966e59ace000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff966e59ace000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg230-server: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg230-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg230-server oleg230-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff966e59ace000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff966e59ace000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg230-server: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg230-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (79s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 23:20:45 (1783135245) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.61437 s, 11.4 MB/s -rw-r--r-- 1 root root 52428800 Jul 3 23:20 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.187s user 0m0.000s sys 0m0.180s Stopping /mnt/lustre-ost1 (opts:) on oleg230-server oleg230-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff966e59ace000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff966e59ace000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.178s user 0m0.002s sys 0m0.177s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg230-server: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg230-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg230-server oleg230-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff966e59ace000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff966e59ace000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.104s user 0m0.001s sys 0m0.158s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg230-server: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg230-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg230-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff966e59ace000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff966e59ace000.ost_server_uuid in FULL state after 0 sec PASS 33b (90s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 23:22:15 (1783135335) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 23:22:19 (1783135339) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 23:22:22 (1783135342) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 23:22:27 (1783135347) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 23:22:36 (1783135356) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.704533 s, 17.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 23:22:49 (1783135369) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312437 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635143 s, 16.5 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 23:22:57 (1783135377) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 23:23:07 (1783135387) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00188091 s, 544 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00180463 s, 567 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000848678 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00185679 s, 551 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 23:23:18 (1783135398) create test files with size 15753589 .. 3e0288324b2979233679bd17691800b1 - 653e4b3222f20461b8f2c27b6f6065aa - 76b2ac22ce2159caa663402d8816a867 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 23:23:43 (1783135423) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 23:23:59 (1783135439) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 23:24:06 (1783135446) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776782 s, 13.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352148 s, 29.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xba:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 23:24:17 (1783135457) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.136629 s, 15.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.538193 s, 7.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.16947 s, 12.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 23:24:35 (1783135475) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 23:24:39 (1783135479) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 23:24:43 (1783135483) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234051 s, 44.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 23:24:52 (1783135492) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 23:24:55 (1783135495) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc1:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (12s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 23:25:09 (1783135509) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.309431 s, 33.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 23:25:22 (1783135522) llite.lustre-ffff966e582d1800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.441966 s, 23.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff966e457f0000.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 23:25:38 (1783135538) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (50s) == sanity-flr test 46: Verify setstripe --copy option ==== 23:26:29 (1783135589) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (43s) == sanity-flr test 47: Verify mirror obj alloc =========== 23:27:12 (1783135632) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 23:27:16 (1783135636) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.259084 s, 12.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.346624 s, 9.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xfda473f2 (nosync,stale) Mirror 2: 0xfda473f2 (nosync,stale) ** verify mirror 2 doesn't change original checksum: ab053ae8f813803007afd24c5730a157 - mirror 1 checksum: c106936bc610495bc212d1d601f73f54 - mirror 2 checksum: ab053ae8f813803007afd24c5730a157 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-03T23:27:20-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-03T23:27:20-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd8:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: ab053ae8f813803007afd24c5730a157 - mirror 1 checksum: c106936bc610495bc212d1d601f73f54 - mirror 2 checksum: c106936bc610495bc212d1d601f73f54 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-03T23:27:20-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-03T23:27:20-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd8:0x0] } PASS 48 (15s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 23:27:34 (1783135654) Failing mds1 on oleg230-server Stopping /mnt/lustre-mds1 (opts:) on oleg230-server 23:27:42 (1783135662) shut down facet: mds1 facet_host: oleg230-server facet_failover_host: oleg230-server Failover mds1 to oleg230-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-MDT0000 23:28:18 (1783135698) targets are mounted 23:28:18 (1783135698) facet_failover done oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457135 s, 22.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } PASS 50A (66s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 23:28:39 (1783135719) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 23:28:43 (1783135723) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 23:28:47 (1783135727) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.022656 s, 723 kB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 23:28:56 (1783135736) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 23:29:00 (1783135740) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 23:29:04 (1783135744) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (30s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 23:29:34 (1783135774) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (34s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 23:30:08 (1783135808) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 23:30:34 (1783135834) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 23:30:58 (1783135858) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 23:31:05 (1783135865) mirror create pid 53744 mirror split pid 53745 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53744 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53745 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 23:32:13 (1783135933) Seed set to 5935 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000002 1783135936.058682 mapwrite 0x02e8b8 thru 0x02f243 (0x98c bytes) 000003 1783135936.150807 mapread 0x01e5ad thru 0x0268df (0x8333 bytes) truncating to largest ever: 0x360d5 000007 1783135936.162135 trunc from 0x02f244 to 0x0360d4 (0x6e91 bytes) 000014 1783135936.210334 trunc from 0x0360d5 to 0x00786d (0x2e867 bytes) 000016 1783135936.229559 read 0x0064e2 thru 0x00786d (0x138c bytes) 000017 1783135936.235781 write 0x015da4 thru 0x01f28b (0x94e8 bytes) 000018 1783135936.238631 mapwrite 0x00c4b7 thru 0x013f27 (0x7a71 bytes) 000019 1783135936.285322 mapread 0x009b59 thru 0x013d9c (0xa244 bytes) 000022 1783135936.288323 mapread 0x01a097 thru 0x01f28b (0x51f5 bytes) 000029 1783135936.293538 mapread 0x00c4cc thru 0x015897 (0x93cc bytes) 000030 1783135936.294777 write 0x0105c9 thru 0x0199ec (0x9424 bytes) 000032 1783135936.296560 read 0x000cee thru 0x004106 (0x3419 bytes) 000033 1783135936.302883 read 0x01cd61 thru 0x01f28b (0x252b bytes) 000035 1783135936.303762 trunc from 0x01f28c to 0x01be44 (0x3447 bytes) 000040 1783135936.328749 MIRROR_EXTEND to 2 mirrors 000041 1783135937.077288 write 0x021561 thru 0x02609e (0x4b3e bytes) 000043 1783135937.157151 mapread 0x010555 thru 0x01f789 (0xf235 bytes) 000044 1783135937.201746 mapread 0x01c411 thru 0x02609e (0x9c8e bytes) 000045 1783135937.205448 mapread 0x014647 thru 0x016332 (0x1cec bytes) 000048 1783135937.207914 read 0x014702 thru 0x0175cd (0x2ecc bytes) 000050 1783135937.210871 mapread 0x02228b thru 0x02609e (0x3e14 bytes) 000052 1783135937.220743 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000055 1783135937.504538 read 0x01cea3 thru 0x02609e (0x91fc bytes) 000057 1783135937.525686 mapread 0x011380 thru 0x01bc70 (0xa8f1 bytes) 000058 1783135937.562339 write 0x034bcb thru 0x03c4a4 (0x78da bytes) 000060 1783135937.567587 mapread 0x03c26d thru 0x03c4a4 (0x238 bytes) 000061 1783135937.569972 mapread 0x02f09e thru 0x032bc2 (0x3b25 bytes) 000064 1783135937.591918 write 0x007e52 thru 0x00caa3 (0x4c52 bytes) 000067 1783135937.609201 mapread 0x02ddf0 thru 0x031eb1 (0x40c2 bytes) 000068 1783135937.618770 mapwrite 0x036949 thru 0x03aeb2 (0x456a bytes) 000070 1783135937.731105 write 0x01d6ac thru 0x024e08 (0x775d bytes) 000072 1783135937.737362 mapwrite 0x03c88a thru 0x03e5db (0x1d52 bytes) 000074 1783135937.897417 mapread 0x02dacb thru 0x030fe9 (0x351f bytes) 000078 1783135937.903905 write 0x03e1f0 thru 0x03e622 (0x433 bytes) 000080 1783135937.918295 read 0x02e768 thru 0x030560 (0x1df9 bytes) 000081 1783135937.919540 write 0x021d42 thru 0x02ee70 (0xd12f bytes) 000082 1783135937.923604 mapread 0x00e850 thru 0x017120 (0x88d1 bytes) 000088 1783135937.935964 read 0x00a2fe thru 0x01613a (0xbe3d bytes) 000089 1783135937.937738 mapread 0x00b08b thru 0x01b038 (0xffae bytes) 000091 1783135937.943971 trunc from 0x03e623 to 0x022e9b (0x1b787 bytes) 000093 1783135937.973016 mapread 0x01eeca thru 0x022e9b (0x3fd2 bytes) 000095 1783135937.981154 write 0x03ae6e thru 0x03ffff (0x5192 bytes) 000096 1783135937.983018 mapread 0x005f33 thru 0x00e38f (0x845d bytes) truncating to largest ever: 0x39719 000098 1783135938.001494 trunc from 0x040000 to 0x039718 (0x68e7 bytes) 000100 1783135938.023895 write 0x0326ae thru 0x03a0b1 (0x7a04 bytes) 000101 1783135938.025707 mapread 0x01a6ba thru 0x026019 (0xb960 bytes) 000102 1783135938.028202 trunc from 0x03a0b2 to 0x0274c6 (0x12beb bytes) 000106 1783135938.060564 mapwrite 0x00abc0 thru 0x01778e (0xcbcf bytes) 000108 1783135938.129992 write 0x030b1e thru 0x032083 (0x1566 bytes) 000110 1783135938.132560 trunc from 0x032084 to 0x007730 (0x2a953 bytes) 000111 1783135938.156370 mapwrite 0x011995 thru 0x015401 (0x3a6d bytes) 000118 1783135938.267287 write 0x022bb4 thru 0x023cd2 (0x111f bytes) 000120 1783135938.270477 write 0x03420e thru 0x034b45 (0x938 bytes) 000121 1783135938.272013 mapread 0x0184ac thru 0x02301f (0xab74 bytes) 000126 1783135938.332296 write 0x0186fa thru 0x01edef (0x66f6 bytes) 000127 1783135938.334243 read 0x02453e thru 0x031ef9 (0xd9bc bytes) 000128 1783135938.348215 mapread 0x0125fc thru 0x020552 (0xdf57 bytes) 000130 1783135938.351260 mapwrite 0x03436b thru 0x0383e8 (0x407e bytes) 000133 1783135938.430058 write 0x0052d2 thru 0x00b567 (0x6296 bytes) 000134 1783135938.432366 write 0x03fb4b thru 0x03ffff (0x4b5 bytes) 000135 1783135938.433633 read 0x02ebae thru 0x03238b (0x37de bytes) truncating to largest ever: 0x3f3c0 000137 1783135938.436425 trunc from 0x040000 to 0x03f3bf (0xc40 bytes) 000139 1783135938.465386 trunc from 0x03f3c0 to 0x011261 (0x2e15e bytes) 000140 1783135938.494295 read 0x00055c thru 0x00f861 (0xf306 bytes) 000141 1783135938.496255 trunc from 0x011262 to 0x027708 (0x164a7 bytes) 000142 1783135938.518239 mapread 0x019465 thru 0x025666 (0xc202 bytes) 000143 1783135938.535570 mapwrite 0x0389ad thru 0x03ffff (0x7653 bytes) 000144 1783135938.615340 trunc from 0x040000 to 0x00d443 (0x32bbc bytes) 000145 1783135938.640311 mapwrite 0x03d633 thru 0x03ffff (0x29cd bytes) 000146 1783135938.734043 trunc from 0x040000 to 0x033de1 (0xc21e bytes) 000148 1783135938.764211 read 0x02bb1a thru 0x0300e3 (0x45ca bytes) 000149 1783135938.796116 MIRROR_EXTEND to 2 mirrors 000150 1783135939.490660 mapwrite 0x02d2f1 thru 0x0315ca (0x42da bytes) 000152 1783135939.685062 mapwrite 0x021910 thru 0x0253fb (0x3aec bytes) 000153 1783135939.739752 read 0x005849 thru 0x014f7c (0xf734 bytes) 000154 1783135939.751420 mapwrite 0x0017a5 thru 0x00b863 (0xa0bf bytes) 000157 1783135939.814821 write 0x01a71d thru 0x029e5c (0xf740 bytes) 000158 1783135939.821733 mapread 0x0092ca thru 0x00b077 (0x1dae bytes) 000159 1783135939.822802 mapwrite 0x028747 thru 0x02f64b (0x6f05 bytes) 000160 1783135939.878719 write 0x02a934 thru 0x02aaa6 (0x173 bytes) 000163 1783135939.882717 write 0x03a952 thru 0x03ffff (0x56ae bytes) 000165 1783135939.890710 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000168 1783135940.053656 write 0x026e09 thru 0x030b39 (0x9d31 bytes) 000169 1783135940.067914 write 0x001dd6 thru 0x00d4e9 (0xb714 bytes) 000174 1783135940.082741 mapread 0x00c719 thru 0x019a6f (0xd357 bytes) 000175 1783135940.109203 mapread 0x011ae7 thru 0x020b93 (0xf0ad bytes) 000176 1783135940.111345 mapread 0x033066 thru 0x03a9c0 (0x795b bytes) 000177 1783135940.120563 mapwrite 0x002aea thru 0x00a09f (0x75b6 bytes) 000178 1783135940.172223 read 0x00c70b thru 0x00cb1d (0x413 bytes) 000179 1783135940.173149 mapread 0x01d44b thru 0x01f0dd (0x1c93 bytes) 000181 1783135940.176972 read 0x031b27 thru 0x03ffff (0xe4d9 bytes) 000185 1783135940.191469 trunc from 0x040000 to 0x0074f0 (0x38b0f bytes) 000186 1783135940.210675 read 0x001179 thru 0x0074f0 (0x6378 bytes) 000187 1783135940.211609 trunc from 0x0074f1 to 0x02eefa (0x27a0a bytes) 000188 1783135940.233916 mapread 0x01c25f thru 0x028012 (0xbdb4 bytes) 000189 1783135940.246751 read 0x00b127 thru 0x013348 (0x8222 bytes) 000192 1783135940.256561 MIRROR_EXTEND to 2 mirrors 000193 1783135941.218990 mapwrite 0x03e21d thru 0x03ffff (0x1de3 bytes) 000196 1783135941.537929 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000198 1783135941.723329 read 0x031199 thru 0x031f4d (0xdb5 bytes) 000201 1783135941.732140 trunc from 0x040000 to 0x0102dc (0x2fd23 bytes) 000209 1783135941.761145 mapwrite 0x00f136 thru 0x01985b (0xa726 bytes) 000210 1783135941.862587 mapread 0x0110f0 thru 0x01985b (0x876c bytes) 000212 1783135941.866087 read 0x016fdc thru 0x01985b (0x2880 bytes) 000219 1783135941.872878 trunc from 0x01985c to 0x000c02 (0x18c59 bytes) 000222 1783135941.897090 read 0x000359 thru 0x000c02 (0x8aa bytes) 000223 1783135941.903519 trunc from 0x000c03 to 0x03de51 (0x3d24f bytes) 000225 1783135941.927276 mapread 0x037405 thru 0x037654 (0x250 bytes) 000229 1783135941.943355 read 0x01f39c thru 0x02d318 (0xdf7d bytes) 000230 1783135941.944525 mapread 0x02cb5a thru 0x030c38 (0x40df bytes) 000231 1783135941.946247 mapwrite 0x0394c5 thru 0x03ffff (0x6b3b bytes) 000232 1783135942.047154 mapread 0x022755 thru 0x02a20c (0x7ab8 bytes) 000235 1783135942.060727 mapread 0x020256 thru 0x027665 (0x7410 bytes) 000236 1783135942.062928 trunc from 0x040000 to 0x001712 (0x3e8ed bytes) 000237 1783135942.123911 mapread 0x00038a thru 0x001712 (0x1389 bytes) 000239 1783135942.125872 trunc from 0x001713 to 0x01d47b (0x1bd69 bytes) 000240 1783135942.150151 mapread 0x01a612 thru 0x01ba19 (0x1408 bytes) 000241 1783135942.166228 read 0x0105eb thru 0x0138d7 (0x32ed bytes) 000242 1783135942.182563 mapwrite 0x016a40 thru 0x01ac02 (0x41c3 bytes) 000245 1783135942.276293 mapwrite 0x008275 thru 0x01086e (0x85fa bytes) 000246 1783135942.399318 write 0x01f2c3 thru 0x02219b (0x2ed9 bytes) 000247 1783135942.402261 mapwrite 0x006493 thru 0x007117 (0xc85 bytes) 000248 1783135942.450584 trunc from 0x02219c to 0x023336 (0x119b bytes) 000249 1783135942.474518 read 0x01515c thru 0x0231a3 (0xe048 bytes) 000256 1783135942.491102 mapwrite 0x031297 thru 0x032889 (0x15f3 bytes) 000257 1783135942.588074 MIRROR_EXTEND to 2 mirrors 000259 1783135943.171712 read 0x02e5d8 thru 0x032889 (0x42b2 bytes) 000260 1783135943.185630 read 0x03276c thru 0x032889 (0x11e bytes) 000262 1783135943.198270 trunc from 0x03288a to 0x00cbda (0x25caf bytes) 000263 1783135943.321654 MIRROR_RESYNC 2 mirrors 000268 1783135944.192299 write 0x034391 thru 0x03a634 (0x62a4 bytes) 000269 1783135944.326829 write 0x01bad3 thru 0x0246fb (0x8c29 bytes) 000272 1783135944.364584 MIRROR_RESYNC 2 mirrors 000274 1783135945.304930 trunc from 0x03a635 to 0x009871 (0x30dc3 bytes) 000275 1783135945.435376 write 0x00ece0 thru 0x01ca6b (0xdd8c bytes) 000276 1783135945.438634 read 0x0033a9 thru 0x01190f (0xe567 bytes) 000277 1783135945.465545 write 0x03974d thru 0x03ffff (0x68b3 bytes) 000278 1783135945.473427 trunc from 0x040000 to 0x01b39f (0x24c60 bytes) 000281 1783135945.550543 mapwrite 0x038694 thru 0x0392b5 (0xc22 bytes) 000283 1783135945.721816 trunc from 0x0392b6 to 0x034dda (0x44db bytes) 000284 1783135945.749303 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x34ddb): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34ddb): Mirror 2: 0xd754700 (stale) CRC-32 checksum value for chunk [0, 0x34ddb): Mirror 1: 0x11fa8c6f Mirror 3: 0x11fa8c6f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34ddb: skipped 000288 1783135946.747391 read 0x023ed3 thru 0x032fe1 (0xf10f bytes) 000289 1783135946.767685 mapread 0x00e849 thru 0x0121d2 (0x398a bytes) 000292 1783135946.837164 read 0x01347f thru 0x01555a (0x20dc bytes) 000293 1783135946.852758 MIRROR_RESYNC 3 mirrors 000295 1783135947.864871 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000296 1783135948.244714 trunc from 0x034ddb to 0x02d1e7 (0x7bf3 bytes) 000297 1783135948.423041 mapwrite 0x018ec6 thru 0x026291 (0xd3cc bytes) 000299 1783135948.581208 read 0x006adf thru 0x012b3b (0xc05d bytes) 000300 1783135948.611902 read 0x02d0aa thru 0x02d1e7 (0x13e bytes) 000301 1783135948.627240 mapwrite 0x00ed75 thru 0x01ac45 (0xbed1 bytes) 000302 1783135948.745144 trunc from 0x02d1e8 to 0x02bcdb (0x150c bytes) 000303 1783135948.798591 trunc from 0x02bcdc to 0x02b553 (0x788 bytes) 000304 1783135948.825649 mapread 0x0010ad thru 0x002e16 (0x1d6a bytes) 000306 1783135948.827416 read 0x0290a6 thru 0x02b553 (0x24ae bytes) 000307 1783135948.829888 read 0x01f8b8 thru 0x027ec0 (0x8609 bytes) 000308 1783135948.830892 write 0x016f7b thru 0x026a03 (0xfa89 bytes) 000311 1783135948.834776 mapread 0x028863 thru 0x02b553 (0x2cf1 bytes) 000312 1783135948.849709 MIRROR_RESYNC 2 mirrors 000313 1783135949.891731 trunc from 0x02b554 to 0x036aed (0xb59a bytes) 000314 1783135950.029182 read 0x008735 thru 0x0166c2 (0xdf8e bytes) 000315 1783135950.048427 mapwrite 0x020756 thru 0x0264c2 (0x5d6d bytes) 000316 1783135950.247835 mapwrite 0x0049e9 thru 0x00e8e1 (0x9ef9 bytes) 000318 1783135950.415415 read 0x006052 thru 0x008010 (0x1fbf bytes) 000319 1783135950.418287 mapread 0x01463f thru 0x02232e (0xdcf0 bytes) 000320 1783135950.442431 read 0x0322fe thru 0x036aed (0x47f0 bytes) 000321 1783135950.443178 read 0x028dda thru 0x036aed (0xdd14 bytes) 000322 1783135950.443848 mapwrite 0x037568 thru 0x0387fe (0x1297 bytes) 000323 1783135950.661315 mapwrite 0x01dd9b thru 0x027038 (0x929e bytes) 000324 1783135950.747225 mapread 0x0286a4 thru 0x02b4b7 (0x2e14 bytes) 000325 1783135950.753148 read 0x017962 thru 0x022363 (0xaa02 bytes) 000326 1783135950.763925 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x387ff): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x387ff): Mirror 3: 0x1f9aa576 (stale) CRC-32 checksum value for chunk [0, 0x387ff): Mirror 1: 0x49499e42 Mirror 4: 0x49499e42 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x387ff: skipped 000327 1783135952.432891 mapwrite 0x01e84f thru 0x0273d8 (0x8b8a bytes) 000328 1783135952.806555 write 0x010adf thru 0x01d52d (0xca4f bytes) 000331 1783135952.819320 write 0x02793a thru 0x0333db (0xbaa2 bytes) 000332 1783135952.833371 write 0x03b66c thru 0x03ffff (0x4994 bytes) 000334 1783135952.839593 mapwrite 0x009469 thru 0x0171c4 (0xdd5c bytes) 000335 1783135953.171415 write 0x0053a3 thru 0x01350b (0xe169 bytes) 000337 1783135953.205460 mapread 0x01d39e thru 0x01e9d7 (0x163a bytes) 000338 1783135953.216241 mapread 0x03e137 thru 0x03ffff (0x1ec9 bytes) 000340 1783135953.225135 mapwrite 0x027b06 thru 0x035690 (0xdb8b bytes) 000341 1783135953.392676 read 0x03fa85 thru 0x03ffff (0x57b bytes) 000342 1783135953.395428 trunc from 0x040000 to 0x02cc8d (0x13372 bytes) 000343 1783135953.469402 MIRROR_RESYNC 3 mirrors 000346 1783135954.471146 MIRROR_EXTEND to 4 mirrors 000348 1783135955.196997 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000350 1783135955.684629 trunc from 0x02cc8e to 0x011de6 (0x1aea7 bytes) 000352 1783135955.842247 write 0x01c69d thru 0x021e48 (0x57ac bytes) 000353 1783135955.845348 read 0x01fe70 thru 0x021e48 (0x1fd9 bytes) 000354 1783135955.847216 write 0x0131bc thru 0x01a66a (0x74af bytes) 000356 1783135955.865722 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21e49): Mirror 4: 0x1c9143f6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21e49): Mirror 4: 0x1c9143f6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21e49: skipped 000357 1783135956.211847 mapwrite 0x00040c thru 0x008052 (0x7c47 bytes) 000358 1783135956.296924 read 0x01f414 thru 0x01fa36 (0x623 bytes) 000359 1783135956.303234 trunc from 0x021e49 to 0x022c3b (0xdf3 bytes) 000361 1783135956.334118 mapread 0x00e7c9 thru 0x0161b5 (0x79ed bytes) 000362 1783135956.349093 read 0x0215a0 thru 0x022808 (0x1269 bytes) 000364 1783135956.364073 trunc from 0x022c3c to 0x02b762 (0x8b27 bytes) 000366 1783135956.401369 read 0x0220b5 thru 0x02b762 (0x96ae bytes) 000369 1783135956.436179 write 0x01a33b thru 0x01ffcd (0x5c93 bytes) 000370 1783135956.441428 write 0x00e80b thru 0x017257 (0x8a4d bytes) 000372 1783135956.444924 write 0x011611 thru 0x01846b (0x6e5b bytes) 000374 1783135956.448807 trunc from 0x02b763 to 0x00f618 (0x1c14a bytes) 000376 1783135956.487393 mapread 0x004a8e thru 0x0061ca (0x173d bytes) 000377 1783135956.490039 mapwrite 0x0123d0 thru 0x01a331 (0x7f62 bytes) 000378 1783135956.636043 mapwrite 0x023c54 thru 0x02da34 (0x9de1 bytes) 000380 1783135956.772830 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2da35): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2da35): Mirror 4: 0x5656f0ab (stale) CRC-32 checksum value for chunk [0, 0x2da35): Mirror 1: 0x88acf0db Mirror 5: 0x88acf0db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2da35: skipped 000383 1783135957.889895 mapread 0x013d07 thru 0x01863f (0x4939 bytes) 000384 1783135957.950116 mapwrite 0x0359dd thru 0x03ffff (0xa623 bytes) 000386 1783135958.127337 write 0x01c989 thru 0x023cf2 (0x736a bytes) 000388 1783135958.146565 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x999d5aa9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x999d5aa9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000389 1783135958.682910 mapwrite 0x007b63 thru 0x007ff9 (0x497 bytes) 000390 1783135958.731332 mapread 0x03d789 thru 0x03ffff (0x2877 bytes) 000392 1783135958.751355 MIRROR_RESYNC 2 mirrors 000393 1783135959.562427 read 0x0178f0 thru 0x024188 (0xc899 bytes) 000394 1783135959.575203 mapwrite 0x02f944 thru 0x03c635 (0xccf2 bytes) 000396 1783135959.821305 read 0x01a0b2 thru 0x0240b5 (0xa004 bytes) 000397 1783135959.836536 mapread 0x00d3c4 thru 0x017827 (0xa464 bytes) 000398 1783135959.838552 read 0x0203c8 thru 0x02dd58 (0xd991 bytes) 000400 1783135959.842214 read 0x03ff96 thru 0x03ffff (0x6a bytes) 000401 1783135959.843480 read 0x01a739 thru 0x02a23c (0xfb04 bytes) 000402 1783135959.845659 mapread 0x024f45 thru 0x02b4cc (0x6588 bytes) 000404 1783135959.849308 write 0x02dc6a thru 0x0367dd (0x8b74 bytes) 000405 1783135959.854496 mapwrite 0x0056a7 thru 0x0115af (0xbf09 bytes) 000406 1783135959.940096 write 0x01fb46 thru 0x023a2b (0x3ee6 bytes) 000408 1783135959.944343 trunc from 0x040000 to 0x00a36e (0x35c91 bytes) 000411 1783135959.980370 trunc from 0x00a36f to 0x037dca (0x2da5c bytes) 000413 1783135960.009113 read 0x00287b thru 0x00e83a (0xbfc0 bytes) 000415 1783135960.059732 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000416 1783135960.314190 trunc from 0x037dcb to 0x034005 (0x3dc5 bytes) 000418 1783135960.351393 trunc from 0x034006 to 0x00f6c7 (0x2493e bytes) 000419 1783135960.385814 mapwrite 0x02dcbb thru 0x03c337 (0xe67d bytes) 000420 1783135960.634149 mapread 0x03050e thru 0x039a55 (0x9548 bytes) 000421 1783135960.647548 write 0x0373a0 thru 0x03ffff (0x8c60 bytes) 000422 1783135960.655586 MIRROR_EXTEND to 2 mirrors 000426 1783135961.311204 write 0x028f72 thru 0x03261a (0x96a9 bytes) 000427 1783135961.381003 trunc from 0x040000 to 0x009b45 (0x364ba bytes) 000429 1783135961.405601 mapwrite 0x029a4a thru 0x038ca9 (0xf260 bytes) 000430 1783135961.511324 mapread 0x0054a9 thru 0x0145c9 (0xf121 bytes) 000431 1783135961.532875 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x38caa): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38caa): Mirror 2: 0x110aede0 (stale) CRC-32 checksum value for chunk [0, 0x38caa): Mirror 1: 0xd53f7778 Mirror 3: 0xd53f7778 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38caa: skipped 000432 1783135962.393187 trunc from 0x038caa to 0x00dd0b (0x2af9e bytes) 000434 1783135962.512909 read 0x009513 thru 0x00dd0b (0x47f9 bytes) 000435 1783135962.529551 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xdd0c): Mirror 2: 0x6f163153 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdd0c): Mirror 2: 0x6f163153 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdd0c: skipped 000436 1783135962.855881 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000438 1783135963.004237 read 0x009b7e thru 0x00dd0b (0x418e bytes) 000440 1783135963.013882 read 0x006c3b thru 0x00dd0b (0x70d1 bytes) 000442 1783135963.029902 mapwrite 0x035247 thru 0x03ffff (0xadb9 bytes) 000443 1783135963.122293 mapread 0x0280c3 thru 0x02e6fe (0x663c bytes) 000447 1783135963.129294 write 0x01ef64 thru 0x0274a1 (0x853e bytes) 000448 1783135963.131779 mapread 0x03b089 thru 0x03ffff (0x4f77 bytes) 000450 1783135963.134584 mapread 0x030790 thru 0x03ce0d (0xc67e bytes) 000451 1783135963.137368 mapread 0x012888 thru 0x01d656 (0xadcf bytes) 000455 1783135963.142702 mapread 0x0114f8 thru 0x018ecf (0x79d8 bytes) 000457 1783135963.144757 read 0x004c79 thru 0x005279 (0x601 bytes) 000458 1783135963.146874 read 0x007996 thru 0x007ab3 (0x11e bytes) 000461 1783135963.150163 trunc from 0x040000 to 0x028acb (0x17534 bytes) 000462 1783135963.171888 mapread 0x009dc4 thru 0x012110 (0x834d bytes) 000463 1783135963.175153 trunc from 0x028acc to 0x02e85a (0x5d8f bytes) 000464 1783135963.193356 mapread 0x00d542 thru 0x01ac44 (0xd703 bytes) 000466 1783135963.201513 MIRROR_EXTEND to 2 mirrors 000471 1783135963.845801 MIRROR_EXTEND to 3 mirrors 000472 1783135964.453962 trunc from 0x02e85b to 0x00e34f (0x2050b bytes) 000473 1783135964.580317 mapread 0x002a0d thru 0x0044de (0x1ad2 bytes) 000475 1783135964.600221 read 0x0022e3 thru 0x00da8f (0xb7ad bytes) 000476 1783135964.601512 trunc from 0x00e350 to 0x001802 (0xcb4d bytes) 000478 1783135964.628147 MIRROR_RESYNC 3 mirrors 000479 1783135965.578719 write 0x008807 thru 0x00928b (0xa85 bytes) 000480 1783135965.638165 read 0x002ec4 thru 0x006c79 (0x3db6 bytes) 000482 1783135965.647854 write 0x03449d thru 0x03de82 (0x99e6 bytes) 000484 1783135965.651434 write 0x0192fa thru 0x0211fa (0x7f01 bytes) 000485 1783135965.665870 MIRROR_RESYNC 3 mirrors 000487 1783135966.449186 trunc from 0x03de83 to 0x000c98 (0x3d1ea bytes) 000488 1783135966.566030 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xc99): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc99): Mirror 2: 0x44c4df26 (stale) Mirror 3: 0x44c4df26 (stale) CRC-32 checksum value for chunk [0, 0xc99): Mirror 1: 0x44c4df26 Mirror 4: 0x44c4df26 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc99: skipped 000497 1783135967.694687 trunc from 0x000c99 to 0x00a158 (0x94c0 bytes) 000499 1783135967.817721 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xa159): Mirror 2: 0xc39b9da2 (stale) Mirror 3: 0xc39b9da2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa159): Mirror 2: 0xc39b9da2 (stale) Mirror 3: 0xc39b9da2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa159: skipped 000501 1783135968.243172 read 0x004e50 thru 0x009b2c (0x4cdd bytes) 000503 1783135968.262072 read 0x00064f thru 0x00a158 (0x9b0a bytes) 000504 1783135968.274666 mapwrite 0x026268 thru 0x026ded (0xb86 bytes) 000506 1783135968.421689 mapwrite 0x02e00f thru 0x038a0d (0xa9ff bytes) 000508 1783135968.519262 mapread 0x0054eb thru 0x0060dd (0xbf3 bytes) 000509 1783135968.525231 read 0x01385b thru 0x016ebd (0x3663 bytes) 000510 1783135968.526493 read 0x0303d6 thru 0x038a0d (0x8638 bytes) 000511 1783135968.532267 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38a0e): Mirror 2: 0xbcb96c8d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38a0e): Mirror 2: 0xbcb96c8d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38a0e: skipped 000512 1783135968.924391 mapread 0x02a069 thru 0x02ab51 (0xae9 bytes) 000513 1783135968.931500 mapread 0x0013a6 thru 0x006d4f (0x59aa bytes) 000514 1783135968.957704 mapwrite 0x0025d0 thru 0x00af84 (0x89b5 bytes) 000516 1783135969.036168 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000517 1783135969.200025 write 0x026390 thru 0x02da22 (0x7693 bytes) 000519 1783135969.214097 mapwrite 0x0026b4 thru 0x00b266 (0x8bb3 bytes) 000520 1783135969.292297 write 0x014ab2 thru 0x024021 (0xf570 bytes) 000523 1783135969.306453 read 0x022c58 thru 0x02fb14 (0xcebd bytes) 000524 1783135969.315168 trunc from 0x038a0e to 0x00b933 (0x2d0da bytes) 000530 1783135969.353752 mapread 0x001414 thru 0x007588 (0x6175 bytes) 000533 1783135969.358652 mapread 0x006d46 thru 0x00b933 (0x4bee bytes) 000536 1783135969.365493 read 0x0019d9 thru 0x00b933 (0x9f5b bytes) 000538 1783135969.368184 write 0x03ee2b thru 0x03ffff (0x11d5 bytes) 000539 1783135969.370499 write 0x03e3b4 thru 0x03ffff (0x1c4c bytes) 000540 1783135969.372526 mapread 0x0199d9 thru 0x020666 (0x6c8e bytes) 000541 1783135969.391573 write 0x010ebb thru 0x018e60 (0x7fa6 bytes) 000544 1783135969.400701 trunc from 0x040000 to 0x002ca1 (0x3d35e bytes) 000545 1783135969.420275 trunc from 0x002ca2 to 0x015d0b (0x1306a bytes) 000546 1783135969.447281 mapread 0x005dde thru 0x00928e (0x34b1 bytes) 000547 1783135969.480920 mapwrite 0x02632e thru 0x02ee3c (0x8b0f bytes) 000548 1783135969.611210 mapread 0x026b62 thru 0x02ee3c (0x82db bytes) 000550 1783135969.619448 MIRROR_EXTEND to 2 mirrors 000551 1783135970.323176 read 0x00abd8 thru 0x017042 (0xc46b bytes) 000553 1783135970.362139 read 0x002b5a thru 0x0084ee (0x5995 bytes) 000556 1783135970.393092 write 0x01d852 thru 0x029a65 (0xc214 bytes) 000557 1783135970.525204 read 0x009bc2 thru 0x0157d7 (0xbc16 bytes) 000558 1783135970.541982 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2ee3d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ee3d): Mirror 2: 0x9a78760c (stale) CRC-32 checksum value for chunk [0, 0x2ee3d): Mirror 1: 0x20baed88 Mirror 3: 0x20baed88 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ee3d: skipped 000559 1783135971.457610 mapwrite 0x008d7f thru 0x0180ad (0xf32f bytes) 000568 1783135971.806123 read 0x01859d thru 0x0254ad (0xcf11 bytes) 000570 1783135971.821410 write 0x02aff4 thru 0x038dda (0xdde7 bytes) 000574 1783135971.843964 mapread 0x032b35 thru 0x038dda (0x62a6 bytes) 000575 1783135971.849921 write 0x015fc0 thru 0x022bdb (0xcc1c bytes) 000577 1783135971.860528 MIRROR_RESYNC 3 mirrors 000579 1783135972.981716 mapread 0x026e23 thru 0x029016 (0x21f4 bytes) 000580 1783135973.010769 mapwrite 0x014141 thru 0x014b3e (0x9fe bytes) 000581 1783135973.184284 trunc from 0x038ddb to 0x0031d2 (0x35c08 bytes) 000583 1783135973.210021 mapread 0x0024df thru 0x002c93 (0x7b5 bytes) 000584 1783135973.218462 read 0x000a5a thru 0x0031d2 (0x2779 bytes) 000585 1783135973.228800 write 0x03ef80 thru 0x03ffff (0x1080 bytes) 000587 1783135973.232351 mapread 0x01b94d thru 0x023912 (0x7fc6 bytes) 000588 1783135973.243204 write 0x0147a4 thru 0x020533 (0xbd90 bytes) 000591 1783135973.257658 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x5175f26 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x5175f26 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000592 1783135973.807824 mapread 0x0038e9 thru 0x008e52 (0x556a bytes) 000594 1783135973.835216 read 0x03cf78 thru 0x03ffff (0x3088 bytes) 000595 1783135973.842830 read 0x014936 thru 0x022cc0 (0xe38b bytes) 000596 1783135973.857559 trunc from 0x040000 to 0x014b3e (0x2b4c1 bytes) 000598 1783135973.889130 trunc from 0x014b3f to 0x022340 (0xd802 bytes) 000604 1783135973.934637 mapwrite 0x01c293 thru 0x01dca5 (0x1a13 bytes) 000606 1783135973.993796 write 0x0005a2 thru 0x00ffa7 (0xfa06 bytes) 000608 1783135973.998360 trunc from 0x022341 to 0x00b2df (0x17061 bytes) 000609 1783135974.023741 trunc from 0x00b2e0 to 0x019b0f (0xe830 bytes) 000610 1783135974.049337 trunc from 0x019b10 to 0x009501 (0x1060e bytes) 000615 1783135974.086506 read 0x00321e thru 0x009501 (0x62e4 bytes) 000616 1783135974.088284 read 0x002d3b thru 0x00642c (0x36f2 bytes) 000617 1783135974.094451 write 0x02bc3e thru 0x02f1d3 (0x3596 bytes) 000618 1783135974.102210 trunc from 0x02f1d4 to 0x027f45 (0x728e bytes) 000619 1783135974.135247 mapread 0x00897c thru 0x011138 (0x87bd bytes) 000621 1783135974.168092 read 0x016557 thru 0x0211b7 (0xac61 bytes) 000625 1783135974.176996 mapwrite 0x03434c thru 0x035a0c (0x16c1 bytes) 000627 1783135974.309869 write 0x0081bf thru 0x0168f2 (0xe734 bytes) 000628 1783135974.312149 mapread 0x0071f8 thru 0x00bd30 (0x4b39 bytes) 000629 1783135974.314440 read 0x014825 thru 0x01ae1a (0x65f6 bytes) 000631 1783135974.318701 mapwrite 0x0202db thru 0x024330 (0x4056 bytes) 000632 1783135974.409302 MIRROR_RESYNC 2 mirrors 000633 1783135975.160793 write 0x02ad05 thru 0x03a0da (0xf3d6 bytes) 000635 1783135975.268495 read 0x0381e2 thru 0x03a0da (0x1ef9 bytes) 000636 1783135975.283626 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3a0db): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a0db): Mirror 3: 0xe309688d (stale) CRC-32 checksum value for chunk [0, 0x3a0db): Mirror 1: 0x5d56e8a2 Mirror 4: 0x5d56e8a2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a0db: skipped 000637 1783135976.311691 mapread 0x003283 thru 0x007020 (0x3d9e bytes) 000638 1783135976.352892 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a0db): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a0db): Mirror 3: 0xe309688d (stale) CRC-32 checksum value for chunk [0, 0x3a0db): Mirror 1: 0x5d56e8a2 Mirror 4: 0x5d56e8a2 Mirror 5: 0x5d56e8a2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a0db: skipped 000639 1783135977.348426 mapread 0x011226 thru 0x0174cd (0x62a8 bytes) 000640 1783135977.374408 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3a0db): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a0db): Mirror 3: 0xe309688d (stale) CRC-32 checksum value for chunk [0, 0x3a0db): Mirror 1: 0x5d56e8a2 Mirror 4: 0x5d56e8a2 Mirror 5: 0x5d56e8a2 Mirror 6: 0x5d56e8a2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a0db: skipped 000642 1783135978.357903 read 0x0267a9 thru 0x034f58 (0xe7b0 bytes) 000643 1783135978.384290 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3a0db): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a0db): Mirror 3: 0xe309688d (stale) CRC-32 checksum value for chunk [0, 0x3a0db): Mirror 1: 0x5d56e8a2 Mirror 4: 0x5d56e8a2 Mirror 5: 0x5d56e8a2 Mirror 6: 0x5d56e8a2 Mirror 7: 0x5d56e8a2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a0db: skipped 000645 1783135979.489842 mapwrite 0x026d73 thru 0x0333a4 (0xc632 bytes) 000646 1783135979.737702 trunc from 0x03a0db to 0x030814 (0x98c6 bytes) 000648 1783135979.793652 read 0x01e831 thru 0x02a73c (0xbf0c bytes) 000651 1783135979.806863 mapwrite 0x013ff5 thru 0x017f4a (0x3f56 bytes) 000652 1783135979.870275 trunc from 0x030815 to 0x02d797 (0x307d bytes) 000653 1783135979.896695 MIRROR_RESYNC 6 mirrors 000657 1783135981.373321 trunc from 0x02d798 to 0x021094 (0xc703 bytes) 000658 1783135981.454930 trunc from 0x021095 to 0x03931e (0x1828a bytes) 000659 1783135981.486813 read 0x010784 thru 0x01cc76 (0xc4f3 bytes) 000660 1783135981.496106 mapwrite 0x00cb0f thru 0x013a0d (0x6eff bytes) 000662 1783135981.574460 trunc from 0x03931f to 0x002cb8 (0x36666 bytes) 000664 1783135981.607044 trunc from 0x002cb9 to 0x03ebec (0x3bf34 bytes) 000665 1783135981.636247 read 0x01485d thru 0x022fa8 (0xe74c bytes) 000667 1783135981.652841 trunc from 0x03ebed to 0x016007 (0x28be5 bytes) 000668 1783135981.684830 trunc from 0x016008 to 0x03b31d (0x25316 bytes) 000669 1783135981.716740 write 0x009ea4 thru 0x012374 (0x84d1 bytes) 000670 1783135981.718635 read 0x005b8c thru 0x00de66 (0x82db bytes) 000671 1783135981.730391 trunc from 0x03b31e to 0x0034cc (0x37e51 bytes) 000673 1783135981.776158 mapread 0x002702 thru 0x0034cc (0xdcb bytes) 000677 1783135981.808692 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34cd): Mirror 4: 0xc8440646 (stale) Mirror 5: 0xc8440646 (stale) Mirror 6: 0xc8440646 (stale) Mirror 7: 0xc8440646 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34cd): Mirror 4: 0xc8440646 (stale) Mirror 5: 0xc8440646 (stale) Mirror 6: 0xc8440646 (stale) Mirror 7: 0xc8440646 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34cd: skipped 000678 1783135982.308519 mapwrite 0x015037 thru 0x021af4 (0xcabe bytes) 000679 1783135982.539739 mapread 0x012436 thru 0x01a1f3 (0x7dbe bytes) 000681 1783135982.544726 mapread 0x0186e2 thru 0x021af4 (0x9413 bytes) 000682 1783135982.546099 trunc from 0x021af5 to 0x028f93 (0x749f bytes) 000686 1783135982.583286 trunc from 0x028f94 to 0x02bb6b (0x2bd8 bytes) 000687 1783135982.631512 write 0x0373b2 thru 0x03ffff (0x8c4e bytes) 000689 1783135982.637378 trunc from 0x040000 to 0x01aa9e (0x25561 bytes) 000691 1783135982.681799 read 0x0178a3 thru 0x01aa9e (0x31fc bytes) 000692 1783135982.687915 MIRROR_RESYNC 5 mirrors 000693 1783135983.814601 write 0x02887b thru 0x029144 (0x8ca bytes) 000694 1783135983.940540 mapwrite 0x02152d thru 0x024488 (0x2f5c bytes) 000695 1783135984.060707 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29145): Mirror 4: 0xef6d07b5 (stale) Mirror 6: 0xef6d07b5 (stale) Mirror 7: 0xef6d07b5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29145): Mirror 4: 0xef6d07b5 (stale) Mirror 6: 0xef6d07b5 (stale) Mirror 7: 0xef6d07b5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29145: skipped 000696 1783135984.713796 mapread 0x01c57b thru 0x020a30 (0x44b6 bytes) 000697 1783135984.800682 write 0x010620 thru 0x01e305 (0xdce6 bytes) 000698 1783135984.829806 mapwrite 0x00aa71 thru 0x00de9b (0x342b bytes) 000699 1783135984.960899 write 0x0162e0 thru 0x025406 (0xf127 bytes) 000702 1783135984.977673 write 0x02b40d thru 0x032ab2 (0x76a6 bytes) 000703 1783135984.980478 mapread 0x024d0e thru 0x0305ec (0xb8df bytes) 000705 1783135985.006243 trunc from 0x032ab3 to 0x03f1ad (0xc6fb bytes) 000706 1783135985.045879 mapread 0x018781 thru 0x01d3f4 (0x4c74 bytes) 000709 1783135985.053677 mapread 0x003ae1 thru 0x00971f (0x5c3f bytes) 000710 1783135985.059541 mapread 0x02fc63 thru 0x03602c (0x63ca bytes) 000713 1783135985.084867 mapread 0x03b7e2 thru 0x03f1ad (0x39cc bytes) 000714 1783135985.110412 write 0x022c72 thru 0x02b362 (0x86f1 bytes) 000715 1783135985.113525 trunc from 0x03f1ae to 0x03d37c (0x1e31 bytes) 000716 1783135985.135232 trunc from 0x03d37d to 0x0154fd (0x27e7f bytes) 000717 1783135985.178836 read 0x00b894 thru 0x0154fd (0x9c6a bytes) 000718 1783135985.181676 read 0x002642 thru 0x00cd4e (0xa70d bytes) 000719 1783135985.199140 mapread 0x0033e4 thru 0x00f480 (0xc09d bytes) 000722 1783135985.205571 mapread 0x000241 thru 0x00e410 (0xe1d0 bytes) 000723 1783135985.216329 write 0x01b6e9 thru 0x022e2a (0x7742 bytes) 000724 1783135985.220797 mapread 0x013044 thru 0x014b82 (0x1b3f bytes) 000725 1783135985.231402 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22e2b): Mirror 6: 0xbab07706 (stale) Mirror 7: 0xbab07706 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22e2b): Mirror 6: 0xbab07706 (stale) Mirror 7: 0xbab07706 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22e2b: skipped 000726 1783135985.738884 mapwrite 0x03df19 thru 0x03ea4a (0xb32 bytes) 000727 1783135985.808203 write 0x03ae5c thru 0x03ffff (0x51a4 bytes) 000729 1783135985.818397 mapread 0x01b295 thru 0x01e16d (0x2ed9 bytes) 000730 1783135985.832791 trunc from 0x040000 to 0x0129c0 (0x2d63f bytes) 000732 1783135985.868390 mapwrite 0x007494 thru 0x012616 (0xb183 bytes) 000733 1783135985.950682 mapread 0x001598 thru 0x00cb5b (0xb5c4 bytes) 000734 1783135985.955363 trunc from 0x0129c1 to 0x0242ea (0x1192a bytes) 000735 1783135985.973275 trunc from 0x0242eb to 0x03224e (0xdf64 bytes) 000736 1783135985.992992 write 0x018235 thru 0x01de37 (0x5c03 bytes) 000737 1783135986.003185 mapwrite 0x02fc1a thru 0x03b1bf (0xb5a6 bytes) 000738 1783135986.091068 mapwrite 0x02507b thru 0x030d62 (0xbce8 bytes) 000740 1783135986.157133 mapwrite 0x00ab13 thru 0x00b999 (0xe87 bytes) 000741 1783135986.196906 read 0x01889a thru 0x01cd92 (0x44f9 bytes) 000745 1783135986.203380 mapwrite 0x020410 thru 0x02444f (0x4040 bytes) 000746 1783135986.256741 mapread 0x01c521 thru 0x0299c2 (0xd4a2 bytes) 000748 1783135986.261361 mapwrite 0x03262a thru 0x03ffff (0xd9d6 bytes) 000749 1783135986.372126 mapwrite 0x00054c thru 0x005b4c (0x5601 bytes) 000751 1783135986.472146 write 0x01cc22 thru 0x025023 (0x8402 bytes) 000754 1783135986.476710 mapwrite 0x01e168 thru 0x02dbe0 (0xfa79 bytes) 000758 1783135986.546806 mapread 0x032a78 thru 0x03ffff (0xd588 bytes) 000760 1783135986.548815 read 0x024a6d thru 0x02e885 (0x9e19 bytes) 000761 1783135986.554107 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xf6d4ef3e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xf6d4ef3e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000762 1783135987.091027 mapwrite 0x03e93d thru 0x03ffff (0x16c3 bytes) 000763 1783135987.207001 mapread 0x03865e thru 0x03ed3b (0x66de bytes) 000764 1783135987.233889 read 0x005ef8 thru 0x011304 (0xb40d bytes) 000765 1783135987.246649 mapwrite 0x00de0c thru 0x01b1d9 (0xd3ce bytes) 000766 1783135987.301980 mapwrite 0x01cd8d thru 0x0249a7 (0x7c1b bytes) 000768 1783135987.362052 write 0x03e3d8 thru 0x03ffff (0x1c28 bytes) 000771 1783135987.364555 mapread 0x00ef07 thru 0x015102 (0x61fc bytes) 000772 1783135987.369806 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000775 1783135987.577769 read 0x03b6b9 thru 0x03ffff (0x4947 bytes) 000782 1783135987.597899 write 0x004071 thru 0x01299b (0xe92b bytes) 000784 1783135987.615132 read 0x01cb94 thru 0x029146 (0xc5b3 bytes) 000785 1783135987.643679 write 0x01e3ba thru 0x023edd (0x5b24 bytes) 000787 1783135987.645653 write 0x010d5d thru 0x01706b (0x630f bytes) 000789 1783135987.647813 write 0x01a018 thru 0x025b44 (0xbb2d bytes) 000792 1783135987.659870 MIRROR_EXTEND to 2 mirrors 000795 1783135988.568514 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000797 1783135988.742580 mapwrite 0x006c7d thru 0x0077f3 (0xb77 bytes) 000798 1783135988.855673 trunc from 0x040000 to 0x0046eb (0x3b914 bytes) 000800 1783135988.907282 write 0x027d7d thru 0x02a1fe (0x2482 bytes) 000801 1783135988.908784 read 0x02364a thru 0x02a1fe (0x6bb5 bytes) 000804 1783135988.923508 read 0x002b6e thru 0x007c62 (0x50f5 bytes) 000805 1783135988.937731 read 0x007183 thru 0x010f8b (0x9e09 bytes) 000806 1783135988.938851 trunc from 0x02a1ff to 0x017eb6 (0x12348 bytes) 000809 1783135988.971560 write 0x024c53 thru 0x02a812 (0x5bc0 bytes) 000812 1783135988.981063 read 0x016655 thru 0x021631 (0xafdd bytes) 000814 1783135988.992860 mapread 0x0090c5 thru 0x013075 (0x9fb1 bytes) 000815 1783135988.994667 mapread 0x0143f6 thru 0x017570 (0x317b bytes) 000816 1783135989.001631 trunc from 0x02a813 to 0x03f2cd (0x14abb bytes) 000817 1783135989.025874 mapread 0x020058 thru 0x0250a5 (0x504e bytes) 000818 1783135989.027674 mapwrite 0x012efe thru 0x01c617 (0x971a bytes) 000819 1783135989.142357 trunc from 0x03f2ce to 0x0373a8 (0x7f25 bytes) 000820 1783135989.189737 mapread 0x003b7a thru 0x010233 (0xc6ba bytes) 000821 1783135989.198683 mapread 0x01aa11 thru 0x01e6b5 (0x3ca5 bytes) 000823 1783135989.203391 mapread 0x01edb1 thru 0x02a0a5 (0xb2f5 bytes) 000829 1783135989.227346 MIRROR_EXTEND to 2 mirrors 000830 1783135990.125640 write 0x00393c thru 0x006123 (0x27e8 bytes) 000834 1783135990.267699 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000835 1783135990.456701 write 0x034ad5 thru 0x037304 (0x2830 bytes) 000838 1783135990.474490 mapwrite 0x01e9e8 thru 0x023ea0 (0x54b9 bytes) 000839 1783135990.560506 read 0x030776 thru 0x03535e (0x4be9 bytes) 000841 1783135990.570296 write 0x00c34f thru 0x016cb6 (0xa968 bytes) 000842 1783135990.592348 read 0x00e558 thru 0x01be14 (0xd8bd bytes) 000844 1783135990.610640 mapread 0x009ce0 thru 0x00f21c (0x553d bytes) 000845 1783135990.613422 write 0x024573 thru 0x029c59 (0x56e7 bytes) 000846 1783135990.615783 mapwrite 0x033d47 thru 0x03ca44 (0x8cfe bytes) 000847 1783135990.733645 MIRROR_EXTEND to 2 mirrors 000849 1783135991.433863 trunc from 0x03ca45 to 0x0089dd (0x34067 bytes) 000850 1783135991.553150 write 0x005dd4 thru 0x00fdf1 (0xa01e bytes) 000851 1783135991.562575 mapread 0x00546c thru 0x005dd9 (0x96e bytes) 000852 1783135991.564507 mapwrite 0x02bf72 thru 0x02e003 (0x2092 bytes) 000853 1783135991.661641 mapwrite 0x01ec60 thru 0x024197 (0x5538 bytes) 000856 1783135991.729821 mapread 0x022362 thru 0x02d875 (0xb514 bytes) 000857 1783135991.732324 read 0x028238 thru 0x02e003 (0x5dcc bytes) 000858 1783135991.733486 write 0x02b998 thru 0x0304b5 (0x4b1e bytes) 000859 1783135991.736119 write 0x03a0ca thru 0x03c3f9 (0x2330 bytes) 000861 1783135991.739522 read 0x011133 thru 0x01acd7 (0x9ba5 bytes) 000862 1783135991.748779 mapread 0x010e5f thru 0x01d21a (0xc3bc bytes) 000863 1783135991.750318 mapwrite 0x03881f thru 0x03ffff (0x77e1 bytes) 000864 1783135991.822560 trunc from 0x040000 to 0x005492 (0x3ab6d bytes) 000865 1783135991.843285 write 0x030eb0 thru 0x03862f (0x7780 bytes) 000870 1783135991.852929 write 0x032029 thru 0x03e7a8 (0xc780 bytes) 000871 1783135991.855066 mapread 0x01cbb6 thru 0x02547a (0x88c5 bytes) 000872 1783135991.906300 write 0x000858 thru 0x001819 (0xfc2 bytes) 000876 1783135991.912004 mapread 0x032e3f thru 0x03badc (0x8c9e bytes) 000877 1783135991.914051 write 0x001a37 thru 0x008d5f (0x7329 bytes) 000878 1783135991.924305 mapread 0x01a03c thru 0x0263e0 (0xc3a5 bytes) 000879 1783135991.955993 read 0x026303 thru 0x030395 (0xa093 bytes) 000880 1783135991.973815 write 0x03b309 thru 0x03ffff (0x4cf7 bytes) 000881 1783135991.982547 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000883 1783135992.158835 read 0x019e01 thru 0x01c664 (0x2864 bytes) 000885 1783135992.171500 trunc from 0x040000 to 0x03a352 (0x5cad bytes) 000886 1783135992.192735 trunc from 0x03a353 to 0x025005 (0x1534d bytes) 000887 1783135992.212257 trunc from 0x025006 to 0x019ea1 (0xb164 bytes) 000889 1783135992.231343 read 0x00cacf thru 0x00f9e6 (0x2f18 bytes) 000890 1783135992.243776 read 0x005bd9 thru 0x008fbf (0x33e7 bytes) 000891 1783135992.244393 mapread 0x018694 thru 0x019ea1 (0x180e bytes) 000892 1783135992.245334 mapread 0x001d20 thru 0x006efd (0x51de bytes) 000895 1783135992.249210 trunc from 0x019ea2 to 0x023eda (0xa039 bytes) 000897 1783135992.280526 write 0x033f13 thru 0x03417f (0x26d bytes) 000900 1783135992.289420 trunc from 0x034180 to 0x01df2a (0x16255 bytes) 000902 1783135992.338181 mapread 0x00af62 thru 0x00da2c (0x2acb bytes) 000903 1783135992.343294 write 0x029983 thru 0x02ce3a (0x34b8 bytes) 000910 1783135992.359847 read 0x0250c8 thru 0x0294e1 (0x441a bytes) 000911 1783135992.399848 mapread 0x019843 thru 0x025d7f (0xc53d bytes) 000913 1783135992.404927 trunc from 0x02ce3b to 0x0009b9 (0x2c481 bytes) 000915 1783135992.448301 MIRROR_EXTEND to 2 mirrors 000918 1783135993.105520 trunc from 0x0009ba to 0x02aca6 (0x2a2ed bytes) 000919 1783135993.232303 read 0x011f7d thru 0x020fa0 (0xf024 bytes) 000922 1783135993.250012 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000923 1783135993.396187 mapread 0x0135bf thru 0x01ac05 (0x7647 bytes) 000929 1783135993.424569 trunc from 0x02aca7 to 0x034a2e (0x9d88 bytes) 000933 1783135993.446876 mapwrite 0x03b1ca thru 0x03fdd7 (0x4c0e bytes) 000934 1783135993.527526 mapwrite 0x0221ba thru 0x02a9f3 (0x883a bytes) 000938 1783135993.589907 read 0x024398 thru 0x02fad3 (0xb73c bytes) 000940 1783135993.605586 mapwrite 0x011fce thru 0x0192c2 (0x72f5 bytes) 000941 1783135993.696250 mapread 0x03cef9 thru 0x03fdd7 (0x2edf bytes) 000943 1783135993.702191 mapwrite 0x01e859 thru 0x0278f6 (0x909e bytes) 000944 1783135993.766466 mapread 0x031abf thru 0x03fdd7 (0xe319 bytes) 000945 1783135993.775123 mapread 0x0199d9 thru 0x019b4a (0x172 bytes) 000946 1783135993.776558 read 0x025a5f thru 0x02bed5 (0x6477 bytes) 000949 1783135993.779986 read 0x03e19b thru 0x03fdd7 (0x1c3d bytes) 000950 1783135993.780778 mapwrite 0x02cafc thru 0x035525 (0x8a2a bytes) 000951 1783135993.865477 MIRROR_EXTEND to 2 mirrors 000956 1783135994.556443 mapwrite 0x009afa thru 0x014eb0 (0xb3b7 bytes) 000957 1783135994.735036 mapread 0x0003bc thru 0x001800 (0x1445 bytes) 000958 1783135994.742747 mapwrite 0x03fb7a thru 0x03ffff (0x486 bytes) 000960 1783135994.817185 write 0x03402e thru 0x039986 (0x5959 bytes) 000963 1783135994.826419 write 0x004bed thru 0x007e85 (0x3299 bytes) 000964 1783135994.832011 MIRROR_RESYNC 2 mirrors 000965 1783135995.473602 trunc from 0x040000 to 0x01971e (0x268e1 bytes) 000967 1783135995.539819 read 0x00ad6d thru 0x01249b (0x772f bytes) 000968 1783135995.548006 mapwrite 0x02bbea thru 0x02e6ee (0x2b05 bytes) 000969 1783135995.685838 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000972 1783135995.848831 trunc from 0x02e6ef to 0x01f2a2 (0xf44c bytes) 000973 1783135995.872277 write 0x02f962 thru 0x03b157 (0xb7f6 bytes) 000974 1783135995.879546 MIRROR_EXTEND to 2 mirrors 000975 1783135996.595874 trunc from 0x03b158 to 0x021a9e (0x196b9 bytes) 000981 1783135996.675748 mapwrite 0x0310ba thru 0x031d71 (0xcb8 bytes) 000982 1783135996.757334 trunc from 0x031d72 to 0x02384e (0xe523 bytes) 000983 1783135996.779327 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000985 1783135996.945817 trunc from 0x02384f to 0x01cd31 (0x6b1d bytes) 000986 1783135996.970009 read 0x014f5b thru 0x016f36 (0x1fdc bytes) 000991 1783135996.981300 read 0x010619 thru 0x016bbd (0x65a5 bytes) 000992 1783135996.991679 read 0x00cf23 thru 0x0156c5 (0x87a3 bytes) 000995 1783135996.999072 MIRROR_EXTEND to 2 mirrors 000998 1783135997.587876 MIRROR_EXTEND to 3 mirrors 000999 1783135998.764316 mapwrite 0x032dc8 thru 0x0333f3 (0x62c bytes) 001000 1783135998.976065 mapwrite 0x008c01 thru 0x01687b (0xdc7b bytes) All operations completed A-OK! PASS 70a (72s) == sanity-flr test 71: check mirror extend parent fid ==== 23:33:25 (1783136005) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 23:33:29 (1783136009) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.215765 s, 9.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263398 s, 4.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 23:33:40 (1783136020) Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 289088 .. Extending file size to 2434528 .. failed Extending file size to 3153440 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 4666976 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6454592 .. Extending file size to 5121664 .. Extending file size to 7315424 .. Extending file size to 7155872 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7892960 .. Extending file size to 7641536 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8835424 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8261312 .. Extending file size to 9090720 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 56745 56746 56749 56750 56752 done 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) 192.168.202.130@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 23:35:03 (1783136103) Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1699136 .. Extending file size to 3052896 .. done Extending file size to 5725312 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6944480 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8054368 .. failed Extending file size to 7695872 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8153568 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8709248 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8835552 .. Extending file size to 8976064 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58367 58369 58370 58372 58375 done 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) 192.168.202.130@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (84s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 23:36:27 (1783136187) Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.103301 s, 20.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.393061 s, 5.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xe0ffc149 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 23:36:50 (1783136210) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.141422 s, 14.8 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 23:36:58 (1783136218) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c3:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 23:37:06 (1783136226) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.27693 s, 7.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 23:37:14 (1783136234) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.247152 s, 8.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 23:37:22 (1783136242) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.224004 s, 9.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } PASS 204c (37s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 23:37:59 (1783136279) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg230-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965825 s, 10.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e5:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e5:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } PASS 204d (58s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 23:38:57 (1783136337) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg230-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.220263 s, 9.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (66s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 23:40:03 (1783136403) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg230-server mds-ost sync done. Creating new pool test_204f oleg230-server: Pool lustre.test_204f created Adding targets to pool oleg230-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg230-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.25881 s, 8.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg230-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg230-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg230-server: Pool lustre.test_204f destroyed PASS 204f (83s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 23:41:26 (1783136486) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 23:41:35 (1783136495) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 23:41:43 (1783136503) oleg230-server: Pool lustre.test_206 created Adding targets to pool oleg230-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg230-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg230-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg230-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg230-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 23:42:10 (1783136530) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051081 s, 20.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654269 s, 16.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x45126d0a Mirror 3: 0x45126d0a PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 23:42:22 (1783136542) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 23:42:26 (1783136546) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 23:42:30 (1783136550) Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.13559 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2520000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xbf0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xbf0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xbf0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x11d0000 Read this (4 bytes): BBBB 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) PASS 209a (45s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 23:43:14 (1783136594) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.052869 s, 155 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 23:43:23 (1783136603) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068168 s, 15.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1e8 lmm_fid: [0x200000404:0x1e8:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1e8 lmm_fid: [0x200000404:0x1e8:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834408 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-03 23:43:24.000000000 -0400 Modify: 2026-07-03 23:43:24.000000000 -0400 Change: 2026-07-03 23:43:26.000000000 -0400 Birth: 2026-07-03 23:43:25.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 23:43:31 (1783136611) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 23:43:36 (1783136616) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.891493 s, 45.9 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0849338 s, 48.2 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 3 23:43 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ea:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 3 23:43 /mnt/lustre/f211.sanity-flr PASS 211 (11s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 23:43:47 (1783136627) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 23:43:55 (1783136635) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (13s) == sanity-flr test complete, duration 1742 sec =========== 23:44:08 (1783136648) === sanity-flr: start cleanup 23:44:10 (1783136650) === === sanity-flr: finish cleanup 23:44:20 (1783136660) ===