-----============= acceptance-small: sanity-flr ============----- Mon Jul 13 08:57:45 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg335-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 08:58:07 (1783947487) === oleg335-client.virtnet: executing check_config_client /mnt/lustre oleg335-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg335-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff91a8c7e23800.idle_timeout=debug osc.lustre-OST0001-osc-ffff91a8c7e23800.idle_timeout=debug disable quota as required oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 08:58:47 (1783947527) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7601] == sanity-flr test 0a: lfs mirror create with -N option == 08:58:50 (1783947530) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 08:59:09 (1783947549) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 08:59:13 (1783947553) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 08:59:17 (1783947557) PASS 0d (27s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 08:59:43 (1783947583) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 08:59:46 (1783947586) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 08:59:50 (1783947590) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff91a8c7e23800.stats=clear osc.lustre-OST0001-osc-ffff91a8c7e23800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 08:59:58 (1783947598) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:00:10 (1783947610) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:00:18 (1783947618) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 09:00:22 (1783947622) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:00:30 (1783947630) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 09:00:40 (1783947640) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:00:47 (1783947647) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:00:51 (1783947651) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:00:59 (1783947659) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (9s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:01:12 (1783947672) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:01:19 (1783947679) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 2.9736 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190166 s, 5.5 MB/s PASS 21 (13s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:01:32 (1783947692) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.388026 s, 40.5 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 09:01:40 (1783947700) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:01:51 (1783947711) Stopping /mnt/lustre-ost1 (opts:) on oleg335-server oleg335-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff91a8c7e23800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff91a8c7e23800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg335-server: oleg335-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg335-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg335-server oleg335-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff91a8c7e23800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff91a8c7e23800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg335-server: oleg335-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg335-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (71s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:03:02 (1783947782) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.456768 s, 20.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg335-server oleg335-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff91a8c7e23800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff91a8c7e23800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg335-server: oleg335-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg335-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (41s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:03:43 (1783947823) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg335-server oleg335-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff91a8c7e23800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff91a8c7e23800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg335-server: oleg335-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg335-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg335-server oleg335-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff91a8c7e23800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff91a8c7e23800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg335-server: oleg335-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg335-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (83s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:05:07 (1783947907) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.97489 s, 13.2 MB/s -rw-r--r-- 1 root root 52428800 Jul 13 09:05 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.936s user 0m0.001s sys 0m0.175s Stopping /mnt/lustre-ost1 (opts:) on oleg335-server oleg335-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff91a8c7e23800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff91a8c7e23800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.130s user 0m0.000s sys 0m0.183s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg335-server: oleg335-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg335-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg335-server oleg335-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff91a8c7e23800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff91a8c7e23800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.180s user 0m0.000s sys 0m0.201s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg335-server: oleg335-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg335-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg335-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff91a8c7e23800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff91a8c7e23800.ost_server_uuid in FULL state after 0 sec PASS 33b (86s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:06:32 (1783947992) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:06:36 (1783947996) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:06:39 (1783947999) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 09:06:43 (1783948003) PASS 35 (10s) == sanity-flr test 36a: write to mirrored files ========== 09:06:53 (1783948013) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.696479 s, 18.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:07:08 (1783948028) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749804 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679735 s, 15.4 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:07:17 (1783948037) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:07:26 (1783948046) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00217074 s, 472 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00194731 s, 526 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00563907 s, 182 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00134925 s, 759 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 09:07:37 (1783948057) create test files with size 15735538 .. e84dfe5d6d51ad2364767c706e5fb9f5 - 0738465c50a3b60e83312aaab4e21069 - 2fa934f8913dfd228bc2388ccf7b8344 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg335-client.virtnet /mnt/lustre (opts:) Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 09:08:04 (1783948084) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:08:20 (1783948100) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:08:27 (1783948107) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143382 s, 7.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280276 s, 37.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 09:08:36 (1783948116) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.161045 s, 13.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.394844 s, 10.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.244459 s, 8.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 09:08:53 (1783948133) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 09:08:57 (1783948137) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:09:00 (1783948140) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620416 s, 16.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 09:09:09 (1783948149) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:09:13 (1783948153) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:09:26 (1783948166) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.3211 s, 32.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg335-client.virtnet /mnt/lustre (opts:) Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:09:39 (1783948179) llite.lustre-ffff91a8c7a2c800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.337776 s, 31.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg335-client.virtnet /mnt/lustre (opts:) Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff91a8d046b800.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:09:54 (1783948194) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (42s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:10:36 (1783948236) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (43s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:11:20 (1783948280) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 09:11:23 (1783948283) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.298591 s, 10.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.186463 s, 16.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x1332d2e2 (nosync,stale) Mirror 2: 0x1332d2e2 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 33a57f53e1faa807698199d14b3d3017 - mirror 1 checksum: 2ff80d9ea3e1682eb4aed7b76c4405fc - mirror 2 checksum: 33a57f53e1faa807698199d14b3d3017 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-13T09:11:27-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-13T09:11:27-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 33a57f53e1faa807698199d14b3d3017 - mirror 1 checksum: 2ff80d9ea3e1682eb4aed7b76c4405fc - mirror 2 checksum: 2ff80d9ea3e1682eb4aed7b76c4405fc - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-13T09:11:27-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-13T09:11:27-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:11:37 (1783948297) Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server 09:11:45 (1783948305) shut down facet: mds1 facet_host: oleg335-server facet_failover_host: oleg335-server Failover mds1 to oleg335-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 09:12:18 (1783948338) targets are mounted 09:12:18 (1783948338) facet_failover done oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336651 s, 31.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xee:0x0] } PASS 50A (63s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:12:40 (1783948360) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:12:44 (1783948364) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:12:48 (1783948368) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00749962 s, 2.2 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:12:56 (1783948376) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:12:59 (1783948379) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:13:03 (1783948383) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:13:28 (1783948408) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (33s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:14:01 (1783948441) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 09:14:27 (1783948467) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 09:14:52 (1783948492) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 09:15:00 (1783948500) mirror create pid 53958 mirror split pid 53959 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 53958 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 53959 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (70s) == sanity-flr test 70a: flr mode fsx test ================ 09:16:10 (1783948570) Seed set to 8573 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1783948574.015893 write 0x01b9a3 thru 0x01df01 (0x255f bytes) truncating to largest ever: 0x13a14 000004 1783948574.024862 trunc from 0x01df02 to 0x013a13 (0xa4ee bytes) 000005 1783948574.057448 MIRROR_EXTEND to 2 mirrors 000009 1783948574.613752 read 0x00cae4 thru 0x013a13 (0x6f30 bytes) 000012 1783948574.684501 read 0x00f9ac thru 0x013a13 (0x4068 bytes) 000013 1783948574.706045 mapread 0x005c5f thru 0x00f0da (0x947c bytes) 000014 1783948574.766109 mapread 0x011138 thru 0x013a13 (0x28dc bytes) truncating to largest ever: 0x27a95 000021 1783948574.857700 trunc from 0x013a14 to 0x027a94 (0x14081 bytes) 000023 1783948574.946723 read 0x019916 thru 0x023c8a (0xa375 bytes) 000024 1783948574.978094 mapread 0x015a04 thru 0x01900a (0x3607 bytes) 000025 1783948574.979211 write 0x006ce2 thru 0x00ef84 (0x82a3 bytes) truncating to largest ever: 0x3591c 000026 1783948574.980503 trunc from 0x027a95 to 0x03591b (0xde87 bytes) 000028 1783948575.014878 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) 000030 1783948575.326983 read 0x023292 thru 0x028f41 (0x5cb0 bytes) 000034 1783948575.350236 read 0x00ae29 thru 0x00b279 (0x451 bytes) 000037 1783948575.388405 mapwrite 0x027949 thru 0x0307ba (0x8e72 bytes) 000038 1783948575.530423 mapread 0x02e8fd thru 0x03591b (0x701f bytes) 000039 1783948575.562816 MIRROR_EXTEND to 2 mirrors 000040 1783948576.310831 mapread 0x02fd91 thru 0x030ac8 (0xd38 bytes) 000041 1783948576.345825 read 0x0247be thru 0x02d89e (0x90e1 bytes) 000042 1783948576.374692 write 0x03b251 thru 0x03f901 (0x46b1 bytes) 000044 1783948576.459416 mapwrite 0x031570 thru 0x035942 (0x43d3 bytes) 000046 1783948576.590092 write 0x021567 thru 0x02ecd0 (0xd76a bytes) 000047 1783948576.604328 write 0x01b1c9 thru 0x028c51 (0xda89 bytes) 000048 1783948576.616577 write 0x007f4a thru 0x00a8c7 (0x297e bytes) 000050 1783948576.671641 trunc from 0x03f902 to 0x00c252 (0x336af bytes) 000051 1783948576.726324 mapread 0x0036a4 thru 0x00bb47 (0x84a4 bytes) truncating to largest ever: 0x3861e 000052 1783948576.741867 trunc from 0x00c253 to 0x03861d (0x2c3cb bytes) 000053 1783948576.765920 read 0x0370e1 thru 0x03861d (0x153d bytes) 000055 1783948576.788019 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) 000056 1783948577.091255 write 0x03c875 thru 0x03f648 (0x2dd4 bytes) 000059 1783948577.123713 MIRROR_EXTEND to 2 mirrors 000060 1783948578.676307 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) 000061 1783948579.069320 write 0x016ef7 thru 0x01a225 (0x332f bytes) 000062 1783948579.104956 mapread 0x03b887 thru 0x03ce1d (0x1597 bytes) 000063 1783948579.126032 MIRROR_EXTEND to 2 mirrors 000064 1783948580.064179 MIRROR_RESYNC 2 mirrors 000065 1783948580.460853 mapwrite 0x03e515 thru 0x03ffff (0x1aeb bytes) 000068 1783948580.616935 write 0x000fc0 thru 0x002fed (0x202e bytes) 000071 1783948580.634775 MIRROR_RESYNC 2 mirrors 000074 1783948581.808240 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) 000076 1783948582.072278 mapread 0x011dfd thru 0x01e36d (0xc571 bytes) 000079 1783948582.114196 trunc from 0x040000 to 0x02c24d (0x13db2 bytes) 000080 1783948582.160632 write 0x0243d5 thru 0x02dd6a (0x9996 bytes) 000083 1783948582.172954 trunc from 0x02dd6b to 0x019a02 (0x14368 bytes) 000084 1783948582.213009 write 0x02d524 thru 0x033e2a (0x6907 bytes) 000087 1783948582.218115 write 0x009357 thru 0x00ffd1 (0x6c7b bytes) 000088 1783948582.234344 mapwrite 0x0217ef thru 0x024c32 (0x3444 bytes) 000089 1783948582.353780 mapread 0x01a0de thru 0x01cee0 (0x2e03 bytes) 000091 1783948582.356905 write 0x024ff0 thru 0x02b388 (0x6399 bytes) 000092 1783948582.373296 write 0x0173ca thru 0x01f0f9 (0x7d30 bytes) 000094 1783948582.381422 trunc from 0x033e2b to 0x0146b8 (0x1f772 bytes) 000095 1783948582.408135 mapwrite 0x015f99 thru 0x016b1b (0xb83 bytes) 000099 1783948582.569904 trunc from 0x016b1c to 0x033992 (0x1ce77 bytes) 000100 1783948582.625885 mapwrite 0x025246 thru 0x02fb27 (0xa8e2 bytes) 000101 1783948582.756712 mapwrite 0x02c85f thru 0x03bc02 (0xf3a4 bytes) 000102 1783948582.936910 write 0x01e907 thru 0x02dd42 (0xf43c bytes) 000103 1783948582.940145 mapread 0x00ec05 thru 0x01251d (0x3919 bytes) 000104 1783948582.951657 trunc from 0x03bc03 to 0x027368 (0x1489a bytes) 000105 1783948582.982933 write 0x00cc11 thru 0x00e61a (0x1a0a bytes) 000106 1783948582.988994 read 0x0167f5 thru 0x01dfb0 (0x77bc bytes) truncating to largest ever: 0x3d60e 000108 1783948583.008582 trunc from 0x027369 to 0x03d60d (0x162a5 bytes) 000111 1783948583.038601 read 0x0121b8 thru 0x01a1e2 (0x802b bytes) 000113 1783948583.049108 write 0x029448 thru 0x0390bc (0xfc75 bytes) 000115 1783948583.091003 MIRROR_EXTEND to 2 mirrors 000116 1783948585.000878 mapread 0x012291 thru 0x0217a9 (0xf519 bytes) 000117 1783948585.104828 mapwrite 0x017ada thru 0x023e6b (0xc392 bytes) 000118 1783948585.521363 trunc from 0x03d60e to 0x013e4a (0x297c3 bytes) 000119 1783948585.574930 write 0x008650 thru 0x01453b (0xbeec bytes) 000120 1783948585.591496 read 0x00be99 thru 0x01453b (0x86a3 bytes) 000122 1783948585.601085 mapwrite 0x0082e1 thru 0x009b46 (0x1866 bytes) 000123 1783948585.754070 mapwrite 0x022e63 thru 0x030503 (0xd6a1 bytes) 000124 1783948585.987987 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x30504): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30504): Mirror 2: 0x9aa16c9c (stale) CRC-32 checksum value for chunk [0, 0x30504): Mirror 1: 0x74076812 Mirror 3: 0x74076812 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30504: skipped 000125 1783948587.585941 mapread 0x01a026 thru 0x020961 (0x693c bytes) 000127 1783948587.628279 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x30504): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30504): Mirror 2: 0x9aa16c9c (stale) CRC-32 checksum value for chunk [0, 0x30504): Mirror 1: 0x74076812 Mirror 3: 0x74076812 Mirror 4: 0x74076812 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30504: skipped 000128 1783948588.557456 mapwrite 0x022470 thru 0x02a772 (0x8303 bytes) 000129 1783948588.991773 trunc from 0x030504 to 0x02bbf2 (0x4911 bytes) 000131 1783948589.080590 mapwrite 0x02bfa4 thru 0x02fcaa (0x3d07 bytes) 000132 1783948589.300935 read 0x01659a thru 0x019010 (0x2a77 bytes) 000133 1783948589.336757 MIRROR_RESYNC 4 mirrors 000134 1783948591.153413 write 0x03a563 thru 0x03ffff (0x5a9d bytes) 000136 1783948591.333574 write 0x013afa thru 0x01d9a3 (0x9eaa bytes) 000137 1783948591.363743 mapread 0x0121dc thru 0x01c096 (0x9ebb bytes) 000138 1783948591.384870 trunc from 0x040000 to 0x00d72c (0x328d3 bytes) 000139 1783948591.441682 mapwrite 0x03a25f thru 0x03ffff (0x5da1 bytes) 000140 1783948591.602849 mapread 0x01eb5c thru 0x0276d8 (0x8b7d bytes) 000141 1783948591.672643 mapread 0x00056f thru 0x005f6b (0x59fd bytes) 000142 1783948591.698628 trunc from 0x040000 to 0x02f955 (0x106aa bytes) 000143 1783948591.771724 mapwrite 0x0303fd thru 0x0304bd (0xc1 bytes) 000144 1783948591.882063 mapread 0x01be6a thru 0x02ade3 (0xef7a bytes) 000146 1783948591.884569 mapread 0x01cf29 thru 0x026260 (0x9338 bytes) 000149 1783948591.890837 mapread 0x0039ad thru 0x0080e3 (0x4737 bytes) 000150 1783948591.896714 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x304be): Mirror 3: 0xa007f330 (stale) Mirror 4: 0xa007f330 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x304be): Mirror 3: 0xa007f330 (stale) Mirror 4: 0xa007f330 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x304be: skipped 000151 1783948592.503373 MIRROR_RESYNC 3 mirrors 000152 1783948593.801757 mapread 0x00cdca thru 0x01bedb (0xf112 bytes) 000155 1783948593.855076 MIRROR_RESYNC 3 mirrors 000156 1783948594.302727 read 0x02119f thru 0x0292b0 (0x8112 bytes) 000157 1783948594.317859 mapread 0x000cfa thru 0x010763 (0xfa6a bytes) 000159 1783948594.349426 write 0x031959 thru 0x03e1d9 (0xc881 bytes) 000160 1783948594.447909 write 0x01fd1f thru 0x02b62c (0xb90e bytes) 000161 1783948594.467009 read 0x015792 thru 0x0220e0 (0xc94f bytes) 000166 1783948594.488888 write 0x02707e thru 0x02bc19 (0x4b9c bytes) 000168 1783948594.499374 MIRROR_RESYNC 3 mirrors 000171 1783948595.269777 write 0x03cae0 thru 0x03ffff (0x3520 bytes) 000174 1783948595.379701 MIRROR_RESYNC 3 mirrors 000175 1783948596.857701 trunc from 0x040000 to 0x03d2a9 (0x2d56 bytes) 000179 1783948597.054240 mapwrite 0x00f3e5 thru 0x019c15 (0xa831 bytes) 000180 1783948597.215202 mapwrite 0x03f8af thru 0x03ffff (0x751 bytes) 000184 1783948597.362715 MIRROR_RESYNC 3 mirrors 000185 1783948598.146320 write 0x01da5d thru 0x02085c (0x2e00 bytes) 000186 1783948598.220602 mapread 0x01721c thru 0x01926c (0x2051 bytes) 000187 1783948598.248425 trunc from 0x040000 to 0x018e66 (0x27199 bytes) 000191 1783948598.292394 trunc from 0x018e67 to 0x02539f (0xc539 bytes) 000192 1783948598.314471 trunc from 0x0253a0 to 0x0083d8 (0x1cfc7 bytes) 000193 1783948598.331153 mapread 0x007489 thru 0x0083d8 (0xf50 bytes) 000197 1783948598.344320 read 0x000921 thru 0x0015f1 (0xcd1 bytes) 000199 1783948598.353962 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x83d9): Mirror 4: 0xa2915f00 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x83d9): Mirror 4: 0xa2915f00 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x83d9: skipped 000200 1783948598.628445 mapread 0x007596 thru 0x0083d8 (0xe43 bytes) 000203 1783948598.655687 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x83d9): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x83d9): Mirror 4: 0xa2915f00 (stale) CRC-32 checksum value for chunk [0, 0x83d9): Mirror 1: 0xa2915f00 Mirror 5: 0xa2915f00 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x83d9: skipped 000204 1783948599.237687 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x83d9): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x83d9): Mirror 4: 0xa2915f00 (stale) CRC-32 checksum value for chunk [0, 0x83d9): Mirror 1: 0xa2915f00 Mirror 5: 0xa2915f00 Mirror 6: 0xa2915f00 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x83d9: skipped 000205 1783948600.122592 mapread 0x002212 thru 0x0083d8 (0x61c7 bytes) 000207 1783948600.186487 MIRROR_RESYNC 4 mirrors 000208 1783948601.154749 mapread 0x004525 thru 0x0083d8 (0x3eb4 bytes) 000212 1783948601.216753 read 0x005e50 thru 0x0083d8 (0x2589 bytes) 000213 1783948601.217141 write 0x0172d2 thru 0x024c34 (0xd963 bytes) 000214 1783948601.403115 trunc from 0x024c35 to 0x0248a2 (0x392 bytes) 000217 1783948601.464287 trunc from 0x0248a3 to 0x01d2d6 (0x75cc bytes) 000218 1783948601.518622 MIRROR_RESYNC 4 mirrors 000220 1783948603.015356 mapwrite 0x02509d thru 0x0332a7 (0xe20b bytes) 000222 1783948603.261570 read 0x01ded3 thru 0x020545 (0x2673 bytes) 000224 1783948603.279320 read 0x02a3fa thru 0x02cf0e (0x2b15 bytes) 000226 1783948603.283076 mapread 0x02fab0 thru 0x0332a7 (0x37f8 bytes) 000228 1783948603.285943 trunc from 0x0332a8 to 0x018ab4 (0x1a7f3 bytes) 000229 1783948603.324304 mapwrite 0x01b27d thru 0x01ea0b (0x378f bytes) 000232 1783948603.461403 trunc from 0x01ea0c to 0x0379e9 (0x18fde bytes) 000233 1783948603.485208 read 0x025736 thru 0x031cb0 (0xc57b bytes) 000235 1783948603.498305 read 0x0150a6 thru 0x022063 (0xcfbe bytes) 000237 1783948603.499793 mapread 0x004c89 thru 0x00ae5f (0x61d7 bytes) 000238 1783948603.506891 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x379ea): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x379ea): Mirror 4: 0x76801e1d (stale) Mirror 5: 0x76801e1d (stale) Mirror 6: 0x76801e1d (stale) CRC-32 checksum value for chunk [0, 0x379ea): Mirror 1: 0x25d602c2 Mirror 7: 0x25d602c2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x379ea: skipped 000239 1783948604.478405 trunc from 0x0379ea to 0x02a13e (0xd8ab bytes) 000240 1783948604.612675 mapread 0x028232 thru 0x02a13e (0x1f0d bytes) 000243 1783948604.664805 mapread 0x003bc5 thru 0x011e77 (0xe2b3 bytes) 000244 1783948604.752416 write 0x01b85c thru 0x023c9d (0x8442 bytes) 000245 1783948604.767453 read 0x01ff0a thru 0x02a13e (0xa235 bytes) 000246 1783948604.768345 write 0x02a9c3 thru 0x036311 (0xb94f bytes) 000248 1783948604.777944 read 0x034871 thru 0x036311 (0x1aa1 bytes) 000249 1783948604.787268 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36312): Mirror 5: 0x51f1b649 (stale) Mirror 6: 0x51f1b649 (stale) Mirror 7: 0xb208a3d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36312): Mirror 5: 0x51f1b649 (stale) Mirror 6: 0x51f1b649 (stale) Mirror 7: 0xb208a3d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36312: skipped 000251 1783948605.216628 mapread 0x00c5d8 thru 0x0113cf (0x4df8 bytes) 000253 1783948605.238706 mapread 0x011708 thru 0x0137dd (0x20d6 bytes) 000254 1783948605.239999 read 0x0351a3 thru 0x036311 (0x116f bytes) 000256 1783948605.248299 mapread 0x026221 thru 0x02de3f (0x7c1f bytes) 000257 1783948605.262382 write 0x03a003 thru 0x03ffff (0x5ffd bytes) 000258 1783948605.263812 write 0x00466a thru 0x009fd9 (0x5970 bytes) 000259 1783948605.280397 write 0x010e14 thru 0x016900 (0x5aed bytes) 000260 1783948605.283726 mapread 0x011322 thru 0x01bf6d (0xac4c bytes) 000262 1783948605.300146 trunc from 0x040000 to 0x02a177 (0x15e88 bytes) 000263 1783948605.315425 read 0x010b2e thru 0x01b5e5 (0xaab8 bytes) 000264 1783948605.316675 write 0x0017d3 thru 0x004cab (0x34d9 bytes) 000265 1783948605.324410 trunc from 0x02a178 to 0x000153 (0x2a024 bytes) 000266 1783948605.355671 write 0x008b8b thru 0x00f450 (0x68c6 bytes) 000269 1783948605.361382 mapread 0x004ea5 thru 0x00f450 (0xa5ac bytes) 000270 1783948605.390098 write 0x03faaf thru 0x03ffff (0x551 bytes) 000274 1783948605.394415 mapread 0x001eec thru 0x00c1b1 (0xa2c6 bytes) 000275 1783948605.417967 read 0x029ce3 thru 0x0314af (0x77cd bytes) 000277 1783948605.430352 trunc from 0x040000 to 0x02b9b0 (0x1464f bytes) 000278 1783948605.446098 mapread 0x00f8b1 thru 0x0161b0 (0x6900 bytes) 000279 1783948605.447813 mapread 0x026a04 thru 0x02b9b0 (0x4fad bytes) 000280 1783948605.455194 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2b9b1): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b9b1): Mirror 5: 0x4891ce86 (stale) Mirror 6: 0x4891ce86 (stale) Mirror 7: 0x989b2519 (stale) CRC-32 checksum value for chunk [0, 0x2b9b1): Mirror 1: 0xa948cbbc Mirror 8: 0xa948cbbc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b9b1: skipped 000282 1783948606.483929 write 0x02d883 thru 0x02e990 (0x110e bytes) 000284 1783948606.592881 trunc from 0x02e991 to 0x020e9b (0xdaf5 bytes) 000286 1783948606.659929 mapwrite 0x01fc84 thru 0x02582b (0x5ba8 bytes) 000287 1783948606.818963 read 0x012d33 thru 0x0137af (0xa7d bytes) 000290 1783948606.830398 write 0x03f83c thru 0x03ff45 (0x70a bytes) 000291 1783948606.838814 MIRROR_RESYNC 5 mirrors 000292 1783948608.104736 write 0x028acf thru 0x037bf2 (0xf124 bytes) 000293 1783948608.266577 read 0x01421e thru 0x01f7bd (0xb5a0 bytes) 000294 1783948608.276266 write 0x02b834 thru 0x032aa0 (0x726d bytes) 000295 1783948608.278352 trunc from 0x03ff46 to 0x03655c (0x99e9 bytes) 000296 1783948608.333641 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3655d): Mirror 5: 0x257f763 (stale) Mirror 6: 0x257f763 (stale) Mirror 7: 0x257f763 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3655d): Mirror 5: 0x257f763 (stale) Mirror 6: 0x257f763 (stale) Mirror 7: 0x257f763 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3655d: skipped 000299 1783948608.885796 write 0x01ae82 thru 0x02224b (0x73ca bytes) 000300 1783948608.912020 trunc from 0x03655d to 0x011c6c (0x248f0 bytes) 000302 1783948608.942857 mapread 0x00df3c thru 0x011c6c (0x3d31 bytes) 000307 1783948608.983298 read 0x011784 thru 0x011c6c (0x4e9 bytes) 000309 1783948608.985509 read 0x00331b thru 0x00dd0a (0xa9f0 bytes) 000314 1783948609.013356 mapwrite 0x019222 thru 0x01d0ab (0x3e8a bytes) 000315 1783948609.106724 mapread 0x005f7c thru 0x00d91e (0x79a3 bytes) 000316 1783948609.107898 trunc from 0x01d0ac to 0x02949c (0xc3f1 bytes) 000317 1783948609.130806 read 0x00d942 thru 0x013acc (0x618b bytes) 000318 1783948609.132198 mapwrite 0x013db3 thru 0x01bb86 (0x7dd4 bytes) 000320 1783948609.181675 mapread 0x00f477 thru 0x0142fe (0x4e88 bytes) 000323 1783948609.190211 MIRROR_RESYNC 4 mirrors 000324 1783948609.919634 mapread 0x011f2f thru 0x0205c4 (0xe696 bytes) 000325 1783948609.947618 read 0x016295 thru 0x025082 (0xedee bytes) 000328 1783948609.956566 read 0x023fc5 thru 0x02949c (0x54d8 bytes) 000329 1783948609.968642 mapwrite 0x03ccda thru 0x03ffff (0x3326 bytes) 000331 1783948610.140805 mapread 0x00b7d0 thru 0x00bd56 (0x587 bytes) 000338 1783948610.152341 write 0x01a259 thru 0x021392 (0x713a bytes) 000341 1783948610.165340 trunc from 0x040000 to 0x003219 (0x3cde6 bytes) 000342 1783948610.206231 read 0x00034d thru 0x000670 (0x324 bytes) 000344 1783948610.226662 mapread 0x001cae thru 0x003219 (0x156c bytes) 000346 1783948610.232471 mapwrite 0x035fb3 thru 0x03ffff (0xa04d bytes) 000349 1783948610.390431 trunc from 0x040000 to 0x012ee1 (0x2d11e bytes) 000350 1783948610.429652 read 0x00f779 thru 0x012ee1 (0x3769 bytes) 000353 1783948610.440959 mapwrite 0x03564f thru 0x0371be (0x1b70 bytes) 000355 1783948610.603341 write 0x01bc5a thru 0x01eacf (0x2e76 bytes) 000357 1783948610.609725 read 0x030111 thru 0x0371be (0x70ae bytes) 000358 1783948610.610406 mapwrite 0x031eda thru 0x03a254 (0x837b bytes) 000360 1783948610.760810 write 0x012682 thru 0x012e7f (0x7fe bytes) 000361 1783948610.765248 mapwrite 0x02df1b thru 0x034dbf (0x6ea5 bytes) 000362 1783948610.846114 read 0x015b4a thru 0x02103e (0xb4f5 bytes) 000364 1783948610.853772 write 0x0043cf thru 0x00df3d (0x9b6f bytes) 000367 1783948610.867927 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a255): Mirror 5: 0xa293e0aa (stale) Mirror 6: 0xa293e0aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a255): Mirror 5: 0xa293e0aa (stale) Mirror 6: 0xa293e0aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a255: skipped 000368 1783948611.344660 trunc from 0x03a255 to 0x02aeef (0xf365 bytes) 000369 1783948611.366211 write 0x02c8d6 thru 0x0344f2 (0x7c1d bytes) 000371 1783948611.372581 mapread 0x02f025 thru 0x0344f2 (0x54ce bytes) 000372 1783948611.377035 read 0x010576 thru 0x010d5d (0x7e8 bytes) 000374 1783948611.387125 read 0x016f2d thru 0x01f7be (0x8892 bytes) 000375 1783948611.424276 mapwrite 0x03136f thru 0x035cec (0x497e bytes) 000376 1783948611.612514 trunc from 0x035ced to 0x02fddd (0x5f0f bytes) 000378 1783948611.676752 MIRROR_RESYNC 3 mirrors 000379 1783948612.801939 write 0x02e5aa thru 0x039df2 (0xb849 bytes) 000380 1783948612.923280 MIRROR_RESYNC 3 mirrors 000381 1783948613.765235 read 0x001970 thru 0x009d52 (0x83e3 bytes) 000383 1783948613.774909 mapread 0x013b03 thru 0x013de0 (0x2de bytes) 000385 1783948613.782375 write 0x0321bc thru 0x03ab23 (0x8968 bytes) 000386 1783948613.877849 mapread 0x029275 thru 0x02f61b (0x63a7 bytes) 000387 1783948613.902923 write 0x014cf8 thru 0x016ced (0x1ff6 bytes) 000388 1783948613.915485 mapwrite 0x036663 thru 0x03ffff (0x999d bytes) 000389 1783948614.035004 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xadd091ec (stale) Mirror 6: 0xadd091ec (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2c465d44 Mirror 7: 0x2c465d44 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000391 1783948615.078036 mapread 0x015f1b thru 0x0222d4 (0xc3ba bytes) 000392 1783948615.136772 write 0x021f7d thru 0x028309 (0x638d bytes) 000394 1783948615.268169 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xadd091ec (stale) Mirror 7: 0x2c465d44 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xadd091ec (stale) Mirror 7: 0x2c465d44 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000395 1783948615.780911 mapread 0x039333 thru 0x03c951 (0x361f bytes) 000396 1783948615.847845 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xadd091ec (stale) Mirror 7: 0x2c465d44 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9640544a Mirror 8: 0x9640544a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000401 1783948617.197270 write 0x03ac87 thru 0x03ffff (0x5379 bytes) 000403 1783948617.308586 read 0x030341 thru 0x036eba (0x6b7a bytes) 000404 1783948617.331785 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xadd091ec (stale) Mirror 8: 0x9640544a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xadd091ec (stale) Mirror 8: 0x9640544a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000407 1783948617.929172 mapread 0x034934 thru 0x037172 (0x283f bytes) 000408 1783948617.963776 write 0x031fa1 thru 0x0375d4 (0x5634 bytes) 000409 1783948617.966327 mapwrite 0x003248 thru 0x007138 (0x3ef1 bytes) 000410 1783948618.057238 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x9640544a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x9640544a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000411 1783948618.508115 mapwrite 0x0322e4 thru 0x033b2e (0x184b bytes) 000414 1783948618.577419 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x9640544a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc1ac7630 Mirror 9: 0xc1ac7630 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000416 1783948620.054753 trunc from 0x040000 to 0x02bcf3 (0x1430c bytes) 000418 1783948620.257291 mapread 0x01069f thru 0x01f01e (0xe980 bytes) 000419 1783948620.308985 mapwrite 0x025f5d thru 0x028326 (0x23ca bytes) 000421 1783948620.404476 write 0x02ae0d thru 0x02dd4c (0x2f40 bytes) 000422 1783948620.406416 trunc from 0x02dd4d to 0x0139ed (0x1a35f bytes) 000426 1783948620.444884 write 0x01bbbf thru 0x02889d (0xccdf bytes) 000427 1783948620.455059 write 0x033e25 thru 0x037ed0 (0x40ac bytes) 000428 1783948620.472269 MIRROR_RESYNC 3 mirrors 000429 1783948621.607970 read 0x011000 thru 0x01191c (0x91d bytes) 000432 1783948621.622376 read 0x02b8c1 thru 0x03077e (0x4ebe bytes) 000433 1783948621.648841 read 0x02a495 thru 0x036791 (0xc2fd bytes) 000435 1783948621.659912 read 0x011cd4 thru 0x0215bd (0xf8ea bytes) 000437 1783948621.661090 write 0x03e1d1 thru 0x03ffff (0x1e2f bytes) 000438 1783948621.784161 trunc from 0x040000 to 0x018436 (0x27bc9 bytes) 000440 1783948621.839499 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x18437): Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18437): Mirror 8: 0xb7efd6b0 (stale) Mirror 9: 0xb7efd6b0 (stale) CRC-32 checksum value for chunk [0, 0x18437): Mirror 1: 0xb7efd6b0 Mirror 10: 0xb7efd6b0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18437: skipped 000441 1783948623.211419 MIRROR_RESYNC 4 mirrors 000443 1783948624.158484 mapwrite 0x0120b3 thru 0x01d714 (0xb662 bytes) 000444 1783948624.367754 mapwrite 0x0295f2 thru 0x03566b (0xc07a bytes) 000446 1783948624.455922 MIRROR_RESYNC 4 mirrors 000448 1783948625.564963 mapwrite 0x01a95e thru 0x02561b (0xacbe bytes) 000449 1783948625.889672 mapwrite 0x02a96e thru 0x03353a (0x8bcd bytes) 000451 1783948626.069396 mapread 0x007733 thru 0x007825 (0xf3 bytes) 000453 1783948626.111979 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3566c): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3566c): Mirror 8: 0xf71ddb22 (stale) Mirror 9: 0xf71ddb22 (stale) Mirror 10: 0xf71ddb22 (stale) CRC-32 checksum value for chunk [0, 0x3566c): Mirror 1: 0xf8f541c0 Mirror 11: 0xf8f541c0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3566c: skipped 000456 1783948627.772038 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3566c): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3566c): Mirror 8: 0xf71ddb22 (stale) Mirror 9: 0xf71ddb22 (stale) Mirror 10: 0xf71ddb22 (stale) CRC-32 checksum value for chunk [0, 0x3566c): Mirror 1: 0xf8f541c0 Mirror 11: 0xf8f541c0 Mirror 12: 0xf8f541c0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3566c: skipped 000457 1783948629.443076 mapread 0x030719 thru 0x03566b (0x4f53 bytes) 000463 1783948629.523021 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3566c): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3566c): Mirror 8: 0xf71ddb22 (stale) Mirror 9: 0xf71ddb22 (stale) Mirror 10: 0xf71ddb22 (stale) CRC-32 checksum value for chunk [0, 0x3566c): Mirror 1: 0xf8f541c0 Mirror 11: 0xf8f541c0 Mirror 12: 0xf8f541c0 Mirror 13: 0xf8f541c0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3566c: skipped 000464 1783948631.004806 trunc from 0x03566c to 0x004872 (0x30df9 bytes) 000465 1783948631.102626 mapwrite 0x0070e3 thru 0x008e92 (0x1db0 bytes) 000466 1783948631.230520 trunc from 0x008e93 to 0x020219 (0x17387 bytes) 000468 1783948631.255489 trunc from 0x02021a to 0x010827 (0xf9f2 bytes) 000470 1783948631.303046 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x10828): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10828): Mirror 8: 0x1b080b48 (stale) Mirror 9: 0x1b080b48 (stale) Mirror 10: 0x1b080b48 (stale) Mirror 11: 0x1b080b48 (stale) Mirror 12: 0x1b080b48 (stale) Mirror 13: 0x1b080b48 (stale) CRC-32 checksum value for chunk [0, 0x10828): Mirror 1: 0xfe07d83 Mirror 14: 0xfe07d83 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10828: skipped 000471 1783948632.695176 mapread 0x0042ea thru 0x010827 (0xc53e bytes) 000472 1783948632.765215 trunc from 0x010828 to 0x0195de (0x8db7 bytes) 000473 1783948632.928077 mapread 0x002ef0 thru 0x00f93b (0xca4c bytes) 000474 1783948632.977265 read 0x005be8 thru 0x00af60 (0x5379 bytes) 000475 1783948632.981292 write 0x018fee thru 0x01ad18 (0x1d2b bytes) 000480 1783948633.011750 trunc from 0x01ad19 to 0x0165aa (0x476e bytes) 000482 1783948633.095355 write 0x03fe4a thru 0x03ffff (0x1b6 bytes) 000483 1783948633.096642 trunc from 0x040000 to 0x020989 (0x1f676 bytes) 000484 1783948633.149196 mapwrite 0x01f938 thru 0x0247ed (0x4eb6 bytes) 000486 1783948633.279452 mapread 0x01af79 thru 0x022ed8 (0x7f60 bytes) 000488 1783948633.287764 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x247ee): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x247ee): Mirror 8: 0xbcfb2dab (stale) Mirror 9: 0xbcfb2dab (stale) Mirror 10: 0xbcfb2dab (stale) Mirror 11: 0xc6b3b003 (stale) Mirror 12: 0xc6b3b003 (stale) Mirror 13: 0xc6b3b003 (stale) Mirror 14: 0x79c5aa0c (stale) CRC-32 checksum value for chunk [0, 0x247ee): Mirror 1: 0x5eab7a0b Mirror 15: 0x5eab7a0b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x247ee: skipped 000489 1783948634.548389 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x247ee): Mirror 8: 0xbcfb2dab (stale) Mirror 9: 0xbcfb2dab (stale) Mirror 10: 0xbcfb2dab (stale) Mirror 11: 0xc6b3b003 (stale) Mirror 12: 0xc6b3b003 (stale) Mirror 14: 0x79c5aa0c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x247ee): Mirror 8: 0xbcfb2dab (stale) Mirror 9: 0xbcfb2dab (stale) Mirror 10: 0xbcfb2dab (stale) Mirror 11: 0xc6b3b003 (stale) Mirror 12: 0xc6b3b003 (stale) Mirror 14: 0x79c5aa0c (stale) CRC-32 checksum value for chunk [0, 0x247ee): Mirror 1: 0x5eab7a0b Mirror 15: 0x5eab7a0b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x247ee: skipped 000490 1783948635.424194 trunc from 0x0247ee to 0x01641e (0xe3cf bytes) 000491 1783948635.520465 MIRROR_RESYNC 8 mirrors 000492 1783948637.039482 MIRROR_RESYNC 8 mirrors 000494 1783948637.620404 mapread 0x014150 thru 0x01641e (0x22cf bytes) 000495 1783948637.655884 MIRROR_EXTEND to 9 mirrors 000496 1783948638.899068 MIRROR_RESYNC 9 mirrors 000497 1783948639.620707 mapwrite 0x007206 thru 0x007467 (0x262 bytes) 000501 1783948639.862026 read 0x0072c9 thru 0x00d7e5 (0x651d bytes) 000503 1783948639.882677 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1641f): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1641f): Mirror 8: 0x87331957 (stale) Mirror 9: 0x87331957 (stale) Mirror 10: 0x87331957 (stale) Mirror 11: 0x87331957 (stale) Mirror 12: 0x87331957 (stale) Mirror 14: 0x87331957 (stale) Mirror 15: 0x87331957 (stale) Mirror 16: 0x87331957 (stale) CRC-32 checksum value for chunk [0, 0x1641f): Mirror 1: 0xed5a2a0c Mirror 17: 0xed5a2a0c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1641f: skipped 000506 1783948641.939443 mapread 0x001787 thru 0x008a75 (0x72ef bytes) 000507 1783948642.035581 write 0x017382 thru 0x021f63 (0xabe2 bytes) 000511 1783948642.270553 write 0x02f030 thru 0x0380cf (0x90a0 bytes) 000513 1783948642.280013 mapwrite 0x00c7ac thru 0x01621c (0x9a71 bytes) 000515 1783948642.502496 trunc from 0x0380d0 to 0x0318a6 (0x6829 bytes) 000516 1783948642.526496 write 0x0219a8 thru 0x02c5a1 (0xabfa bytes) 000517 1783948642.533680 read 0x00a8eb thru 0x00c56b (0x1c81 bytes) 000518 1783948642.534963 trunc from 0x0318a7 to 0x02d900 (0x3fa6 bytes) 000521 1783948642.563951 MIRROR_RESYNC 10 mirrors 000522 1783948644.764574 mapread 0x012cf3 thru 0x0181f2 (0x5500 bytes) 000523 1783948644.786770 mapread 0x004c07 thru 0x007383 (0x277d bytes) 000529 1783948644.805430 mapread 0x00f172 thru 0x012566 (0x33f5 bytes) 000530 1783948644.806263 trunc from 0x02d901 to 0x0217fd (0xc103 bytes) 000532 1783948644.956774 mapwrite 0x01bfee thru 0x0298ed (0xd900 bytes) 000533 1783948645.088705 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x298ee): Mirror 8: 0xdc98fff9 (stale) Mirror 9: 0xdc98fff9 (stale) Mirror 10: 0xdc98fff9 (stale) Mirror 11: 0xdc98fff9 (stale) Mirror 12: 0xdc98fff9 (stale) Mirror 14: 0xdc98fff9 (stale) Mirror 15: 0xdc98fff9 (stale) Mirror 16: 0xdc98fff9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x298ee): Mirror 8: 0xdc98fff9 (stale) Mirror 9: 0xdc98fff9 (stale) Mirror 10: 0xdc98fff9 (stale) Mirror 11: 0xdc98fff9 (stale) Mirror 12: 0xdc98fff9 (stale) Mirror 14: 0xdc98fff9 (stale) Mirror 15: 0xdc98fff9 (stale) Mirror 16: 0xdc98fff9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x298ee: skipped 000535 1783948645.648299 write 0x029b68 thru 0x031d83 (0x821c bytes) 000536 1783948645.654718 trunc from 0x031d84 to 0x017f27 (0x19e5c bytes) 000538 1783948645.685440 mapwrite 0x0054b7 thru 0x00b180 (0x5cca bytes) 000539 1783948645.775642 write 0x01770a thru 0x026b8d (0xf484 bytes) 000540 1783948645.782743 mapwrite 0x039933 thru 0x03ffff (0x66cd bytes) 000545 1783948645.869694 MIRROR_RESYNC 9 mirrors 000546 1783948648.183639 trunc from 0x040000 to 0x029c06 (0x163f9 bytes) 000548 1783948648.264222 read 0x018ff8 thru 0x01a40e (0x1417 bytes) 000549 1783948648.277878 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29c07): Mirror 8: 0x77cd8c1d (stale) Mirror 9: 0x77cd8c1d (stale) Mirror 10: 0x77cd8c1d (stale) Mirror 11: 0x77cd8c1d (stale) Mirror 12: 0x77cd8c1d (stale) Mirror 14: 0x77cd8c1d (stale) Mirror 15: 0x77cd8c1d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29c07): Mirror 8: 0x77cd8c1d (stale) Mirror 9: 0x77cd8c1d (stale) Mirror 10: 0x77cd8c1d (stale) Mirror 11: 0x77cd8c1d (stale) Mirror 12: 0x77cd8c1d (stale) Mirror 14: 0x77cd8c1d (stale) Mirror 15: 0x77cd8c1d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29c07: skipped 000550 1783948649.127498 read 0x013384 thru 0x016848 (0x34c5 bytes) 000551 1783948649.136945 trunc from 0x029c07 to 0x0022ec (0x2791a bytes) 000552 1783948649.159181 mapwrite 0x00cd5e thru 0x010211 (0x34b4 bytes) 000553 1783948649.274015 trunc from 0x010212 to 0x00d6c6 (0x2b4b bytes) 000554 1783948649.302067 write 0x002740 thru 0x009b3d (0x73fe bytes) 000555 1783948649.312343 mapread 0x0055ee thru 0x00d6c6 (0x80d9 bytes) 000556 1783948649.315325 read 0x003edd thru 0x0050ce (0x11f2 bytes) 000558 1783948649.323182 MIRROR_RESYNC 8 mirrors 000559 1783948650.987387 write 0x00672c thru 0x00b72f (0x5004 bytes) 000560 1783948651.237379 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd6c7): Mirror 8: 0xfabdb6f (stale) Mirror 9: 0xfabdb6f (stale) Mirror 10: 0xfabdb6f (stale) Mirror 12: 0xfabdb6f (stale) Mirror 14: 0xfabdb6f (stale) Mirror 15: 0xfabdb6f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd6c7): Mirror 8: 0xfabdb6f (stale) Mirror 9: 0xfabdb6f (stale) Mirror 10: 0xfabdb6f (stale) Mirror 12: 0xfabdb6f (stale) Mirror 14: 0xfabdb6f (stale) Mirror 15: 0xfabdb6f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd6c7: skipped 000563 1783948652.181600 mapread 0x0038de thru 0x00404d (0x770 bytes) 000564 1783948652.204351 trunc from 0x00d6c7 to 0x0143a0 (0x6cda bytes) 000565 1783948652.241691 mapread 0x001625 thru 0x008072 (0x6a4e bytes) 000566 1783948652.265906 trunc from 0x0143a1 to 0x0071f2 (0xd1ae bytes) 000567 1783948652.345479 read 0x00479a thru 0x0071f2 (0x2a59 bytes) 000568 1783948652.352366 trunc from 0x0071f3 to 0x012deb (0xbbf9 bytes) 000571 1783948652.390207 mapwrite 0x01472e thru 0x022e9b (0xe76e bytes) 000572 1783948652.558571 mapwrite 0x0360f8 thru 0x03ffff (0x9f08 bytes) 000576 1783948652.724455 mapwrite 0x03392d thru 0x03e452 (0xab26 bytes) 000577 1783948652.791421 read 0x020383 thru 0x025e01 (0x5a7f bytes) 000578 1783948652.820335 mapread 0x009710 thru 0x015cc7 (0xc5b8 bytes) 000579 1783948652.822777 trunc from 0x040000 to 0x003809 (0x3c7f6 bytes) 000581 1783948652.871708 write 0x00c051 thru 0x00d6a3 (0x1653 bytes) 000582 1783948652.886318 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xd6a4): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd6a4): Mirror 8: 0xaee90026 (stale) Mirror 9: 0xaee90026 (stale) Mirror 10: 0xaee90026 (stale) Mirror 12: 0xaee90026 (stale) Mirror 14: 0xaee90026 (stale) Mirror 15: 0xaee90026 (stale) CRC-32 checksum value for chunk [0, 0xd6a4): Mirror 1: 0x3ef4dc19 Mirror 16: 0x3ef4dc19 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd6a4: skipped 000583 1783948654.581059 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd6a4): Mirror 9: 0xaee90026 (stale) Mirror 10: 0xaee90026 (stale) Mirror 12: 0xaee90026 (stale) Mirror 14: 0xaee90026 (stale) Mirror 15: 0xaee90026 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd6a4): Mirror 9: 0xaee90026 (stale) Mirror 10: 0xaee90026 (stale) Mirror 12: 0xaee90026 (stale) Mirror 14: 0xaee90026 (stale) Mirror 15: 0xaee90026 (stale) CRC-32 checksum value for chunk [0, 0xd6a4): Mirror 1: 0x3ef4dc19 Mirror 16: 0x3ef4dc19 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd6a4: skipped 000584 1783948655.897206 mapread 0x0071b3 thru 0x007b4b (0x999 bytes) 000585 1783948655.968060 mapwrite 0x035453 thru 0x03847a (0x3028 bytes) 000588 1783948656.230994 read 0x01b2ff thru 0x01b438 (0x13a bytes) 000589 1783948656.242038 read 0x0141a2 thru 0x017e53 (0x3cb2 bytes) 000591 1783948656.311355 mapwrite 0x016fc3 thru 0x01eb41 (0x7b7f bytes) 000594 1783948656.509326 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3847b): Mirror 9: 0xfd21d534 (stale) Mirror 10: 0xfd21d534 (stale) Mirror 12: 0xfd21d534 (stale) Mirror 14: 0xfd21d534 (stale) Mirror 16: 0x82e5985 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3847b): Mirror 9: 0xfd21d534 (stale) Mirror 10: 0xfd21d534 (stale) Mirror 12: 0xfd21d534 (stale) Mirror 14: 0xfd21d534 (stale) Mirror 16: 0x82e5985 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3847b: skipped 000596 1783948657.227325 mapwrite 0x01588a thru 0x023941 (0xe0b8 bytes) 000598 1783948657.504258 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3847b): Mirror 9: 0xfd21d534 (stale) Mirror 10: 0xfd21d534 (stale) Mirror 12: 0xfd21d534 (stale) Mirror 14: 0xfd21d534 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3847b): Mirror 9: 0xfd21d534 (stale) Mirror 10: 0xfd21d534 (stale) Mirror 12: 0xfd21d534 (stale) Mirror 14: 0xfd21d534 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3847b: skipped 000600 1783948658.227106 trunc from 0x03847b to 0x03bf5d (0x3ae3 bytes) 000601 1783948658.261927 mapwrite 0x0336ad thru 0x03eb67 (0xb4bb bytes) 000602 1783948658.401705 trunc from 0x03eb68 to 0x016f9b (0x27bcc bytes) 000603 1783948658.422765 mapread 0x0149d9 thru 0x016f9b (0x25c3 bytes) 000604 1783948658.438372 read 0x01508a thru 0x016f9b (0x1f12 bytes) 000606 1783948658.442142 trunc from 0x016f9c to 0x03524f (0x1e2b4 bytes) 000608 1783948658.466943 trunc from 0x035250 to 0x02df55 (0x72fa bytes) 000609 1783948658.507024 mapwrite 0x01c122 thru 0x0286d8 (0xc5b7 bytes) 000610 1783948658.647281 read 0x028027 thru 0x02df55 (0x5f2f bytes) 000611 1783948658.656449 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2df56): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2df56): Mirror 9: 0x1447ae43 (stale) Mirror 10: 0x1447ae43 (stale) Mirror 12: 0x1447ae43 (stale) Mirror 14: 0x1447ae43 (stale) CRC-32 checksum value for chunk [0, 0x2df56): Mirror 1: 0xa72cf279 Mirror 15: 0xa72cf279 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2df56: skipped 000612 1783948660.191453 read 0x00d19d thru 0x01a419 (0xd27d bytes) 000613 1783948660.226044 trunc from 0x02df56 to 0x00f771 (0x1e7e4 bytes) 000616 1783948660.394090 write 0x009e28 thru 0x00e453 (0x462c bytes) 000619 1783948660.437692 write 0x011a68 thru 0x0214c0 (0xfa59 bytes) 000620 1783948660.453447 mapread 0x0129bf thru 0x01acc1 (0x8303 bytes) 000622 1783948660.461050 mapread 0x020f28 thru 0x0214c0 (0x599 bytes) 000623 1783948660.467049 trunc from 0x0214c1 to 0x03638a (0x14eca bytes) 000624 1783948660.500615 read 0x026e87 thru 0x02dd9c (0x6f16 bytes) 000626 1783948660.527337 write 0x02d50f thru 0x033f92 (0x6a84 bytes) 000627 1783948660.529237 trunc from 0x03638b to 0x021432 (0x14f58 bytes) 000630 1783948660.576835 MIRROR_RESYNC 6 mirrors truncating to largest ever: 0x3f9b8 000631 1783948662.630232 trunc from 0x021433 to 0x03f9b7 (0x1e585 bytes) 000632 1783948662.786962 mapwrite 0x0316f3 thru 0x03b451 (0x9d5f bytes) 000633 1783948663.020207 write 0x009dc9 thru 0x010603 (0x683b bytes) 000634 1783948663.051336 mapwrite 0x01d733 thru 0x02392b (0x61f9 bytes) 000635 1783948663.203018 trunc from 0x03f9b8 to 0x001190 (0x3e827 bytes) 000638 1783948663.233139 trunc from 0x001191 to 0x02a905 (0x29775 bytes) 000639 1783948663.274834 trunc from 0x02a906 to 0x011f1f (0x189e6 bytes) 000640 1783948663.297032 mapread 0x006f0a thru 0x00bd4d (0x4e44 bytes) 000645 1783948663.314453 mapread 0x00532d thru 0x00f249 (0x9f1d bytes) 000647 1783948663.331072 mapwrite 0x03eb1f thru 0x03ffff (0x14e1 bytes) 000648 1783948663.515044 mapread 0x01af1e thru 0x027b3b (0xcc1e bytes) 000649 1783948663.576847 read 0x02bb8b thru 0x02c3d0 (0x846 bytes) 000650 1783948663.598017 mapread 0x021949 thru 0x0219c5 (0x7d bytes) 000651 1783948663.599189 read 0x0158df thru 0x016c45 (0x1367 bytes) 000652 1783948663.617107 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xc535bf08 (stale) Mirror 12: 0xc535bf08 (stale) Mirror 14: 0xc535bf08 (stale) Mirror 15: 0xc535bf08 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xc535bf08 (stale) Mirror 12: 0xc535bf08 (stale) Mirror 14: 0xc535bf08 (stale) Mirror 15: 0xc535bf08 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000654 1783948664.492117 trunc from 0x040000 to 0x037fa2 (0x805d bytes) 000656 1783948664.536250 trunc from 0x037fa3 to 0x0199b0 (0x1e5f2 bytes) 000657 1783948664.607938 read 0x010015 thru 0x0199b0 (0x999c bytes) 000659 1783948664.624837 write 0x0121a9 thru 0x01d095 (0xaeed bytes) 000660 1783948664.628458 write 0x039b01 thru 0x03ffff (0x64ff bytes) 000661 1783948664.634106 trunc from 0x040000 to 0x030c8d (0xf372 bytes) 000664 1783948664.651402 write 0x0229ed thru 0x0242b8 (0x18cc bytes) 000666 1783948664.663055 mapread 0x021f1f thru 0x030c8d (0xed6f bytes) 000668 1783948664.692329 read 0x00250c thru 0x006f28 (0x4a1d bytes) 000669 1783948664.704466 mapwrite 0x026f85 thru 0x02c3f6 (0x5472 bytes) 000670 1783948664.776233 write 0x00d960 thru 0x01a7dd (0xce7e bytes) 000671 1783948664.790240 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30c8e): Mirror 9: 0x2d287758 (stale) Mirror 12: 0x2d287758 (stale) Mirror 14: 0x2d287758 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30c8e): Mirror 9: 0x2d287758 (stale) Mirror 12: 0x2d287758 (stale) Mirror 14: 0x2d287758 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30c8e: skipped 000673 1783948665.411318 mapwrite 0x0390a4 thru 0x03ffff (0x6f5c bytes) 000674 1783948665.562295 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xc535bf08 (stale) Mirror 14: 0xc535bf08 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xc535bf08 (stale) Mirror 14: 0xc535bf08 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000675 1783948666.019190 read 0x01a651 thru 0x024d7a (0xa72a bytes) 000680 1783948666.041232 mapread 0x018040 thru 0x020a81 (0x8a42 bytes) 000681 1783948666.056820 trunc from 0x040000 to 0x01394e (0x2c6b1 bytes) 000682 1783948666.097349 mapread 0x00f768 thru 0x01394e (0x41e7 bytes) 000683 1783948666.125933 mapread 0x002f8b thru 0x0126ee (0xf764 bytes) 000689 1783948666.151265 mapwrite 0x023543 thru 0x0242ba (0xd78 bytes) 000690 1783948666.284289 mapread 0x004226 thru 0x00ff12 (0xbced bytes) 000693 1783948666.302956 write 0x019921 thru 0x01eeb5 (0x5595 bytes) 000699 1783948666.324583 read 0x018cf2 thru 0x023e8e (0xb19d bytes) 000701 1783948666.353075 mapread 0x018179 thru 0x01a84e (0x26d6 bytes) 000702 1783948666.355413 read 0x01da4c thru 0x0242ba (0x686f bytes) 000703 1783948666.357096 read 0x00d0fe thru 0x0115ba (0x44bd bytes) 000705 1783948666.359266 read 0x018618 thru 0x02021d (0x7c06 bytes) 000706 1783948666.361331 read 0x00cf74 thru 0x00ff8e (0x301b bytes) 000707 1783948666.362266 read 0x01aa19 thru 0x0241c8 (0x97b0 bytes) 000708 1783948666.365928 mapwrite 0x001922 thru 0x00a933 (0x9012 bytes) 000709 1783948666.452793 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x242bb): Mirror 14: 0xdf659d29 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x242bb): Mirror 14: 0xdf659d29 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x242bb: skipped 000710 1783948666.831152 mapread 0x022fa3 thru 0x0242ba (0x1318 bytes) 000715 1783948666.890826 trunc from 0x0242bb to 0x002bf0 (0x216ca bytes) 000717 1783948666.914496 mapwrite 0x0042b8 thru 0x00b990 (0x76d9 bytes) 000718 1783948667.118193 MIRROR_RESYNC 2 mirrors 000719 1783948668.105261 trunc from 0x00b991 to 0x00d9b5 (0x2025 bytes) 000722 1783948668.207975 trunc from 0x00d9b6 to 0x02ad55 (0x1d3a0 bytes) 000723 1783948668.250304 write 0x01edaf thru 0x0219d4 (0x2c26 bytes) 000724 1783948668.262314 trunc from 0x02ad56 to 0x0282e6 (0x2a6f bytes) 000726 1783948668.289009 write 0x01b119 thru 0x024222 (0x910a bytes) 000728 1783948668.308148 mapread 0x00ce14 thru 0x013944 (0x6b31 bytes) 000729 1783948668.331296 mapwrite 0x01086e thru 0x011998 (0x112b bytes) 000731 1783948668.395880 MIRROR_RESYNC 2 mirrors 000732 1783948669.297356 trunc from 0x0282e7 to 0x03cbfe (0x14918 bytes) 000733 1783948669.490702 trunc from 0x03cbff to 0x035827 (0x73d7 bytes) 000734 1783948669.528314 read 0x007799 thru 0x008e24 (0x168c bytes) 000738 1783948669.542483 write 0x031798 thru 0x03ebb1 (0xd41a bytes) 000739 1783948669.569346 read 0x02ec74 thru 0x0384b5 (0x9842 bytes) 000741 1783948669.609070 mapwrite 0x03ce35 thru 0x03ffff (0x31cb bytes) 000744 1783948669.860709 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x89165b71 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7e85e105 Mirror 15: 0x7e85e105 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000745 1783948671.128316 mapwrite 0x03c370 thru 0x03ffff (0x3c90 bytes) 000747 1783948671.520453 mapwrite 0x032758 thru 0x03ffff (0xd8a8 bytes) 000748 1783948671.604791 mapread 0x01f498 thru 0x023407 (0x3f70 bytes) 000750 1783948671.631212 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x89165b71 (stale) Mirror 15: 0x7e85e105 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb5854cd Mirror 16: 0xb5854cd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000751 1783948673.046095 mapread 0x009481 thru 0x014424 (0xafa4 bytes) 000753 1783948673.124282 read 0x031397 thru 0x03ffff (0xec69 bytes) 000756 1783948673.157598 read 0x002932 thru 0x002fc4 (0x693 bytes) 000759 1783948673.198390 mapwrite 0x02dbaf thru 0x039ba0 (0xbff2 bytes) 000760 1783948673.435674 mapwrite 0x02df07 thru 0x0393ff (0xb4f9 bytes) 000762 1783948673.515750 read 0x01233a thru 0x01925c (0x6f23 bytes) 000765 1783948673.545521 MIRROR_RESYNC 4 mirrors 000766 1783948674.768945 write 0x00f988 thru 0x0151e0 (0x5859 bytes) 000769 1783948674.922353 read 0x02851c thru 0x02f38c (0x6e71 bytes) 000770 1783948674.933996 mapread 0x0129b5 thru 0x01a032 (0x767e bytes) 000771 1783948674.957339 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x226a37ff (stale) Mirror 16: 0x226a37ff (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x226a37ff (stale) Mirror 16: 0x226a37ff (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000773 1783948675.510485 mapread 0x036d57 thru 0x03ffff (0x92a9 bytes) 000774 1783948675.552501 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x226a37ff (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x226a37ff (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000775 1783948676.026769 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x226a37ff (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1ab0feb8 Mirror 17: 0x1ab0feb8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000777 1783948677.095991 mapwrite 0x0099bd thru 0x010d60 (0x73a4 bytes) 000779 1783948677.388235 read 0x019f75 thru 0x01af88 (0x1014 bytes) 000780 1783948677.403313 mapwrite 0x01b446 thru 0x024f84 (0x9b3f bytes) 000781 1783948677.503053 mapread 0x01491c thru 0x023b8e (0xf273 bytes) 000782 1783948677.505371 write 0x036896 thru 0x037966 (0x10d1 bytes) 000783 1783948677.517335 mapwrite 0x02fb21 thru 0x034ff0 (0x54d0 bytes) 000785 1783948677.583312 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0x1ab0feb8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0x1ab0feb8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000787 1783948677.868839 mapread 0x01b5a4 thru 0x01cb74 (0x15d1 bytes) 000790 1783948677.899006 mapwrite 0x031d2e thru 0x035a10 (0x3ce3 bytes) 000791 1783948678.000043 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) 000792 1783948678.204982 read 0x03ae47 thru 0x03bb53 (0xd0d bytes) 000794 1783948678.216988 mapread 0x035dbe thru 0x03fd6c (0x9faf bytes) 000796 1783948678.284161 read 0x033a13 thru 0x03a457 (0x6a45 bytes) 000797 1783948678.288179 write 0x03dcd4 thru 0x03ffff (0x232c bytes) 000799 1783948678.304318 MIRROR_EXTEND to 2 mirrors 000801 1783948679.145749 trunc from 0x040000 to 0x0003d4 (0x3fc2b bytes) 000802 1783948679.292124 mapread 0x00034c thru 0x0003d4 (0x89 bytes) 000803 1783948679.304231 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d5): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d5): Mirror 2: 0xd6b83a62 (stale) CRC-32 checksum value for chunk [0, 0x3d5): Mirror 1: 0xd6b83a62 Mirror 3: 0xd6b83a62 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d5: skipped 000806 1783948680.553449 MIRROR_RESYNC 3 mirrors 000808 1783948681.303417 read 0x0001db thru 0x0003d4 (0x1fa bytes) 000811 1783948681.309465 mapread 0x00020a thru 0x0003d4 (0x1cb bytes) 000814 1783948681.311049 read 0x0002ad thru 0x0003d4 (0x128 bytes) 000815 1783948681.311489 write 0x00e6ae thru 0x01a1a1 (0xbaf4 bytes) 000816 1783948681.399169 mapwrite 0x01d256 thru 0x02388a (0x6635 bytes) 000820 1783948681.499375 write 0x02fe22 thru 0x03f861 (0xfa40 bytes) 000822 1783948681.507169 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f862): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f862): Mirror 2: 0x97024e50 (stale) Mirror 3: 0x97024e50 (stale) CRC-32 checksum value for chunk [0, 0x3f862): Mirror 1: 0x84417233 Mirror 4: 0x84417233 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f862: skipped 000823 1783948682.462485 trunc from 0x03f862 to 0x0281e0 (0x17681 bytes) 000826 1783948682.604374 trunc from 0x0281e1 to 0x029cc0 (0x1ae0 bytes) 000830 1783948682.634892 write 0x01038f thru 0x011ae2 (0x1754 bytes) 000833 1783948682.647803 trunc from 0x029cc1 to 0x032b36 (0x8e76 bytes) 000834 1783948682.667328 mapwrite 0x03a4f5 thru 0x03d61c (0x3128 bytes) 000835 1783948682.771455 read 0x01a775 thru 0x0256f1 (0xaf7d bytes) 000836 1783948682.795331 mapread 0x02c449 thru 0x035a92 (0x964a bytes) 000838 1783948682.809106 trunc from 0x03d61d to 0x003291 (0x3a38b bytes) 000839 1783948682.836343 write 0x029b55 thru 0x02d39a (0x3846 bytes) 000841 1783948682.840947 trunc from 0x02d39b to 0x0122d1 (0x1b0c9 bytes) 000842 1783948682.875104 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x122d2): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x122d2): Mirror 2: 0xea23a7cb (stale) Mirror 3: 0xea23a7cb (stale) Mirror 4: 0x5437b1a6 (stale) CRC-32 checksum value for chunk [0, 0x122d2): Mirror 1: 0xea23a7cb Mirror 5: 0xea23a7cb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x122d2: skipped 000843 1783948684.167136 mapwrite 0x0387d6 thru 0x03e495 (0x5cc0 bytes) 000846 1783948684.480326 mapread 0x018535 thru 0x01d059 (0x4b25 bytes) 000847 1783948684.515845 read 0x030c26 thru 0x031335 (0x710 bytes) 000849 1783948684.539325 read 0x02bf5a thru 0x03ac30 (0xecd7 bytes) 000850 1783948684.584425 read 0x008f9a thru 0x00a30c (0x1373 bytes) 000851 1783948684.593819 read 0x01b5b6 thru 0x02547a (0x9ec5 bytes) 000854 1783948684.601434 read 0x02a325 thru 0x036153 (0xbe2f bytes) 000856 1783948684.604885 write 0x038b80 thru 0x03cd2d (0x41ae bytes) 000858 1783948684.608024 read 0x003778 thru 0x00ad13 (0x759c bytes) 000860 1783948684.610396 read 0x02e9e5 thru 0x032e44 (0x4460 bytes) 000861 1783948684.611919 mapread 0x0098fd thru 0x0167ba (0xcebe bytes) 000865 1783948684.624476 mapwrite 0x030336 thru 0x0328ec (0x25b7 bytes) 000868 1783948684.772515 trunc from 0x03e496 to 0x011768 (0x2cd2d bytes) 000876 1783948684.857242 mapread 0x009037 thru 0x011768 (0x8732 bytes) 000877 1783948684.864864 trunc from 0x011769 to 0x026029 (0x148c1 bytes) 000879 1783948684.918793 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2602a): Mirror 2: 0x23106176 (stale) Mirror 4: 0xa5fc970e (stale) Mirror 5: 0x23106176 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2602a): Mirror 2: 0x23106176 (stale) Mirror 4: 0xa5fc970e (stale) Mirror 5: 0x23106176 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2602a: skipped 000882 1783948685.623712 read 0x00a21e thru 0x00ca85 (0x2868 bytes) 000883 1783948685.632509 read 0x010220 thru 0x019713 (0x94f4 bytes) 000885 1783948685.650286 read 0x01ab59 thru 0x01df15 (0x33bd bytes) 000886 1783948685.651340 read 0x00acc2 thru 0x01a3d6 (0xf715 bytes) 000887 1783948685.661797 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2602a): Mirror 4: 0xa5fc970e (stale) Mirror 5: 0x23106176 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2602a): Mirror 4: 0xa5fc970e (stale) Mirror 5: 0x23106176 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2602a: skipped 000888 1783948686.215986 read 0x0226e3 thru 0x026029 (0x3947 bytes) 000890 1783948686.245437 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2602a): Mirror 5: 0x23106176 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2602a): Mirror 5: 0x23106176 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2602a: skipped 000891 1783948686.596930 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2602a): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2602a): Mirror 5: 0x23106176 (stale) CRC-32 checksum value for chunk [0, 0x2602a): Mirror 1: 0x23106176 Mirror 6: 0x23106176 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2602a: skipped 000892 1783948687.748129 write 0x027de6 thru 0x0320b2 (0xa2cd bytes) 000894 1783948687.847962 trunc from 0x0320b3 to 0x02efbc (0x30f6 bytes) 000895 1783948687.879296 MIRROR_RESYNC 3 mirrors 000896 1783948689.171759 trunc from 0x02efbd to 0x00a0ca (0x24ef2 bytes) 000898 1783948689.291173 mapwrite 0x000d62 thru 0x007a10 (0x6caf bytes) 000899 1783948689.373983 trunc from 0x00a0cb to 0x03756f (0x2d4a5 bytes) 000900 1783948689.402118 mapread 0x002890 thru 0x00ceb2 (0xa623 bytes) 000901 1783948689.415803 mapwrite 0x015a4e thru 0x019489 (0x3a3c bytes) 000902 1783948689.469907 MIRROR_RESYNC 3 mirrors 000905 1783948690.345833 read 0x0254c6 thru 0x02f049 (0x9b84 bytes) 000908 1783948690.355674 read 0x003ea7 thru 0x0128fb (0xea55 bytes) 000910 1783948690.387935 MIRROR_RESYNC 3 mirrors 000913 1783948690.824606 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) 000915 1783948691.156446 trunc from 0x037570 to 0x02741c (0x10153 bytes) 000918 1783948691.246732 write 0x00f1a0 thru 0x0145f1 (0x5452 bytes) 000921 1783948691.263742 write 0x02ad7f thru 0x034766 (0x99e8 bytes) 000922 1783948691.267259 read 0x02936f thru 0x034766 (0xb3f8 bytes) 000923 1783948691.276777 read 0x01a999 thru 0x02589f (0xaf07 bytes) 000924 1783948691.288990 trunc from 0x034767 to 0x020777 (0x13fef bytes) 000927 1783948691.338361 trunc from 0x020778 to 0x029d1d (0x95a6 bytes) 000928 1783948691.362957 mapwrite 0x032014 thru 0x03cc14 (0xac01 bytes) 000929 1783948691.451933 mapwrite 0x00623b thru 0x006a74 (0x83a bytes) 000930 1783948691.498502 read 0x035d9b thru 0x03cc14 (0x6e7a bytes) 000932 1783948691.499812 trunc from 0x03cc15 to 0x01f15a (0x1daba bytes) 000933 1783948691.515309 write 0x0391bb thru 0x03ffff (0x6e45 bytes) 000935 1783948691.517209 mapwrite 0x00f97f thru 0x014e47 (0x54c9 bytes) 000937 1783948691.580590 mapread 0x03a88d thru 0x03ffff (0x5773 bytes) 000938 1783948691.581721 mapwrite 0x03bb59 thru 0x03ffff (0x44a7 bytes) 000943 1783948691.725872 MIRROR_RESYNC 2 mirrors 000944 1783948692.679603 MIRROR_RESYNC 2 mirrors 000945 1783948693.197970 read 0x00760a thru 0x00ae91 (0x3888 bytes) 000946 1783948693.206168 mapread 0x031569 thru 0x03b1d1 (0x9c69 bytes) 000948 1783948693.268904 write 0x02d14d thru 0x033b53 (0x6a07 bytes) 000949 1783948693.434571 mapwrite 0x035708 thru 0x03b242 (0x5b3b bytes) 000950 1783948693.531379 read 0x02f769 thru 0x03d3b6 (0xdc4e bytes) 000951 1783948693.532134 write 0x0262ad thru 0x033055 (0xcda9 bytes) 000952 1783948693.563333 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x507df128 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7406652 Mirror 6: 0x7406652 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000953 1783948694.711130 trunc from 0x040000 to 0x03a323 (0x5cdc bytes) 000954 1783948694.865716 trunc from 0x03a324 to 0x03abdc (0x8b9 bytes) 000956 1783948694.922260 mapread 0x026696 thru 0x02a515 (0x3e80 bytes) 000957 1783948695.005095 read 0x014100 thru 0x01d883 (0x9784 bytes) 000958 1783948695.045216 trunc from 0x03abdd to 0x01ab98 (0x20044 bytes) 000959 1783948695.074936 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1ab99): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ab99): Mirror 5: 0x7512f47b (stale) Mirror 6: 0x7512f47b (stale) CRC-32 checksum value for chunk [0, 0x1ab99): Mirror 1: 0x7512f47b Mirror 7: 0x7512f47b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ab99: skipped 000960 1783948696.107277 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ab99): Mirror 6: 0x7512f47b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ab99): Mirror 6: 0x7512f47b (stale) CRC-32 checksum value for chunk [0, 0x1ab99): Mirror 1: 0x7512f47b Mirror 7: 0x7512f47b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ab99: skipped 000961 1783948696.664966 mapread 0x0149c2 thru 0x01765f (0x2c9e bytes) 000962 1783948696.689285 trunc from 0x01ab99 to 0x01fc9e (0x5106 bytes) 000963 1783948696.802015 mapwrite 0x01828c thru 0x020ca8 (0x8a1d bytes) 000964 1783948696.884426 trunc from 0x020ca9 to 0x002ee4 (0x1ddc4 bytes) 000965 1783948696.916106 trunc from 0x002ee5 to 0x0231f0 (0x2030c bytes) 000968 1783948696.939368 read 0x01d199 thru 0x0231f0 (0x6058 bytes) 000971 1783948696.967721 read 0x00ff62 thru 0x01aaca (0xab69 bytes) 000973 1783948696.985221 trunc from 0x0231f1 to 0x02a622 (0x7432 bytes) 000974 1783948697.033898 write 0x005145 thru 0x010f20 (0xbddc bytes) 000975 1783948697.035593 read 0x00febd thru 0x0199a4 (0x9ae8 bytes) 000976 1783948697.036334 mapread 0x00b500 thru 0x01619a (0xac9b bytes) 000977 1783948697.046388 write 0x022303 thru 0x02eefa (0xcbf8 bytes) 000978 1783948697.049106 mapread 0x00f858 thru 0x01ec13 (0xf3bc bytes) 000981 1783948697.058360 MIRROR_RESYNC 3 mirrors 000982 1783948698.046406 mapread 0x025e7b thru 0x02eefa (0x9080 bytes) 000984 1783948698.079875 write 0x00e308 thru 0x0195f1 (0xb2ea bytes) 000985 1783948698.185073 trunc from 0x02eefb to 0x019362 (0x15b98 bytes) 000987 1783948698.234172 trunc from 0x019363 to 0x014e79 (0x44e9 bytes) 000989 1783948698.273942 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x14e7a): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14e7a): Mirror 6: 0x4321c4dc (stale) Mirror 7: 0x4321c4dc (stale) CRC-32 checksum value for chunk [0, 0x14e7a): Mirror 1: 0xf0995c77 Mirror 8: 0xf0995c77 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14e7a: skipped 000990 1783948699.596363 read 0x009e13 thru 0x010767 (0x6955 bytes) 000991 1783948699.635897 mapwrite 0x028fac thru 0x033d21 (0xad76 bytes) 000992 1783948699.929806 mapread 0x0047a9 thru 0x00b908 (0x7160 bytes) 000995 1783948699.968794 trunc from 0x033d22 to 0x02f0b1 (0x4c70 bytes) 000996 1783948700.007689 trunc from 0x02f0b2 to 0x019943 (0x1576e bytes) 000999 1783948700.053064 write 0x032f55 thru 0x03ffff (0xd0ab bytes) All operations completed A-OK! PASS 70a (136s) == sanity-flr test 71: check mirror extend parent fid ==== 09:18:26 (1783948706) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (5s) == sanity-flr test 72: check append on FLR file ========== 09:18:31 (1783948711) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.23235 s, 9.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386755 s, 2.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29d:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29d:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a2:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 09:18:42 (1783948722) Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre2 Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1702272 .. Extending file size to 6772448 .. Extending file size to 3895104 .. Extending file size to 8088128 .. Extending file size to 9144288 .. Extending file size to 6828320 .. Extending file size to 9291712 .. Extending file size to 8302336 .. Extending file size to 8866560 .. Extending file size to 9291296 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 57058 57059 57061 57063 57065 failed 192.168.203.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg335-client.virtnet /mnt/lustre2 (opts:) 192.168.203.135@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg335-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:20:03 (1783948803) Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre2 Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 2675840 .. Extending file size to 921024 .. failed Extending file size to 8372608 .. Extending file size to 8456576 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8578784 .. Extending file size to 9079776 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8720864 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8969952 .. Waiting 58590 58591 58592 58595 58597 done 192.168.203.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg335-client.virtnet /mnt/lustre2 (opts:) 192.168.203.135@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg335-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (83s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:21:26 (1783948886) Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.141312 s, 14.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.489717 s, 4.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x89a97462 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg335-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:21:47 (1783948907) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123558 s, 17.0 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 09:21:55 (1783948915) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b3:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b3:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b4:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2b7:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:22:03 (1783948923) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.216128 s, 9.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:22:12 (1783948932) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.185032 s, 11.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ba:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ba:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bb:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:22:20 (1783948940) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.25502 s, 8.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bd:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:22:54 (1783948974) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg335-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764082 s, 13.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b9:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b9:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ba:0x0] } PASS 204d (60s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:23:54 (1783949034) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg335-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.216961 s, 9.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (61s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:24:55 (1783949095) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg335-server mds-ost sync done. Creating new pool test_204f oleg335-server: Pool lustre.test_204f created Adding targets to pool oleg335-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg335-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.352065 s, 6.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg335-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg335-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg335-server: Pool lustre.test_204f destroyed PASS 204f (85s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:26:20 (1783949180) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:26:28 (1783949188) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:26:35 (1783949195) oleg335-server: Pool lustre.test_206 created Adding targets to pool oleg335-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg335-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg335-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg335-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg335-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:27:00 (1783949220) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492011 s, 21.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356037 s, 29.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xaf53ccc2 Mirror 3: 0xaf53ccc2 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:27:11 (1783949231) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:27:15 (1783949235) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:27:19 (1783949239) Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.156081 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2270000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1490000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1490000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1490000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x840000 Read this (4 bytes): BBBB 192.168.203.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg335-client.virtnet /mnt/lustre2 (opts:) PASS 209a (36s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:27:55 (1783949275) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0150851 s, 543 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:28:02 (1783949282) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483416 s, 21.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1e2 lmm_fid: [0x200000404:0x1e2:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1e2 lmm_fid: [0x200000404:0x1e2:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834402 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-13 09:28:03.000000000 -0400 Modify: 2026-07-13 09:28:03.000000000 -0400 Change: 2026-07-13 09:28:04.000000000 -0400 Birth: 2026-07-13 09:28:03.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:28:09 (1783949289) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:28:13 (1783949293) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.561082 s, 73.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.049034 s, 83.5 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 13 09:28 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e4:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 13 09:28 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 09:28:22 (1783949302) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 09:28:30 (1783949310) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1852 sec =========== 09:28:39 (1783949319) === sanity-flr: start cleanup 09:28:41 (1783949321) === === sanity-flr: finish cleanup 09:28:47 (1783949327) ===