-----============= acceptance-small: sanity-flr ============----- Wed Aug 12 15:08:25 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg429-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 15:08:34 (1786561714) === oleg429-client.virtnet: executing check_config_client /mnt/lustre oleg429-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg429-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff997f0a325000.idle_timeout=debug osc.lustre-OST0001-osc-ffff997f0a325000.idle_timeout=debug disable quota as required oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 15:08:49 (1786561729) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7631] == sanity-flr test 0a: lfs mirror create with -N option == 15:08:50 (1786561730) PASS 0a (6s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:08:56 (1786561736) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:08:58 (1786561738) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 15:08:59 (1786561739) PASS 0d (8s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:09:07 (1786561747) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:09:09 (1786561749) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 15:09:10 (1786561750) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff997f0a325000.stats=clear osc.lustre-OST0001-osc-ffff997f0a325000.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:09:14 (1786561754) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (5s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:09:18 (1786561758) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 15:09:21 (1786561761) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 15:09:23 (1786561763) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created error: lfs extend: layout is NULL /mnt/lustre/d0m.sanity-flr/f0m.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000401 lmm_object_id: 0x16 lmm_fid: [0x200000401:0x16:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x20:0x0] } - lcme_id: 0x20001 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000401 lmm_object_id: 0x16 lmm_fid: [0x200000401:0x16:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x20:0x0] } sanity-flr test_0m: @@@@@@ FAIL: failed to call extend /mnt/lustre/d0m.sanity-flr/f0m.sanity-flr Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:899:test_0m() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:909:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_0m.*.1786561764.log rsync: chown "/tmp/testlogs/.sanity-flr.test_0m.debug_log.oleg429-server.1786561764.log.P7ZG0O" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_0m.dmesg.oleg429-server.1786561764.log.OT10Yr" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 0m (3s) == sanity-flr test 0n: lfs mirror extend with --no-resync option ========================================================== 15:09:30 (1786561770) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.364294 s, 17.3 MB/s - original checksum: 6f1e89d272a9095297d7983ded33b2e9 - extend 1 mirror check the 2nd mirror - mirror 2 checksum: 6f1e89d272a9095297d7983ded33b2e9 - extend 2 more mirrors check the stale 3rd mirror - mirror 3 checksum: d41d8cd98f00b204e9800998ecf8427e - check the stale 4th mirror - mirror 4 checksum: d41d8cd98f00b204e9800998ecf8427e - resync the file check the 3rd mirror after resync - mirror 3 checksum after resync: 6f1e89d272a9095297d7983ded33b2e9 - check the 4th mirror after resync - mirror 4 checksum after resync: 6f1e89d272a9095297d7983ded33b2e9 - PASS 0n (34s) == sanity-flr test 1: create components with setstripe options ========================================================== 15:10:04 (1786561804) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x1c:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 15:10:12 (1786561812) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:10:20 (1786561820) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (5s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:10:25 (1786561825) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory error: lfs extend: layout is NULL PASS 4 (12s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:10:37 (1786561837) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 15:10:45 (1786561845) PASS 7 (4s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:10:49 (1786561849) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.889813 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100703 s, 10.4 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:10:56 (1786561856) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.139731 s, 90.1 MB/s PASS 22 (4s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 15:11:00 (1786561860) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (6s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:11:06 (1786561866) Stopping /mnt/lustre-ost1 (opts:) on oleg429-server oleg429-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff997f0a325000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff997f0a325000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg429-server: oleg429-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg429-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg429-server oleg429-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff997f0a325000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff997f0a325000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg429-server: oleg429-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg429-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (38s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 15:11:44 (1786561904) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.248805 s, 29.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg429-server oleg429-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff997f0a325000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff997f0a325000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg429-server: oleg429-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg429-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (24s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:12:08 (1786561928) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg429-server oleg429-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff997f0a325000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff997f0a325000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg429-server: oleg429-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg429-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg429-server oleg429-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff997f0a325000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff997f0a325000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg429-server: oleg429-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg429-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (41s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:12:49 (1786561969) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.48484 s, 21.1 MB/s -rw-r--r-- 1 root root 52428800 Aug 12 15:12 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.585s user 0m0.001s sys 0m0.178s Stopping /mnt/lustre-ost1 (opts:) on oleg429-server oleg429-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff997f0a325000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff997f0a325000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.610s user 0m0.002s sys 0m0.182s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg429-server: oleg429-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg429-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg429-server oleg429-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff997f0a325000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff997f0a325000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.633s user 0m0.003s sys 0m0.176s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg429-server: oleg429-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg429-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg429-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff997f0a325000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff997f0a325000.ost_server_uuid in FULL state after 0 sec PASS 33b (51s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:13:40 (1786562020) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:13:42 (1786562022) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:13:44 (1786562024) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 15:13:45 (1786562025) PASS 35 (4s) == sanity-flr test 36a: write to mirrored files ========== 15:13:49 (1786562029) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.376829 s, 33.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (8s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:13:57 (1786562037) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245379 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183064 s, 57.3 MB/s PASS 36b (4s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:14:01 (1786562041) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:14:05 (1786562045) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00105417 s, 971 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00113553 s, 902 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0031771 s, 322 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00113846 s, 899 kB/s flv=10 olv=10 PASS 36d (6s) == sanity-flr test 37: mirror I/O API verification ======= 15:14:11 (1786562051) create test files with size 15759334 .. e7bef882b50722becedbd1c33d5f121d - ebb821ff5c50c96075c48180b854ee2e - 572686d2b20e73ce7aa458f4266be17f - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg429-client.virtnet /mnt/lustre (opts:) Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre PASS 37 (11s) == sanity-flr test 38: resync ============================ 15:14:22 (1786562062) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (8s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 15:14:30 (1786562070) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 15:14:33 (1786562073) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799046 s, 13.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd1:0x0 - lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd2:0x0 - lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xd0:0x0 - lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20007 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153503 s, 68.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xd1:0x0 - lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xd2:0x0 - lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd3:0x0 - lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20007 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (5s) == sanity-flr test 41: lfs mirror resync check =========== 15:14:38 (1786562078) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0649557 s, 32.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.268997 s, 15.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.129322 s, 16.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (8s) == sanity-flr test 42: lfs mirror verify ================= 15:14:47 (1786562087) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 15:14:48 (1786562088) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 15:14:50 (1786562090) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220331 s, 47.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 15:14:54 (1786562094) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 15:14:56 (1786562096) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } lcme_id: 0x20001 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xda:0x0] } split mirror_id 1 id 0x10001 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 0x20001 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xda:0x0] } split mirror_id 2 id 0x20001 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:15:01 (1786562101) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.118541 s, 88.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg429-client.virtnet /mnt/lustre (opts:) Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:15:15 (1786562115) llite.lustre-ffff997f07896800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.142287 s, 73.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg429-client.virtnet /mnt/lustre (opts:) Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff997f08e0d800.xattr_cache=1 PASS 44e (7s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 15:15:22 (1786562122) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (18s) == sanity-flr test 46: Verify setstripe --copy option ==== 15:15:40 (1786562140) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (15s) == sanity-flr test 47: Verify mirror obj alloc =========== 15:15:55 (1786562155) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 15:15:57 (1786562157) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.119885 s, 26.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0904934 s, 34.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x5b4ed72 (nosync,stale) Mirror 2: 0x5b4ed72 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 07912fd63479d88ee04cf328bf53b210 - mirror 1 checksum: d29aa4a1e266e6d8a1ec6d3d853fa160 - mirror 2 checksum: 07912fd63479d88ee04cf328bf53b210 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-12T15:15:59-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xef:0x0] } - lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-12T15:15:59-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xf0:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 07912fd63479d88ee04cf328bf53b210 - mirror 1 checksum: d29aa4a1e266e6d8a1ec6d3d853fa160 - mirror 2 checksum: d29aa4a1e266e6d8a1ec6d3d853fa160 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-12T15:15:59-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xef:0x0] } - lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-12T15:15:59-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xf0:0x0] } PASS 48 (5s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 15:16:03 (1786562163) Failing mds1 on oleg429-server Stopping /mnt/lustre-mds1 (opts:) on oleg429-server 15:16:06 (1786562166) shut down facet: mds1 facet_host: oleg429-server facet_failover_host: oleg429-server Failover mds1 to oleg429-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0000 15:16:22 (1786562182) targets are mounted 15:16:22 (1786562182) facet_failover done oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141042 s, 74.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 0x20001 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xf1:0x0] } PASS 50A (28s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 15:16:31 (1786562191) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 15:16:32 (1786562192) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 15:16:33 (1786562193) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00618698 s, 2.6 MB/s PASS 50c (4s) == sanity-flr test 50d: mirror rsync keep holes ========== 15:16:37 (1786562197) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 15:16:38 (1786562198) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00308497 s, 1.3 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0360626 s, 114 kB/s PASS 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 15:16:42 (1786562202) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (10s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 15:16:53 (1786562213) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 15:17:13 (1786562233) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (19s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 15:17:32 (1786562252) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 15:17:50 (1786562270) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 15:17:52 (1786562272) mirror create pid 56794 mirror split pid 56795 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 56794 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 56795 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 15:18:55 (1786562335) Seed set to 2336 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000005 1786562336.213416 MIRROR_EXTEND to 2 mirrors 000006 1786562336.343758 mapwrite 0x01a573 thru 0x02014e (0x5bdc bytes) 000008 1786562336.401520 write 0x01d9c3 thru 0x01dae1 (0x11f bytes) 000009 1786562336.404778 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) 000010 1786562336.453922 read 0x004af7 thru 0x012197 (0xd6a1 bytes) 000011 1786562336.457944 read 0x0075c0 thru 0x007cda (0x71b bytes) 000012 1786562336.458374 mapwrite 0x00201a thru 0x004373 (0x235a bytes) 000013 1786562336.476938 read 0x01b657 thru 0x02014e (0x4af8 bytes) 000014 1786562336.477695 mapwrite 0x011c39 thru 0x016172 (0x453a bytes) truncating to largest ever: 0x15c14 000015 1786562336.496957 trunc from 0x02014f to 0x015c13 (0xa53b bytes) 000017 1786562336.506136 mapwrite 0x0112ed thru 0x016e1f (0x5b33 bytes) 000018 1786562336.536444 mapwrite 0x002998 thru 0x00c138 (0x97a1 bytes) 000020 1786562336.558728 mapwrite 0x01d41f thru 0x0217ed (0x43cf bytes) 000021 1786562336.587275 mapwrite 0x012af8 thru 0x01a4de (0x79e7 bytes) 000023 1786562336.605884 MIRROR_EXTEND to 2 mirrors 000025 1786562336.838052 write 0x03ef38 thru 0x03ffff (0x10c8 bytes) 000026 1786562336.862120 trunc from 0x040000 to 0x014e9f (0x2b160 bytes) 000029 1786562336.877933 mapread 0x005b71 thru 0x00b22d (0x56bd bytes) 000032 1786562336.888351 mapread 0x000fa4 thru 0x001f4c (0xfa9 bytes) 000033 1786562336.892110 mapread 0x003a5b thru 0x00cd6d (0x9313 bytes) 000035 1786562336.893613 trunc from 0x014ea0 to 0x00323f (0x11c60 bytes) 000036 1786562336.902399 write 0x017fca thru 0x027dc9 (0xfe00 bytes) 000037 1786562336.904591 read 0x012ac1 thru 0x020934 (0xde74 bytes) 000038 1786562336.909372 mapread 0x005e1b thru 0x00b0d8 (0x52be bytes) truncating to largest ever: 0x315be 000039 1786562336.913523 trunc from 0x027dca to 0x0315bd (0x97f4 bytes) 000040 1786562336.920551 mapwrite 0x007957 thru 0x00a289 (0x2933 bytes) 000042 1786562336.938651 read 0x0148c3 thru 0x01be65 (0x75a3 bytes) 000043 1786562336.939365 read 0x02bc2a thru 0x02f2a2 (0x3679 bytes) 000044 1786562336.946850 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) 000045 1786562337.013701 mapread 0x00eea8 thru 0x01b8e2 (0xca3b bytes) 000048 1786562337.031317 MIRROR_EXTEND to 2 mirrors 000049 1786562337.451086 mapwrite 0x006c3f thru 0x0105dc (0x999e bytes) 000050 1786562337.518965 MIRROR_RESYNC 2 mirrors 000053 1786562337.892924 mapwrite 0x021fd9 thru 0x023eb3 (0x1edb bytes) 000055 1786562337.975241 read 0x02e46c thru 0x0315bd (0x3152 bytes) 000056 1786562337.982544 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) 000058 1786562338.050689 read 0x0003a9 thru 0x006d1e (0x6976 bytes) 000066 1786562338.058546 mapwrite 0x00ad41 thru 0x016fba (0xc27a bytes) 000069 1786562338.095018 mapread 0x013b7d thru 0x019b8a (0x600e bytes) 000072 1786562338.097362 mapwrite 0x0077c8 thru 0x0079e3 (0x21c bytes) 000074 1786562338.114821 trunc from 0x0315be to 0x0252bb (0xc302 bytes) 000081 1786562338.124114 trunc from 0x0252bc to 0x0241af (0x110c bytes) 000086 1786562338.133841 read 0x020efb thru 0x0241af (0x32b5 bytes) 000088 1786562338.139593 write 0x00a0b2 thru 0x015eb1 (0xbe00 bytes) 000090 1786562338.141531 write 0x0182ca thru 0x01f2e3 (0x701a bytes) 000096 1786562338.145368 mapwrite 0x035fea thru 0x03ffff (0xa016 bytes) 000097 1786562338.186433 trunc from 0x040000 to 0x01eb4a (0x214b5 bytes) 000098 1786562338.196138 mapread 0x008b86 thru 0x014f87 (0xc402 bytes) 000100 1786562338.198171 read 0x00490d thru 0x00fcd1 (0xb3c5 bytes) 000101 1786562338.198889 mapwrite 0x0039f9 thru 0x0051d4 (0x17dc bytes) 000102 1786562338.221641 MIRROR_EXTEND to 2 mirrors 000103 1786562338.471940 mapread 0x0013b2 thru 0x0109ab (0xf5fa bytes) 000104 1786562338.481157 trunc from 0x01eb4b to 0x02dea6 (0xf35c bytes) 000105 1786562338.515654 read 0x013b0a thru 0x022dbf (0xf2b6 bytes) 000109 1786562338.521348 mapwrite 0x035843 thru 0x039b18 (0x42d6 bytes) 000111 1786562338.547859 mapread 0x02ae62 thru 0x02c5c3 (0x1762 bytes) 000112 1786562338.553245 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) 000113 1786562338.600984 read 0x00f542 thru 0x010106 (0xbc5 bytes) 000114 1786562338.605814 MIRROR_EXTEND to 2 mirrors 000115 1786562338.850107 write 0x01f516 thru 0x02a3df (0xaeca bytes) 000116 1786562338.881086 read 0x0361d6 thru 0x036f90 (0xdbb bytes) 000117 1786562338.884880 mapread 0x01e01f thru 0x024cf3 (0x6cd5 bytes) 000120 1786562338.889740 mapwrite 0x031f06 thru 0x0338c9 (0x19c4 bytes) 000122 1786562338.920460 MIRROR_RESYNC 2 mirrors 000126 1786562339.217916 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) 000130 1786562339.274492 read 0x005d1f thru 0x00f5f1 (0x98d3 bytes) 000132 1786562339.280446 trunc from 0x039b19 to 0x013b93 (0x25f85 bytes) 000133 1786562339.295905 read 0x00b1da thru 0x013b93 (0x89ba bytes) 000134 1786562339.300785 read 0x00c908 thru 0x00ff44 (0x363d bytes) 000136 1786562339.304417 MIRROR_EXTEND to 2 mirrors 000139 1786562339.560710 read 0x0025c3 thru 0x00f945 (0xd383 bytes) truncating to largest ever: 0x3e9d5 000143 1786562339.565540 trunc from 0x013b94 to 0x03e9d4 (0x2ae41 bytes) 000146 1786562339.596288 mapread 0x01b8db thru 0x01d79d (0x1ec3 bytes) 000148 1786562339.604131 trunc from 0x03e9d5 to 0x02a06e (0x14966 bytes) 000149 1786562339.611562 mapread 0x0164ba thru 0x0205e3 (0xa12a bytes) 000150 1786562339.612729 mapwrite 0x001f23 thru 0x00521d (0x32fb bytes) 000152 1786562339.637298 MIRROR_RESYNC 2 mirrors 000153 1786562339.847972 mapread 0x00eb69 thru 0x0168ec (0x7d84 bytes) 000154 1786562339.854880 write 0x021f29 thru 0x027004 (0x50dc bytes) 000155 1786562339.874480 MIRROR_RESYNC 2 mirrors 000157 1786562340.113738 trunc from 0x02a06f to 0x017f79 (0x120f5 bytes) 000158 1786562340.139380 write 0x03d84f thru 0x03ffff (0x27b1 bytes) 000159 1786562340.142943 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x617a3f74 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfa2292a2 Mirror 3: 0xfa2292a2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000160 1786562340.475054 trunc from 0x040000 to 0x020b88 (0x1f477 bytes) 000162 1786562340.500666 mapread 0x005885 thru 0x006a3d (0x11b9 bytes) 000164 1786562340.504550 read 0x015ffc thru 0x020b88 (0xab8d bytes) 000165 1786562340.507571 trunc from 0x020b89 to 0x006fd6 (0x19bb2 bytes) 000166 1786562340.514042 read 0x000a8c thru 0x003e82 (0x33f7 bytes) 000168 1786562340.516987 trunc from 0x006fd7 to 0x01ad38 (0x13d62 bytes) 000170 1786562340.523749 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ad39): Mirror 3: 0xd7cad66d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ad39): Mirror 3: 0xd7cad66d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ad39: skipped 000171 1786562340.625840 read 0x018c04 thru 0x01ad38 (0x2135 bytes) 000174 1786562340.630757 mapread 0x0077e5 thru 0x00e530 (0x6d4c bytes) 000175 1786562340.642852 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1ad39): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ad39): Mirror 3: 0xd7cad66d (stale) CRC-32 checksum value for chunk [0, 0x1ad39): Mirror 1: 0xc1237da7 Mirror 4: 0xc1237da7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ad39: skipped 000176 1786562341.012000 write 0x009511 thru 0x01543c (0xbf2c bytes) 000181 1786562341.039239 mapwrite 0x020df2 thru 0x0225b9 (0x17c8 bytes) 000182 1786562341.085437 read 0x008ae7 thru 0x00b0ff (0x2619 bytes) 000186 1786562341.090351 write 0x031c06 thru 0x038418 (0x6813 bytes) 000187 1786562341.091682 read 0x0265c1 thru 0x034e40 (0xe880 bytes) 000189 1786562341.096894 trunc from 0x038419 to 0x0306f1 (0x7d27 bytes) 000190 1786562341.103272 mapread 0x006cb4 thru 0x0160fb (0xf448 bytes) 000191 1786562341.104565 mapread 0x00cfe6 thru 0x00e5cd (0x15e8 bytes) 000194 1786562341.106609 write 0x020483 thru 0x02df71 (0xdaef bytes) 000195 1786562341.107900 write 0x0311a0 thru 0x03ffff (0xee60 bytes) 000197 1786562341.109830 mapread 0x017274 thru 0x024eaa (0xdc37 bytes) 000198 1786562341.113456 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfa2292a2 (stale) Mirror 4: 0x83f916de (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7016bb89 Mirror 5: 0x7016bb89 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000200 1786562341.534588 mapread 0x0070a3 thru 0x011265 (0xa1c3 bytes) 000201 1786562341.541248 trunc from 0x040000 to 0x034be8 (0xb417 bytes) 000205 1786562341.578665 mapwrite 0x03151f thru 0x03fb60 (0xe642 bytes) 000206 1786562341.618302 read 0x026a09 thru 0x02c363 (0x595b bytes) 000211 1786562341.624400 mapwrite 0x00ed23 thru 0x017b77 (0x8e55 bytes) 000212 1786562341.654203 trunc from 0x03fb61 to 0x01dc8a (0x21ed6 bytes) 000213 1786562341.665723 mapwrite 0x023341 thru 0x02976c (0x642c bytes) 000219 1786562341.695486 read 0x02918f thru 0x02976c (0x5de bytes) 000221 1786562341.696471 mapwrite 0x019509 thru 0x019ed6 (0x9ce bytes) 000222 1786562341.713373 mapread 0x004eeb thru 0x00aca2 (0x5db8 bytes) 000224 1786562341.718954 mapwrite 0x01064a thru 0x01b744 (0xb0fb bytes) 000225 1786562341.740116 trunc from 0x02976d to 0x03452a (0xadbe bytes) 000226 1786562341.747528 write 0x030265 thru 0x03aa0b (0xa7a7 bytes) 000227 1786562341.752131 mapwrite 0x026979 thru 0x02e633 (0x7cbb bytes) 000228 1786562341.773837 mapread 0x01535e thru 0x024592 (0xf235 bytes) 000229 1786562341.774996 mapwrite 0x01040a thru 0x01aa03 (0xa5fa bytes) 000232 1786562341.796483 trunc from 0x03aa0c to 0x020033 (0x1a9d8 bytes) 000235 1786562341.807167 MIRROR_RESYNC 4 mirrors 000238 1786562342.238066 mapwrite 0x03ff89 thru 0x03ffff (0x77 bytes) 000239 1786562342.286729 mapread 0x035cc4 thru 0x039892 (0x3bcf bytes) 000240 1786562342.293706 mapwrite 0x02e599 thru 0x03a7c0 (0xc228 bytes) 000241 1786562342.318256 read 0x0255d1 thru 0x02d261 (0x7c91 bytes) 000243 1786562342.321815 read 0x01bbcd thru 0x01de81 (0x22b5 bytes) 000244 1786562342.326410 trunc from 0x040000 to 0x01a49e (0x25b61 bytes) 000245 1786562342.335864 write 0x02cc51 thru 0x037dc3 (0xb173 bytes) 000247 1786562342.337804 read 0x036fdd thru 0x037dc3 (0xde7 bytes) 000250 1786562342.339408 write 0x030f67 thru 0x03ee2e (0xdec8 bytes) 000251 1786562342.340988 read 0x03cd0c thru 0x03ee2e (0x2123 bytes) 000253 1786562342.342089 write 0x014fa8 thru 0x01cc5b (0x7cb4 bytes) 000254 1786562342.345437 mapread 0x028929 thru 0x0358df (0xcfb7 bytes) 000259 1786562342.352626 mapwrite 0x019f41 thru 0x022892 (0x8952 bytes) 000260 1786562342.375632 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3ee2f): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ee2f): Mirror 3: 0x4babb8ba (stale) Mirror 4: 0x4babb8ba (stale) Mirror 5: 0x4babb8ba (stale) CRC-32 checksum value for chunk [0, 0x3ee2f): Mirror 1: 0x65c1d3c0 Mirror 6: 0x65c1d3c0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ee2f: skipped 000261 1786562342.722801 trunc from 0x03ee2f to 0x01a0d3 (0x24d5b bytes) 000262 1786562342.765542 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a0d4): Mirror 3: 0xac70703e (stale) Mirror 4: 0xac70703e (stale) Mirror 5: 0xac70703e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a0d4): Mirror 3: 0xac70703e (stale) Mirror 4: 0xac70703e (stale) Mirror 5: 0xac70703e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a0d4: skipped 000265 1786562342.902741 write 0x006169 thru 0x00edd9 (0x8c71 bytes) 000266 1786562342.906662 read 0x00e1d8 thru 0x0163c7 (0x81f0 bytes) 000267 1786562342.909468 read 0x00e7c4 thru 0x0147bb (0x5ff8 bytes) 000271 1786562342.911084 mapread 0x019eb8 thru 0x01a0d3 (0x21c bytes) 000273 1786562342.915591 trunc from 0x01a0d4 to 0x0105c2 (0x9b11 bytes) 000276 1786562342.923024 mapread 0x0027c4 thru 0x006f7a (0x47b7 bytes) 000277 1786562342.924059 trunc from 0x0105c3 to 0x0089f9 (0x7bc9 bytes) 000281 1786562342.932429 trunc from 0x0089fa to 0x02f8c1 (0x26ec8 bytes) 000282 1786562342.938012 write 0x0302a2 thru 0x03b37a (0xb0d9 bytes) 000283 1786562342.939081 read 0x00b739 thru 0x010f4f (0x5817 bytes) 000284 1786562342.943466 write 0x0093b6 thru 0x012de1 (0x9a2c bytes) 000285 1786562342.944464 trunc from 0x03b37b to 0x03abd0 (0x7aa bytes) 000286 1786562342.951557 MIRROR_RESYNC 4 mirrors 000289 1786562343.317452 MIRROR_RESYNC 4 mirrors 000291 1786562343.457089 write 0x01148a thru 0x015031 (0x3ba8 bytes) 000293 1786562343.480944 mapread 0x02a8a5 thru 0x033fff (0x975b bytes) 000294 1786562343.488182 mapread 0x0066a6 thru 0x00c535 (0x5e90 bytes) 000295 1786562343.492583 read 0x005000 thru 0x00bd29 (0x6d2a bytes) 000296 1786562343.493189 mapwrite 0x00dea0 thru 0x017001 (0x9162 bytes) 000297 1786562343.515989 mapwrite 0x013319 thru 0x015f9b (0x2c83 bytes) 000298 1786562343.531131 trunc from 0x03abd1 to 0x01243c (0x28794 bytes) 000301 1786562343.540697 MIRROR_RESYNC 4 mirrors 000302 1786562343.829105 mapread 0x010a3e thru 0x01243c (0x19ff bytes) 000306 1786562343.836547 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) 000307 1786562343.950534 mapread 0x009ebb thru 0x01243c (0x8582 bytes) 000308 1786562343.957008 write 0x00c264 thru 0x01add4 (0xeb71 bytes) 000309 1786562343.978627 mapwrite 0x01dc60 thru 0x01f1bf (0x1560 bytes) 000310 1786562344.007358 trunc from 0x01f1c0 to 0x020aa1 (0x18e2 bytes) 000311 1786562344.013952 read 0x007140 thru 0x0161cf (0xf090 bytes) 000313 1786562344.018152 write 0x016e33 thru 0x01cf43 (0x6111 bytes) 000314 1786562344.019248 mapwrite 0x003473 thru 0x00f8f1 (0xc47f bytes) 000315 1786562344.046511 read 0x00a2ec thru 0x012843 (0x8558 bytes) 000316 1786562344.047106 trunc from 0x020aa2 to 0x016867 (0xa23a bytes) 000317 1786562344.055977 trunc from 0x016868 to 0x00f9bd (0x6eaa bytes) 000320 1786562344.064097 trunc from 0x00f9be to 0x003c05 (0xbdb8 bytes) 000321 1786562344.073127 MIRROR_RESYNC 3 mirrors 000322 1786562344.328039 write 0x018d48 thru 0x01e16d (0x5426 bytes) 000323 1786562344.352926 mapwrite 0x02de63 thru 0x030c7b (0x2e19 bytes) 000324 1786562344.396933 mapread 0x02d228 thru 0x030c7b (0x3a54 bytes) 000325 1786562344.397965 mapwrite 0x014487 thru 0x023a4c (0xf5c6 bytes) 000326 1786562344.424855 write 0x03331f thru 0x03e793 (0xb475 bytes) 000328 1786562344.426787 mapread 0x01bf5b thru 0x02bf11 (0xffb7 bytes) 000330 1786562344.428692 mapwrite 0x02bb09 thru 0x02d886 (0x1d7e bytes) 000331 1786562344.447282 mapwrite 0x038418 thru 0x03aa2a (0x2613 bytes) 000333 1786562344.462063 mapread 0x00fb55 thru 0x010f08 (0x13b4 bytes) 000335 1786562344.463413 trunc from 0x03e794 to 0x028e9b (0x158f8 bytes) 000336 1786562344.470578 write 0x0121b5 thru 0x0143c8 (0x2214 bytes) 000337 1786562344.471598 mapwrite 0x02bfa4 thru 0x0360d3 (0xa130 bytes) 000339 1786562344.504625 read 0x002679 thru 0x004fca (0x2952 bytes) 000340 1786562344.507868 mapwrite 0x0346b5 thru 0x037a5a (0x33a6 bytes) 000341 1786562344.535020 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37a5b): Mirror 4: 0x4931843e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a5b): Mirror 4: 0x4931843e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a5b: skipped 000344 1786562344.648000 mapwrite 0x011607 thru 0x012a89 (0x1483 bytes) 000346 1786562344.666024 mapwrite 0x00cc40 thru 0x014d0f (0x80d0 bytes) 000347 1786562344.685203 mapwrite 0x03ca00 thru 0x03ffff (0x3600 bytes) 000348 1786562344.707168 mapread 0x03797a thru 0x03ab62 (0x31e9 bytes) 000350 1786562344.708773 trunc from 0x040000 to 0x0175c8 (0x28a37 bytes) 000352 1786562344.715703 write 0x010281 thru 0x01bc98 (0xba18 bytes) 000353 1786562344.716936 mapread 0x00ed35 thru 0x017ebc (0x9188 bytes) 000355 1786562344.718337 mapread 0x00f168 thru 0x0178ed (0x8786 bytes) 000356 1786562344.719155 trunc from 0x01bc99 to 0x00ddfe (0xde9a bytes) 000357 1786562344.724752 trunc from 0x00ddff to 0x025313 (0x17515 bytes) 000359 1786562344.730785 mapread 0x00beeb thru 0x014e2f (0x8f45 bytes) 000360 1786562344.736290 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) 000361 1786562344.781051 trunc from 0x025314 to 0x00c6c6 (0x18c4d bytes) 000366 1786562344.790431 write 0x03c9dd thru 0x03ffff (0x3623 bytes) 000369 1786562344.792585 write 0x019290 thru 0x0201f0 (0x6f61 bytes) 000371 1786562344.798093 write 0x006846 thru 0x007c47 (0x1402 bytes) 000373 1786562344.803356 write 0x02f634 thru 0x037708 (0x80d5 bytes) 000375 1786562344.807514 trunc from 0x040000 to 0x01e584 (0x21a7b bytes) 000379 1786562344.815745 write 0x029f55 thru 0x033e25 (0x9ed1 bytes) 000381 1786562344.817712 trunc from 0x033e26 to 0x032420 (0x1a05 bytes) 000382 1786562344.823865 read 0x02ba69 thru 0x0307db (0x4d73 bytes) 000383 1786562344.824448 mapread 0x030f6b thru 0x032420 (0x14b6 bytes) 000385 1786562344.825747 read 0x005182 thru 0x00beea (0x6d69 bytes) 000387 1786562344.830897 mapread 0x01f472 thru 0x028a93 (0x9622 bytes) 000389 1786562344.832812 read 0x02a44d thru 0x032420 (0x7fd4 bytes) 000391 1786562344.834024 mapwrite 0x01e3e3 thru 0x02b9bf (0xd5dd bytes) 000392 1786562344.856699 read 0x00ed08 thru 0x01eb68 (0xfe61 bytes) 000394 1786562344.858073 mapread 0x0136be thru 0x0149c7 (0x130a bytes) 000395 1786562344.858969 read 0x013599 thru 0x021edf (0xe947 bytes) 000397 1786562344.860049 trunc from 0x032421 to 0x012a52 (0x1f9ce bytes) 000398 1786562344.866813 write 0x01d018 thru 0x027db8 (0xada1 bytes) 000401 1786562344.869585 trunc from 0x027db9 to 0x01f9d5 (0x83e3 bytes) 000402 1786562344.876844 mapread 0x00d1d6 thru 0x0171cf (0x9ffa bytes) 000403 1786562344.882479 trunc from 0x01f9d6 to 0x00d596 (0x1243f bytes) 000404 1786562344.889378 mapwrite 0x0150e1 thru 0x01c567 (0x7487 bytes) 000406 1786562344.932329 read 0x0134a5 thru 0x01c567 (0x90c3 bytes) 000407 1786562344.935861 mapwrite 0x0230d5 thru 0x0230f4 (0x20 bytes) 000409 1786562344.959863 mapread 0x01688e thru 0x01e391 (0x7b04 bytes) 000410 1786562344.960914 write 0x0367c6 thru 0x03ffff (0x983a bytes) 000411 1786562344.961989 mapread 0x0381da thru 0x03ffff (0x7e26 bytes) 000414 1786562344.963689 mapread 0x01161c thru 0x015d8c (0x4771 bytes) 000416 1786562344.965639 write 0x017271 thru 0x018da5 (0x1b35 bytes) 000417 1786562344.966470 write 0x0382c0 thru 0x03cfca (0x4d0b bytes) 000418 1786562344.967173 mapwrite 0x00f9bb thru 0x016f32 (0x7578 bytes) 000419 1786562344.983340 write 0x018c1e thru 0x02359e (0xa981 bytes) 000420 1786562344.984465 mapwrite 0x02c40b thru 0x03a323 (0xdf19 bytes) 000423 1786562345.005535 read 0x03228e thru 0x03b442 (0x91b5 bytes) 000424 1786562345.006055 write 0x0397bc thru 0x03ffff (0x6844 bytes) 000429 1786562345.008935 mapwrite 0x01966b thru 0x01ea27 (0x53bd bytes) 000430 1786562345.023123 mapread 0x01196b thru 0x01335d (0x19f3 bytes) 000432 1786562345.024300 read 0x02a235 thru 0x033b09 (0x98d5 bytes) 000439 1786562345.027424 write 0x03cdea thru 0x03ffff (0x3216 bytes) 000440 1786562345.028564 write 0x00b0b8 thru 0x00d283 (0x21cc bytes) 000441 1786562345.029181 write 0x025fa5 thru 0x032553 (0xc5af bytes) 000443 1786562345.030606 write 0x03ae8d thru 0x03ffff (0x5173 bytes) 000444 1786562345.031578 mapwrite 0x00772c thru 0x0172b8 (0xfb8d bytes) 000445 1786562345.051616 trunc from 0x040000 to 0x0162f0 (0x29d0f bytes) 000449 1786562345.060372 mapread 0x000940 thru 0x008e65 (0x8526 bytes) 000450 1786562345.061262 mapwrite 0x02fd7a thru 0x034ce0 (0x4f67 bytes) 000453 1786562345.090879 trunc from 0x034ce1 to 0x03a913 (0x5c33 bytes) 000454 1786562345.097106 trunc from 0x03a914 to 0x01e979 (0x1bf9a bytes) 000455 1786562345.103588 mapwrite 0x00618d thru 0x00c144 (0x5fb8 bytes) 000456 1786562345.122046 MIRROR_EXTEND to 2 mirrors 000457 1786562345.369825 mapread 0x00222a thru 0x007906 (0x56dd bytes) 000459 1786562345.377002 MIRROR_EXTEND to 3 mirrors 000460 1786562345.617795 write 0x029666 thru 0x0374bc (0xde57 bytes) 000461 1786562345.639327 trunc from 0x0374bd to 0x02cffd (0xa4bf bytes) 000462 1786562345.651659 write 0x031f57 thru 0x03ffff (0xe0a9 bytes) 000464 1786562345.653735 read 0x03623b thru 0x03c3bb (0x6181 bytes) 000465 1786562345.654343 write 0x013c6e thru 0x021723 (0xdab6 bytes) 000466 1786562345.659055 write 0x0360c2 thru 0x03ffff (0x9f3e bytes) 000468 1786562345.660782 trunc from 0x040000 to 0x026999 (0x19666 bytes) 000469 1786562345.666711 trunc from 0x02699a to 0x024ce2 (0x1cb7 bytes) 000470 1786562345.672643 mapwrite 0x032f3a thru 0x0375c9 (0x4690 bytes) 000471 1786562345.705190 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x375ca): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x375ca): Mirror 2: 0x6bf09774 (stale) Mirror 3: 0x6bf09774 (stale) CRC-32 checksum value for chunk [0, 0x375ca): Mirror 1: 0x8fa09cc9 Mirror 4: 0x8fa09cc9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x375ca: skipped 000472 1786562346.030937 write 0x0102ed thru 0x01ca22 (0xc736 bytes) 000473 1786562346.054108 MIRROR_RESYNC 4 mirrors 000475 1786562346.537446 write 0x02e2be thru 0x03d132 (0xee75 bytes) 000476 1786562346.559063 read 0x01eb75 thru 0x02160b (0x2a97 bytes) 000477 1786562346.562166 write 0x035523 thru 0x03d40d (0x7eeb bytes) 000478 1786562346.563261 mapwrite 0x0233e5 thru 0x032bab (0xf7c7 bytes) 000481 1786562346.594522 mapread 0x0147b0 thru 0x01a6c2 (0x5f13 bytes) 000482 1786562346.602495 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d40e): Mirror 2: 0xb3103d78 (stale) Mirror 4: 0xb3103d78 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d40e): Mirror 2: 0xb3103d78 (stale) Mirror 4: 0xb3103d78 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d40e: skipped 000484 1786562346.757093 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3d40e): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d40e): Mirror 2: 0xb3103d78 (stale) Mirror 4: 0xb3103d78 (stale) CRC-32 checksum value for chunk [0, 0x3d40e): Mirror 1: 0xbe35fa92 Mirror 5: 0xbe35fa92 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d40e: skipped 000486 1786562347.109175 mapwrite 0x02da59 thru 0x03c649 (0xebf1 bytes) 000487 1786562347.188564 read 0x0019a0 thru 0x003dec (0x244d bytes) 000488 1786562347.192184 mapread 0x00dca1 thru 0x010058 (0x23b8 bytes) 000489 1786562347.197986 mapread 0x03b4d1 thru 0x03d40d (0x1f3d bytes) 000491 1786562347.200002 mapwrite 0x02739f thru 0x02c1cf (0x4e31 bytes) 000492 1786562347.219007 write 0x00dca3 thru 0x012c64 (0x4fc2 bytes) 000495 1786562347.221317 trunc from 0x03d40e to 0x017fb5 (0x25458 bytes) 000497 1786562347.244538 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17fb6): Mirror 2: 0xd6b4c152 (stale) Mirror 5: 0xd6b4c152 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17fb6): Mirror 2: 0xd6b4c152 (stale) Mirror 5: 0xd6b4c152 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17fb6: skipped 000501 1786562347.376980 trunc from 0x017fb6 to 0x020e19 (0x8e64 bytes) 000503 1786562347.385851 write 0x02197a thru 0x025879 (0x3f00 bytes) 000504 1786562347.387116 trunc from 0x02587a to 0x01803d (0xd83c bytes) 000506 1786562347.395929 read 0x002eae thru 0x00da3c (0xab8f bytes) 000507 1786562347.400935 trunc from 0x01803e to 0x008f31 (0xf10c bytes) 000508 1786562347.409734 read 0x000a70 thru 0x001307 (0x898 bytes) 000512 1786562347.414365 write 0x01611a thru 0x01e577 (0x845e bytes) 000513 1786562347.418797 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e578): Mirror 5: 0xe524c731 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e578): Mirror 5: 0xe524c731 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e578: skipped 000518 1786562347.538091 read 0x013e8c thru 0x01a207 (0x637c bytes) 000519 1786562347.546029 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1e578): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e578): Mirror 5: 0xe524c731 (stale) CRC-32 checksum value for chunk [0, 0x1e578): Mirror 1: 0x566e915e Mirror 6: 0x566e915e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e578: skipped 000521 1786562347.921477 write 0x00800e thru 0x00d36c (0x535f bytes) 000522 1786562347.957614 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1e578): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e578): Mirror 5: 0xe524c731 (stale) Mirror 6: 0x566e915e (stale) CRC-32 checksum value for chunk [0, 0x1e578): Mirror 1: 0x7708dccb Mirror 7: 0x7708dccb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e578: skipped 000523 1786562348.420301 mapread 0x003eb3 thru 0x0059fb (0x1b49 bytes) 000527 1786562348.438152 write 0x030b65 thru 0x037b8e (0x702a bytes) 000528 1786562348.477429 trunc from 0x037b8f to 0x01046a (0x27724 bytes) 000531 1786562348.495377 trunc from 0x01046b to 0x01f243 (0xedd9 bytes) 000533 1786562348.511206 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1f244): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f244): Mirror 5: 0xcf535e36 (stale) Mirror 6: 0xfbd6c51 (stale) Mirror 7: 0x106f2c98 (stale) CRC-32 checksum value for chunk [0, 0x1f244): Mirror 1: 0xecadd180 Mirror 8: 0xecadd180 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f244: skipped 000534 1786562348.970760 trunc from 0x01f244 to 0x00a05c (0x151e7 bytes) 000535 1786562349.002734 trunc from 0x00a05d to 0x02133d (0x172e1 bytes) 000536 1786562349.017536 read 0x00a850 thru 0x00fbbf (0x5370 bytes) 000537 1786562349.022053 mapwrite 0x01fccd thru 0x02e13a (0xe46e bytes) 000539 1786562349.068777 mapwrite 0x008ced thru 0x0189d1 (0xfce5 bytes) 000540 1786562349.107633 trunc from 0x02e13b to 0x019087 (0x150b3 bytes) 000541 1786562349.119385 mapread 0x00c792 thru 0x019087 (0xc8f6 bytes) 000543 1786562349.121738 write 0x020c57 thru 0x02c32d (0xb6d7 bytes) 000544 1786562349.123417 write 0x03f7b3 thru 0x03ffff (0x84d bytes) 000545 1786562349.124676 trunc from 0x040000 to 0x01ee59 (0x211a6 bytes) 000546 1786562349.137099 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ee5a): Mirror 6: 0xd736ef0d (stale) Mirror 7: 0x82c07d85 (stale) Mirror 8: 0xf394e1a3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ee5a): Mirror 6: 0xd736ef0d (stale) Mirror 7: 0x82c07d85 (stale) Mirror 8: 0xf394e1a3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ee5a: skipped 000547 1786562349.279571 read 0x01e362 thru 0x01e871 (0x510 bytes) 000549 1786562349.284085 mapread 0x00a06e thru 0x00a7d2 (0x765 bytes) 000550 1786562349.290627 MIRROR_RESYNC 4 mirrors 000552 1786562349.863491 trunc from 0x01ee5a to 0x0371b1 (0x18358 bytes) 000557 1786562349.898025 read 0x0257ea thru 0x032059 (0xc870 bytes) 000562 1786562349.905484 read 0x01d79b thru 0x020c82 (0x34e8 bytes) truncating to largest ever: 0x3f8f1 000565 1786562349.913613 trunc from 0x0371b2 to 0x03f8f0 (0x873f bytes) 000566 1786562349.925995 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3f8f1): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f8f1): Mirror 6: 0x7d65cba0 (stale) Mirror 7: 0x7d65cba0 (stale) Mirror 8: 0x7d65cba0 (stale) CRC-32 checksum value for chunk [0, 0x3f8f1): Mirror 1: 0x7d65cba0 Mirror 9: 0x7d65cba0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f8f1: skipped 000567 1786562350.350760 trunc from 0x03f8f1 to 0x00c07a (0x33876 bytes) 000570 1786562350.387592 write 0x02826b thru 0x0357a4 (0xd53a bytes) 000571 1786562350.389084 write 0x01111a thru 0x0193a1 (0x8288 bytes) 000573 1786562350.399752 MIRROR_RESYNC 5 mirrors 000576 1786562351.149917 mapwrite 0x0207a4 thru 0x020ce1 (0x53e bytes) 000581 1786562351.213642 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x357a5): Mirror 7: 0x172b8b0e (stale) Mirror 8: 0x172b8b0e (stale) Mirror 9: 0x172b8b0e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x357a5): Mirror 7: 0x172b8b0e (stale) Mirror 8: 0x172b8b0e (stale) Mirror 9: 0x172b8b0e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x357a5: skipped 000583 1786562351.352005 mapread 0x004127 thru 0x00f301 (0xb1db bytes) 000584 1786562351.359914 trunc from 0x0357a5 to 0x02a729 (0xb07b bytes) 000586 1786562351.366779 read 0x010c98 thru 0x020582 (0xf8eb bytes) 000587 1786562351.371295 trunc from 0x02a72a to 0x0386fe (0xdfd5 bytes) 000589 1786562351.380195 trunc from 0x0386ff to 0x026bbc (0x11b42 bytes) 000591 1786562351.388641 MIRROR_RESYNC 4 mirrors 000592 1786562352.012296 write 0x01f7d2 thru 0x021391 (0x1bc0 bytes) 000593 1786562352.038077 write 0x009950 thru 0x00a71d (0xdce bytes) 000594 1786562352.043801 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26bbd): Mirror 8: 0x8e5690f7 (stale) Mirror 9: 0x8e5690f7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26bbd): Mirror 8: 0x8e5690f7 (stale) Mirror 9: 0x8e5690f7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26bbd: skipped 000595 1786562352.161452 read 0x01f282 thru 0x022925 (0x36a4 bytes) 000596 1786562352.164328 trunc from 0x026bbd to 0x017ee2 (0xecda bytes) 000599 1786562352.176165 read 0x005526 thru 0x00892b (0x3406 bytes) 000602 1786562352.183317 MIRROR_RESYNC 3 mirrors 000603 1786562352.571114 write 0x01201b thru 0x016ff7 (0x4fdd bytes) 000606 1786562352.594518 write 0x01a329 thru 0x01c7af (0x2487 bytes) 000608 1786562352.595662 mapread 0x008c16 thru 0x00eb12 (0x5efd bytes) 000609 1786562352.603577 mapwrite 0x02d708 thru 0x031cb3 (0x45ac bytes) 000611 1786562352.638043 read 0x029ca2 thru 0x031830 (0x7b8f bytes) 000614 1786562352.639336 mapwrite 0x03deae thru 0x03ffff (0x2152 bytes) 000615 1786562352.669111 trunc from 0x040000 to 0x009285 (0x36d7a bytes) 000617 1786562352.675346 read 0x001dff thru 0x009285 (0x7487 bytes) 000620 1786562352.679258 trunc from 0x009286 to 0x015042 (0xbdbd bytes) 000623 1786562352.688825 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15043): Mirror 9: 0xae70e59e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15043): Mirror 9: 0xae70e59e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15043: skipped 000625 1786562352.815926 mapread 0x00625f thru 0x008a06 (0x27a8 bytes) 000626 1786562352.825853 trunc from 0x015043 to 0x035874 (0x20832 bytes) 000627 1786562352.836593 MIRROR_RESYNC 2 mirrors 000629 1786562353.146907 read 0x01ccef thru 0x028ca4 (0xbfb6 bytes) 000630 1786562353.151302 trunc from 0x035875 to 0x00648b (0x2f3e9 bytes) 000632 1786562353.183980 read 0x00647c thru 0x00648b (0x10 bytes) 000633 1786562353.189259 mapread 0x000e2a thru 0x00648b (0x5662 bytes) 000635 1786562353.190680 write 0x02f336 thru 0x0381c2 (0x8e8d bytes) 000636 1786562353.191979 mapread 0x023f23 thru 0x02af6f (0x704d bytes) 000637 1786562353.198602 trunc from 0x0381c3 to 0x00d4e6 (0x2acdc bytes) 000638 1786562353.214529 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) 000639 1786562353.266665 write 0x03e1ee thru 0x03f06c (0xe7f bytes) 000640 1786562353.267526 trunc from 0x03f06d to 0x001c39 (0x3d433 bytes) 000641 1786562353.274523 write 0x004022 thru 0x00d638 (0x9617 bytes) 000642 1786562353.275891 mapread 0x00ce5a thru 0x00d638 (0x7df bytes) 000643 1786562353.276930 mapread 0x001091 thru 0x002cd3 (0x1c43 bytes) 000645 1786562353.283396 write 0x038bad thru 0x03ffff (0x7453 bytes) 000647 1786562353.285294 read 0x025ac6 thru 0x0266d9 (0xc14 bytes) 000648 1786562353.288480 read 0x027214 thru 0x034f99 (0xdd86 bytes) 000651 1786562353.295103 mapread 0x002607 thru 0x00e3c3 (0xbdbd bytes) 000653 1786562353.297133 mapread 0x031dc0 thru 0x03ea77 (0xccb8 bytes) 000655 1786562353.298810 mapread 0x006ed3 thru 0x007b29 (0xc57 bytes) 000656 1786562353.299592 trunc from 0x040000 to 0x002c2a (0x3d3d5 bytes) 000657 1786562353.308325 mapread 0x002af6 thru 0x002c2a (0x135 bytes) 000659 1786562353.312618 MIRROR_EXTEND to 2 mirrors 000661 1786562353.627256 MIRROR_EXTEND to 3 mirrors 000664 1786562353.873250 mapwrite 0x016ef7 thru 0x0238bd (0xc9c7 bytes) 000667 1786562353.940058 trunc from 0x0238be to 0x000a22 (0x22e9b bytes) 000668 1786562353.947342 read 0x00029e thru 0x000a22 (0x785 bytes) 000670 1786562353.952564 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xa23): Mirror 3: 0xba99d0df (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa23): Mirror 3: 0xba99d0df (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa23: skipped 000671 1786562354.047137 write 0x03abea thru 0x03cccc (0x20e3 bytes) 000678 1786562354.051889 write 0x023e92 thru 0x029a37 (0x5ba6 bytes) 000679 1786562354.056608 mapwrite 0x02daca thru 0x035d3d (0x8274 bytes) 000680 1786562354.080034 write 0x02a7a5 thru 0x02cb22 (0x237e bytes) 000682 1786562354.081514 mapwrite 0x008f5a thru 0x011e7a (0x8f21 bytes) 000683 1786562354.103464 trunc from 0x03cccd to 0x035b19 (0x71b3 bytes) 000684 1786562354.108983 mapread 0x006b69 thru 0x01194f (0xade7 bytes) 000685 1786562354.110040 write 0x00311a thru 0x00b168 (0x804f bytes) 000686 1786562354.111087 read 0x007e7a thru 0x008c6b (0xdf2 bytes) 000690 1786562354.113347 read 0x003c2e thru 0x00e99f (0xad72 bytes) 000692 1786562354.114476 mapwrite 0x00a9ea thru 0x0182d4 (0xd8eb bytes) 000693 1786562354.133824 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x35b1a): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35b1a): Mirror 3: 0x5372f7d7 (stale) CRC-32 checksum value for chunk [0, 0x35b1a): Mirror 1: 0xb8b813be Mirror 4: 0xb8b813be lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35b1a: skipped 000694 1786562354.449097 read 0x02ac01 thru 0x035b19 (0xaf19 bytes) 000695 1786562354.452402 mapread 0x00addf thru 0x00e885 (0x3aa7 bytes) 000696 1786562354.459283 write 0x0001c4 thru 0x00e6c4 (0xe501 bytes) 000697 1786562354.485022 mapread 0x00b2d5 thru 0x010536 (0x5262 bytes) 000698 1786562354.490514 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x35b1a): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35b1a): Mirror 3: 0x5372f7d7 (stale) Mirror 4: 0xb8b813be (stale) CRC-32 checksum value for chunk [0, 0x35b1a): Mirror 1: 0x54d0c28d Mirror 5: 0x54d0c28d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35b1a: skipped 000699 1786562354.822280 read 0x0003ca thru 0x00a1e8 (0x9e1f bytes) 000701 1786562354.826512 trunc from 0x035b1a to 0x00d12f (0x289ea bytes) 000702 1786562354.861570 mapwrite 0x0148b3 thru 0x0186b2 (0x3e00 bytes) 000705 1786562354.905266 read 0x005d91 thru 0x009808 (0x3a78 bytes) 000706 1786562354.909156 mapread 0x009ef9 thru 0x00dbce (0x3cd6 bytes) 000707 1786562354.910014 mapwrite 0x002a3e thru 0x00d950 (0xaf13 bytes) 000708 1786562354.927766 read 0x00c29c thru 0x012535 (0x629a bytes) 000710 1786562354.928885 mapread 0x00771a thru 0x014519 (0xce00 bytes) 000711 1786562354.929911 read 0x00b60a thru 0x0109be (0x53b5 bytes) 000712 1786562354.930361 mapread 0x00e520 thru 0x0186b2 (0xa193 bytes) 000713 1786562354.931277 mapread 0x0138f1 thru 0x0186b2 (0x4dc2 bytes) 000716 1786562354.933034 read 0x00f210 thru 0x013e5b (0x4c4c bytes) 000718 1786562354.936636 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x186b3): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x186b3): Mirror 3: 0x21bd1831 (stale) Mirror 4: 0xba127340 (stale) Mirror 5: 0x934ec2c5 (stale) CRC-32 checksum value for chunk [0, 0x186b3): Mirror 1: 0x2a806716 Mirror 6: 0x2a806716 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x186b3: skipped 000721 1786562355.346704 mapwrite 0x00624f thru 0x00efd5 (0x8d87 bytes) 000722 1786562355.440715 MIRROR_RESYNC 5 mirrors 000723 1786562355.979014 trunc from 0x0186b3 to 0x0398ab (0x211f9 bytes) 000725 1786562356.024703 read 0x01933f thru 0x028488 (0xf14a bytes) 000726 1786562356.030885 mapread 0x00c841 thru 0x015f57 (0x9717 bytes) 000730 1786562356.043464 mapwrite 0x017da2 thru 0x027572 (0xf7d1 bytes) 000731 1786562356.073401 write 0x028e95 thru 0x02b274 (0x23e0 bytes) 000732 1786562356.077117 write 0x01e816 thru 0x025dc6 (0x75b1 bytes) 000733 1786562356.078668 trunc from 0x0398ac to 0x019145 (0x20766 bytes) 000735 1786562356.098860 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x19146): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19146): Mirror 3: 0x38b401ba (stale) Mirror 4: 0x38b401ba (stale) Mirror 5: 0x38b401ba (stale) Mirror 6: 0x38b401ba (stale) CRC-32 checksum value for chunk [0, 0x19146): Mirror 1: 0x2bbaa52c Mirror 7: 0x2bbaa52c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19146: skipped 000736 1786562356.446695 write 0x027511 thru 0x029100 (0x1bf0 bytes) 000737 1786562356.476855 trunc from 0x029101 to 0x02b842 (0x2742 bytes) 000738 1786562356.493035 mapwrite 0x013949 thru 0x01a9cf (0x7087 bytes) 000740 1786562356.528803 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2b843): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b843): Mirror 3: 0x62cf63fe (stale) Mirror 4: 0x62cf63fe (stale) Mirror 5: 0x62cf63fe (stale) Mirror 6: 0x62cf63fe (stale) Mirror 7: 0xd0d027e2 (stale) CRC-32 checksum value for chunk [0, 0x2b843): Mirror 1: 0x2e887a0c Mirror 8: 0x2e887a0c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b843: skipped 000742 1786562357.008132 trunc from 0x02b843 to 0x03f3d3 (0x13b91 bytes) 000743 1786562357.059502 read 0x017fe2 thru 0x019bff (0x1c1e bytes) 000744 1786562357.062767 write 0x0373ad thru 0x03ffff (0x8c53 bytes) 000745 1786562357.066143 write 0x01ece3 thru 0x02b4aa (0xc7c8 bytes) 000746 1786562357.071942 trunc from 0x040000 to 0x02a11c (0x15ee3 bytes) 000749 1786562357.082655 read 0x006bed thru 0x008e46 (0x225a bytes) 000751 1786562357.091168 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2a11d): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a11d): Mirror 3: 0xcd7ef46b (stale) Mirror 4: 0xcd7ef46b (stale) Mirror 5: 0xcd7ef46b (stale) Mirror 6: 0xcd7ef46b (stale) Mirror 7: 0xa9062f59 (stale) Mirror 8: 0xbdc932e0 (stale) CRC-32 checksum value for chunk [0, 0x2a11d): Mirror 1: 0xce007386 Mirror 9: 0xce007386 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a11d: skipped 000752 1786562357.679793 mapwrite 0x018a01 thru 0x02774d (0xed4d bytes) 000756 1786562357.784746 read 0x017703 thru 0x018a24 (0x1322 bytes) 000757 1786562357.785779 mapwrite 0x007b6e thru 0x00c4fc (0x498f bytes) 000758 1786562357.826352 mapread 0x001e8d thru 0x00555a (0x36ce bytes) 000760 1786562357.838542 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a11d): Mirror 3: 0xcd7ef46b (stale) Mirror 5: 0xcd7ef46b (stale) Mirror 6: 0xcd7ef46b (stale) Mirror 7: 0xa9062f59 (stale) Mirror 8: 0xbdc932e0 (stale) Mirror 9: 0xce007386 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a11d): Mirror 3: 0xcd7ef46b (stale) Mirror 5: 0xcd7ef46b (stale) Mirror 6: 0xcd7ef46b (stale) Mirror 7: 0xa9062f59 (stale) Mirror 8: 0xbdc932e0 (stale) Mirror 9: 0xce007386 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a11d: skipped 000761 1786562358.122202 write 0x00e30c thru 0x0148b6 (0x65ab bytes) 000762 1786562358.127200 trunc from 0x02a11d to 0x035b7c (0xba60 bytes) 000764 1786562358.143203 write 0x01f4a8 thru 0x02c148 (0xcca1 bytes) 000765 1786562358.149899 write 0x02a0df thru 0x02fa15 (0x5937 bytes) 000769 1786562358.156480 write 0x020631 thru 0x02a5a9 (0x9f79 bytes) 000771 1786562358.159249 mapwrite 0x033163 thru 0x0331b6 (0x54 bytes) 000772 1786562358.192906 write 0x02054a thru 0x02e7fa (0xe2b1 bytes) 000773 1786562358.199833 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x35b7d): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35b7d): Mirror 3: 0x6fcc624a (stale) Mirror 5: 0x6fcc624a (stale) Mirror 6: 0x6fcc624a (stale) Mirror 7: 0xc5ec1551 (stale) Mirror 8: 0x57eb3ae2 (stale) Mirror 9: 0xb7194eaf (stale) CRC-32 checksum value for chunk [0, 0x35b7d): Mirror 1: 0x618c7881 Mirror 10: 0x618c7881 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35b7d: skipped 000774 1786562358.817316 write 0x02ed97 thru 0x036c28 (0x7e92 bytes) 000777 1786562358.848617 mapwrite 0x008aa4 thru 0x0098ce (0xe2b bytes) 000778 1786562358.884488 read 0x00a647 thru 0x010ffe (0x69b8 bytes) 000779 1786562358.892538 read 0x011a5b thru 0x01fdd8 (0xe37e bytes) 000781 1786562358.900826 read 0x00d28a thru 0x01660c (0x9383 bytes) 000782 1786562358.901721 read 0x01de8a thru 0x02bece (0xe045 bytes) 000783 1786562358.905749 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x36c29): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36c29): Mirror 3: 0xb23556fa (stale) Mirror 5: 0xb23556fa (stale) Mirror 6: 0xb23556fa (stale) Mirror 7: 0xd3270b66 (stale) Mirror 8: 0x35d50bd4 (stale) Mirror 9: 0x3a6bdc02 (stale) Mirror 10: 0x9462266 (stale) CRC-32 checksum value for chunk [0, 0x36c29): Mirror 1: 0x1122c6d2 Mirror 11: 0x1122c6d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36c29: skipped 000784 1786562359.424866 write 0x020045 thru 0x02ac0e (0xabca bytes) 000785 1786562359.465121 MIRROR_RESYNC 9 mirrors 000786 1786562360.294071 read 0x030d6f thru 0x03437e (0x3610 bytes) 000787 1786562360.298915 trunc from 0x036c29 to 0x012c4c (0x23fdc bytes) 000788 1786562360.342153 trunc from 0x012c4d to 0x00b1a9 (0x7aa3 bytes) 000789 1786562360.355606 mapwrite 0x02088b thru 0x0209e2 (0x158 bytes) 000790 1786562360.403270 read 0x01a3d3 thru 0x0206d1 (0x62ff bytes) 000791 1786562360.413841 mapwrite 0x01791d thru 0x01cc2c (0x5310 bytes) 000792 1786562360.453027 mapwrite 0x00b824 thru 0x00d10b (0x18e8 bytes) 000794 1786562360.487349 trunc from 0x0209e3 to 0x0158ec (0xb0f6 bytes) 000795 1786562360.496020 read 0x005919 thru 0x00ce1e (0x7506 bytes) 000796 1786562360.496870 mapwrite 0x03f436 thru 0x03ffff (0xbca bytes) 000799 1786562360.546235 mapread 0x00ec09 thru 0x01940e (0xa806 bytes) 000800 1786562360.547562 read 0x020cc0 thru 0x0276e9 (0x6a2a bytes) 000805 1786562360.550260 read 0x038a88 thru 0x039340 (0x8b9 bytes) 000807 1786562360.551297 mapwrite 0x02a09d thru 0x037732 (0xd696 bytes) 000808 1786562360.586964 trunc from 0x040000 to 0x03970c (0x68f3 bytes) 000809 1786562360.599089 mapwrite 0x03e9e0 thru 0x03ffff (0x1620 bytes) 000810 1786562360.653073 read 0x00026a thru 0x006bca (0x6961 bytes) 000811 1786562360.653855 trunc from 0x040000 to 0x0071a1 (0x38e5e bytes) 000814 1786562360.671531 mapread 0x0059af thru 0x0071a1 (0x17f3 bytes) 000817 1786562360.674299 mapwrite 0x0252e4 thru 0x02f732 (0xa44f bytes) 000818 1786562360.730955 mapread 0x02aab2 thru 0x02c5dd (0x1b2c bytes) 000819 1786562360.732129 read 0x0075ff thru 0x015906 (0xe308 bytes) 000820 1786562360.734350 read 0x0296f3 thru 0x02f732 (0x6040 bytes) 000822 1786562360.736139 mapread 0x000ccb thru 0x00c0fc (0xb432 bytes) 000823 1786562360.738038 mapwrite 0x03bb87 thru 0x03fc79 (0x40f3 bytes) 000824 1786562360.813113 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fc7a): Mirror 3: 0x556bc772 (stale) Mirror 5: 0x556bc772 (stale) Mirror 7: 0x556bc772 (stale) Mirror 8: 0x556bc772 (stale) Mirror 9: 0x556bc772 (stale) Mirror 10: 0x556bc772 (stale) Mirror 11: 0x556bc772 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fc7a): Mirror 3: 0x556bc772 (stale) Mirror 5: 0x556bc772 (stale) Mirror 7: 0x556bc772 (stale) Mirror 8: 0x556bc772 (stale) Mirror 9: 0x556bc772 (stale) Mirror 10: 0x556bc772 (stale) Mirror 11: 0x556bc772 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fc7a: skipped 000825 1786562361.241121 mapread 0x03069b thru 0x035624 (0x4f8a bytes) 000826 1786562361.274121 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fc7a): Mirror 3: 0x556bc772 (stale) Mirror 7: 0x556bc772 (stale) Mirror 8: 0x556bc772 (stale) Mirror 9: 0x556bc772 (stale) Mirror 10: 0x556bc772 (stale) Mirror 11: 0x556bc772 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fc7a): Mirror 3: 0x556bc772 (stale) Mirror 7: 0x556bc772 (stale) Mirror 8: 0x556bc772 (stale) Mirror 9: 0x556bc772 (stale) Mirror 10: 0x556bc772 (stale) Mirror 11: 0x556bc772 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fc7a: skipped 000827 1786562361.746534 read 0x03b757 thru 0x03fc79 (0x4523 bytes) 000828 1786562361.756344 mapwrite 0x03b96c thru 0x03ffff (0x4694 bytes) 000829 1786562361.839310 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfdd3aff0 (stale) Mirror 7: 0xfdd3aff0 (stale) Mirror 8: 0xfdd3aff0 (stale) Mirror 9: 0xfdd3aff0 (stale) Mirror 10: 0xfdd3aff0 (stale) Mirror 11: 0xfdd3aff0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf7ec3682 Mirror 12: 0xf7ec3682 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000831 1786562363.065518 mapread 0x02b1cf thru 0x02b5c4 (0x3f6 bytes) 000833 1786562363.118820 trunc from 0x040000 to 0x00fdd2 (0x3022d bytes) 000834 1786562363.257743 read 0x00b4be thru 0x00fdd2 (0x4915 bytes) 000836 1786562363.277803 write 0x01c5d5 thru 0x024b43 (0x856f bytes) 000837 1786562363.282484 mapread 0x019568 thru 0x019825 (0x2be bytes) 000839 1786562363.300360 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24b44): Mirror 3: 0x6593e929 (stale) Mirror 7: 0x6593e929 (stale) Mirror 9: 0x6593e929 (stale) Mirror 10: 0x6593e929 (stale) Mirror 11: 0x6593e929 (stale) Mirror 12: 0x54d42ccc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24b44): Mirror 3: 0x6593e929 (stale) Mirror 7: 0x6593e929 (stale) Mirror 9: 0x6593e929 (stale) Mirror 10: 0x6593e929 (stale) Mirror 11: 0x6593e929 (stale) Mirror 12: 0x54d42ccc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24b44: skipped 000842 1786562363.773172 mapread 0x007957 thru 0x011860 (0x9f0a bytes) 000843 1786562363.804491 write 0x0257c2 thru 0x033c47 (0xe486 bytes) 000846 1786562363.811395 trunc from 0x033c48 to 0x026de3 (0xce64 bytes) 000848 1786562363.841791 write 0x02aab0 thru 0x02ceeb (0x243c bytes) 000851 1786562363.846424 mapwrite 0x012879 thru 0x014cbd (0x2445 bytes) 000852 1786562363.911501 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ceec): Mirror 3: 0xc3eb30ad (stale) Mirror 7: 0xc3eb30ad (stale) Mirror 9: 0xc3eb30ad (stale) Mirror 10: 0xc3eb30ad (stale) Mirror 11: 0xc3eb30ad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ceec): Mirror 3: 0xc3eb30ad (stale) Mirror 7: 0xc3eb30ad (stale) Mirror 9: 0xc3eb30ad (stale) Mirror 10: 0xc3eb30ad (stale) Mirror 11: 0xc3eb30ad (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ceec: skipped 000853 1786562364.284647 trunc from 0x02ceec to 0x00398d (0x2955e bytes) 000855 1786562364.302099 trunc from 0x00398e to 0x02411c (0x2078f bytes) 000858 1786562364.333049 MIRROR_RESYNC 6 mirrors 000860 1786562365.998198 mapread 0x004f64 thru 0x006a11 (0x1aae bytes) 000861 1786562366.028438 mapread 0x01cddf thru 0x01f13b (0x235d bytes) 000862 1786562366.041046 mapwrite 0x038ab1 thru 0x03c443 (0x3993 bytes) 000864 1786562366.216335 write 0x036ba8 thru 0x03dbfc (0x7055 bytes) 000865 1786562366.218012 write 0x0020df thru 0x00620b (0x412d bytes) 000871 1786562366.242193 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dbfd): Mirror 3: 0x18046381 (stale) Mirror 9: 0x18046381 (stale) Mirror 10: 0x18046381 (stale) Mirror 11: 0x18046381 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dbfd): Mirror 3: 0x18046381 (stale) Mirror 9: 0x18046381 (stale) Mirror 10: 0x18046381 (stale) Mirror 11: 0x18046381 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dbfd: skipped 000872 1786562366.847835 write 0x021088 thru 0x02839a (0x7313 bytes) 000873 1786562366.868742 mapread 0x00cf78 thru 0x018a56 (0xbadf bytes) 000875 1786562366.919637 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3dbfd): Mirror 3: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dbfd): Mirror 3: 0x18046381 (stale) Mirror 9: 0x18046381 (stale) Mirror 10: 0x18046381 (stale) Mirror 11: 0x18046381 (stale) CRC-32 checksum value for chunk [0, 0x3dbfd): Mirror 1: 0x537fc30 Mirror 12: 0x537fc30 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dbfd: skipped 000879 1786562368.840930 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dbfd): Mirror 3: 0x18046381 (stale) Mirror 9: 0x18046381 (stale) Mirror 11: 0x18046381 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dbfd): Mirror 3: 0x18046381 (stale) Mirror 9: 0x18046381 (stale) Mirror 11: 0x18046381 (stale) CRC-32 checksum value for chunk [0, 0x3dbfd): Mirror 1: 0x537fc30 Mirror 12: 0x537fc30 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dbfd: skipped 000881 1786562370.601612 trunc from 0x03dbfd to 0x013de5 (0x29e17 bytes) 000882 1786562370.827698 trunc from 0x013de6 to 0x01e493 (0xa6ae bytes) 000884 1786562370.858021 mapwrite 0x0331b3 thru 0x03ffff (0xce4d bytes) 000885 1786562371.000505 MIRROR_RESYNC 5 mirrors 000887 1786562372.444155 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) 000888 1786562373.047663 read 0x0181e4 thru 0x027110 (0xef2d bytes) 000889 1786562373.059999 mapwrite 0x036d4c thru 0x03916e (0x2423 bytes) 000891 1786562373.250532 write 0x010ef5 thru 0x01cda5 (0xbeb1 bytes) 000893 1786562373.265671 read 0x0344da thru 0x03d5c5 (0x90ec bytes) 000894 1786562373.266890 write 0x01136a thru 0x02072d (0xf3c4 bytes) 000895 1786562373.296873 MIRROR_RESYNC 4 mirrors 000897 1786562375.355650 mapread 0x00b236 thru 0x010655 (0x5420 bytes) 000901 1786562375.385489 write 0x01a188 thru 0x01faa5 (0x591e bytes) 000902 1786562375.571585 mapread 0x024362 thru 0x02bbe9 (0x7888 bytes) 000903 1786562375.603403 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xafc4f196 (stale) Mirror 12: 0xafc4f196 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xafc4f196 (stale) Mirror 12: 0xafc4f196 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000904 1786562376.214176 trunc from 0x040000 to 0x010d42 (0x2f2bd bytes) 000905 1786562376.253556 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10d43): Mirror 12: 0xd496f7bd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10d43): Mirror 12: 0xd496f7bd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10d43: skipped 000906 1786562376.618472 write 0x00b5e5 thru 0x010f79 (0x5995 bytes) 000907 1786562376.634917 mapread 0x005612 thru 0x00ff1e (0xa90d bytes) 000908 1786562376.657586 mapread 0x004a09 thru 0x00c102 (0x76fa bytes) 000911 1786562376.662613 trunc from 0x010f7a to 0x032941 (0x219c8 bytes) 000912 1786562376.687970 read 0x01ae3d thru 0x01fb00 (0x4cc4 bytes) 000915 1786562376.697787 mapread 0x01efe1 thru 0x021993 (0x29b3 bytes) 000917 1786562376.707658 mapread 0x008e29 thru 0x016c46 (0xde1e bytes) 000918 1786562376.718957 mapread 0x020974 thru 0x02c40c (0xba99 bytes) 000919 1786562376.731028 read 0x0121e7 thru 0x01464f (0x2469 bytes) 000920 1786562376.731963 mapwrite 0x0079d1 thru 0x0170bc (0xf6ec bytes) 000925 1786562376.814633 trunc from 0x032942 to 0x0031a8 (0x2f799 bytes) 000927 1786562376.846528 read 0x0003b4 thru 0x0031a8 (0x2df5 bytes) 000928 1786562376.858439 write 0x02bbb1 thru 0x0370de (0xb52e bytes) 000929 1786562376.867972 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x490:0x0]: Invalid argument (22) 000930 1786562377.086725 read 0x0163c9 thru 0x01989e (0x34d6 bytes) 000932 1786562377.109402 trunc from 0x0370df to 0x038ae3 (0x1a05 bytes) 000935 1786562377.140971 write 0x013fc4 thru 0x01ab73 (0x6bb0 bytes) 000939 1786562377.158962 write 0x015fed thru 0x025d04 (0xfd18 bytes) 000940 1786562377.167937 read 0x001fa5 thru 0x00b4f0 (0x954c bytes) 000942 1786562377.183881 mapread 0x0089da thru 0x00b40e (0x2a35 bytes) 000943 1786562377.186092 write 0x011de7 thru 0x01b0f7 (0x9311 bytes) 000944 1786562377.189787 read 0x01791d thru 0x01ae4f (0x3533 bytes) 000946 1786562377.195484 trunc from 0x038ae4 to 0x01e74e (0x1a395 bytes) 000947 1786562377.218014 mapwrite 0x037f8e thru 0x03ffff (0x8072 bytes) 000948 1786562377.328642 read 0x0141eb thru 0x014289 (0x9f bytes) 000949 1786562377.330541 mapwrite 0x0299f5 thru 0x039932 (0xff3e bytes) 000951 1786562377.403631 read 0x01978c thru 0x01bd43 (0x25b8 bytes) 000954 1786562377.407122 mapread 0x034c99 thru 0x034e6b (0x1d3 bytes) 000956 1786562377.408948 write 0x038406 thru 0x03ffff (0x7bfa bytes) 000959 1786562377.415661 mapwrite 0x03d8c7 thru 0x03ffff (0x2739 bytes) 000960 1786562377.468453 trunc from 0x040000 to 0x03d543 (0x2abc bytes) 000962 1786562377.496715 mapwrite 0x016580 thru 0x0248b5 (0xe336 bytes) 000963 1786562377.589820 mapread 0x005877 thru 0x00e9a7 (0x9131 bytes) 000965 1786562377.595873 mapread 0x031ebf thru 0x033a3a (0x1b7c bytes) 000966 1786562377.598500 mapread 0x023fcc thru 0x0269f0 (0x2a25 bytes) 000968 1786562377.606086 mapwrite 0x0026bf thru 0x005bdb (0x351d bytes) 000970 1786562377.694629 trunc from 0x03d544 to 0x014749 (0x28dfa bytes) 000972 1786562377.736742 mapread 0x009229 thru 0x012cab (0x9a83 bytes) 000973 1786562377.757051 write 0x03a1b1 thru 0x03ffff (0x5e4f bytes) 000974 1786562377.758424 write 0x022662 thru 0x0235f9 (0xf98 bytes) 000980 1786562377.800099 mapwrite 0x033edf thru 0x036f96 (0x30b8 bytes) 000982 1786562377.915568 mapwrite 0x01a17a thru 0x01a448 (0x2cf bytes) 000985 1786562378.019201 mapread 0x03d4de thru 0x03fcd7 (0x27fa bytes) 000987 1786562378.028774 read 0x02514a thru 0x02bbc1 (0x6a78 bytes) 000988 1786562378.030956 trunc from 0x040000 to 0x038f2e (0x70d1 bytes) 000990 1786562378.065264 mapwrite 0x01537c thru 0x016490 (0x1115 bytes) 000991 1786562378.153177 read 0x034bc9 thru 0x038b76 (0x3fae bytes) 000993 1786562378.161649 read 0x025230 thru 0x0272c1 (0x2092 bytes) 000995 1786562378.163507 mapwrite 0x005012 thru 0x006c38 (0x1c27 bytes) 000996 1786562378.261803 write 0x009a00 thru 0x013d41 (0xa342 bytes) All operations completed A-OK! PASS 70a (50s) == sanity-flr test 71: check mirror extend parent fid ==== 15:19:45 (1786562385) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 15:19:49 (1786562389) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x706:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x10004 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x722:0x0] } lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20007 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.353081 s, 5.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x706:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x707:0x0] } lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x10004 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x722:0x0] } lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20007 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209543 s, 5.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x706:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x707:0x0] } lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x723:0x0] } lcme_id: 0x10004 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x722:0x0] } lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20007 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x706:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x707:0x0] } lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x723:0x0] } lcme_id: 0x10004 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x722:0x0] } lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x708:0x0] } lcme_id: 0x20007 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 15:20:00 (1786562400) Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre2 Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 4297184 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8686432 .. failed Extending file size to 6619424 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8162304 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8784288 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9024800 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9108672 .. Waiting 61952 61953 61954 61956 61959 done 192.168.204.129@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg429-client.virtnet /mnt/lustre2 (opts:) 192.168.204.129@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg429-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (77s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:21:17 (1786562477) Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre2 Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7287776 .. Extending file size to 3045120 .. failed Extending file size to 3336640 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3513120 .. Extending file size to 8399744 .. Extending file size to 6553088 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6611744 .. Extending file size to 6664576 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9376928 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 63373 63375 63377 63378 63379 done 192.168.204.129@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg429-client.virtnet /mnt/lustre2 (opts:) 192.168.204.129@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg429-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (87s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 15:22:44 (1786562564) Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3200 2205184 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 64512 7474176 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3200 2205184 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 64512 7474176 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.13986 s, 15.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.508932 s, 4.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x25037a39 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.129@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg429-client.virtnet /mnt/lustre2 (opts:) PASS 200c (23s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 15:23:09 (1786562589) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.17513 s, 12.0 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 15:23:17 (1786562597) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4dd:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 0x20002 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x739:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 0x20002 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x739:0x0] } lcme_id: 0x30001 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x73a:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x71d:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 15:23:25 (1786562605) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.169948 s, 12.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x73b:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x71e:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x71f:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 15:23:33 (1786562613) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.266704 s, 7.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x73c:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x73d:0x0] } lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x720:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x73c:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x73d:0x0] } lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x720:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x721:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 15:23:42 (1786562622) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.164396 s, 12.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x73e:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x722:0x0] } lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x73e:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x722:0x0] } lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x723:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:24:16 (1786562656) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg429-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164089 s, 6.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x73f:0x0] } lcme_id: 0x10005 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x725:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x724:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x73f:0x0] } lcme_id: 0x10005 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x725:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x724:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x740:0x0] } PASS 204d (57s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:25:13 (1786562713) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg429-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x741:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x742:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.244007 s, 8.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x741:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x742:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x726:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x741:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x742:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x726:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (67s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:26:20 (1786562780) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg429-server mds-ost sync done. Creating new pool test_204f oleg429-server: Pool lustre.test_204f created Adding targets to pool oleg429-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg429-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Updated after 2s: want 'lustre-OST0000_UUID lustre-OST0001_UUID ' got 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.377831 s, 5.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x743:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x727:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x743:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x727:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg429-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg429-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg429-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 15:27:41 (1786562861) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 15:27:48 (1786562868) lfs mirror mirror: Invalid layout: Wrong flags error: lfs extend: /mnt/lustre/f205b.sanity-flr: cannot create volatile file: Invalid argument sanity-flr test_205b: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs mirror extend -N --flags=nocompr /mnt/lustre/f205b.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:4398:test_205b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:4415:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_205b.*.1786562872.log rsync: chown "/tmp/testlogs/.sanity-flr.test_205b.debug_log.oleg429-server.1786562872.log.KpLjww" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_205b.dmesg.oleg429-server.1786562872.log.9U8GCT" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 15:28:06 (1786562886) oleg429-server: Pool lustre.test_206 created Adding targets to pool oleg429-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg429-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg429-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg429-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg429-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 15:28:32 (1786562912) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989994 s, 10.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533356 s, 19.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x4411a39e Mirror 3: 0x4411a39e PASS 207 (14s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 15:28:45 (1786562925) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 15:28:50 (1786562930) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 15:28:54 (1786562934) Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.300651 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2610000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1190000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1190000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1190000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2510000 Read this (4 bytes): BBBB 192.168.204.129@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg429-client.virtnet /mnt/lustre2 (opts:) PASS 209a (45s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 15:29:39 (1786562979) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0178524 s, 459 kB/s the very first read read with warm cache PASS 209b (10s) == sanity-flr test 210a: handle broken mirrored lovea ==== 15:29:49 (1786562989) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103132 s, 10.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x500 lmm_fid: [0x200000404:0x500:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 0x20001 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x500 lmm_fid: [0x200000404:0x500:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x731:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835200 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-12 15:29:51.000000000 -0400 Modify: 2026-08-12 15:29:51.000000000 -0400 Change: 2026-08-12 15:29:53.000000000 -0400 Birth: 2026-08-12 15:29:52.000000000 -0400 PASS 210a (13s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 15:30:02 (1786563002) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (5s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 15:30:07 (1786563007) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 1.36795 s, 29.9 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0757621 s, 54.1 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 12 15:30 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x502:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 12 15:30 /mnt/lustre/f211.sanity-flr PASS 211 (14s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 15:30:21 (1786563021) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 15:30:30 (1786563030) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1335 sec =========== 15:30:41 (1786563041) sanity-flr: FAIL: test_0m failed to call extend /mnt/lustre/d0m.sanity-flr/f0m.sanity-flr sanity-flr: FAIL: test_205b /home/green/git/lustre-release/lustre/utils/lfs mirror extend -N --flags=nocompr /mnt/lustre/f205b.sanity-flr failed === sanity-flr: start cleanup 15:30:43 (1786563043) === === sanity-flr: finish cleanup 15:30:50 (1786563050) ===