-----============= acceptance-small: sanity-flr ============----- Thu May 1 18:33:40 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg422-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 18:33:49 (1746138829) === oleg422-client.virtnet: executing check_config_client /mnt/lustre oleg422-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg422-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b6301000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6301000.idle_timeout=debug disable quota as required oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 18:34:05 (1746138845) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7650] == sanity-flr test 0a: lfs mirror create with -N option == 18:34:06 (1746138846) PASS 0a (7s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 18:34:13 (1746138853) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 18:34:14 (1746138854) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 18:34:16 (1746138856) PASS 0d (10s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 18:34:26 (1746138866) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 18:34:28 (1746138868) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 18:34:29 (1746138869) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror mirror: fail to pread 0-4096 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 0-4096 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/f0g.sanity-flr': Invalid argument (22) sanity-flr test_0g: @@@@@@ FAIL: error resync-ing file '/mnt/lustre/f0g.sanity-flr' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:740:test_0g() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:751:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_0g.*.1746138872.log rsync: chown "/tmp/testlogs/.sanity-flr.test_0g.debug_log.oleg422-server.1746138872.log.Do7aky" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_0g.dmesg.oleg422-server.1746138872.log.FfcVtX" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 0g (5s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 18:34:34 (1746138874) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead lfs mirror mirror: fail to pread 0-4096 of mirror 2: Invalid argument (22) lfs mirror mirror: error reading bytes 0-4096 of mirror 2: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/d0h.sanity-flr/f0h.sanity-flr': Invalid argument (22) sanity-flr test_0h: @@@@@@ FAIL: error resync-ing file '/mnt/lustre/d0h.sanity-flr/f0h.sanity-flr' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:793:test_0h() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:831:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_0h.*.1746138876.log rsync: chown "/tmp/testlogs/.sanity-flr.test_0h.debug_log.oleg422-server.1746138876.log.NQSwhG" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_0h.dmesg.oleg422-server.1746138876.log.1W3vte" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 18:34:38 (1746138878) lfs mirror mirror: fail to pread 0-4096 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 0-4096 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/f0j.sanity-flr': Invalid argument (22) sanity-flr test_0j: @@@@@@ FAIL: resync /mnt/lustre/f0j.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:837:test_0j() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:849:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_0j.*.1746138880.log rsync: chown "/tmp/testlogs/.sanity-flr.test_0j.debug_log.oleg422-server.1746138880.log.UakUFa" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_0j.dmesg.oleg422-server.1746138880.log.CHpERR" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 18:34:41 (1746138881) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 18:34:43 (1746138883) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 18:34:47 (1746138887) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 18:34:50 (1746138890) PASS 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 18:34:54 (1746138894) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 18:34:57 (1746138897) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (4s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 18:35:03 (1746138903) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 18:35:06 (1746138906) 20+0 records in 20+0 records out 20971520 bytes (21 MB) copied, 0.763037 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.044143 s, 23.8 MB/s PASS 21 (4s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 18:35:10 (1746138910) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.162199 s, 77.6 MB/s PASS 22 (5s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 18:35:15 (1746138915) Stopping /mnt/lustre-ost1 (opts:) on oleg422-server oleg422-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6301000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6301000.ost_server_uuid in DISCONN state after 0 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg422-server: oleg422-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg422-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg422-server oleg422-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6301000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6301000.ost_server_uuid in DISCONN state after 0 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg422-server: oleg422-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg422-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (33s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 18:35:49 (1746138949) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB) copied, 0.166939 s, 37.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg422-server oleg422-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6301000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6301000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg422-server: oleg422-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg422-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (15s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 18:36:05 (1746138965) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg422-server oleg422-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6301000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6301000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg422-server: oleg422-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg422-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg422-server oleg422-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6301000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6301000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg422-server: oleg422-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg422-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (37s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 18:36:42 (1746139002) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 1.16085 s, 45.2 MB/s -rw-r--r-- 1 root root 52428800 May 1 18:36 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.406s user 0m0.000s sys 0m0.130s Stopping /mnt/lustre-ost1 (opts:) on oleg422-server oleg422-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6301000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6301000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.424s user 0m0.000s sys 0m0.130s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg422-server: oleg422-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg422-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg422-server oleg422-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6301000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6301000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.516s user 0m0.000s sys 0m0.132s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg422-server: oleg422-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg422-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg422-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b6301000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6301000.ost_server_uuid in FULL state after 0 sec PASS 33b (37s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 18:37:19 (1746139039) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 18:37:21 (1746139041) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 18:37:22 (1746139042) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 18:37:23 (1746139043) PASS 35 (4s) == sanity-flr test 36a: write to mirrored files ========== 18:37:27 (1746139047) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.286021 s, 44.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (5s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 18:37:32 (1746139052) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0231465 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0210769 s, 49.8 MB/s PASS 36b (4s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 18:37:36 (1746139056) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 18:37:40 (1746139060) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00111807 s, 916 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00200318 s, 511 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr lfs mirror mirror: fail to pread 0-4096 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 0-4096 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/f36d.sanity-flr': Invalid argument (22) sanity-flr test_36d: @@@@@@ FAIL: mirror resync /mnt/lustre/f36d.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:1643:test_36d() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:1674:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_36d.*.1746139063.log rsync: chown "/tmp/testlogs/.sanity-flr.test_36d.debug_log.oleg422-server.1746139063.log.ofKCrV" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_36d.dmesg.oleg422-server.1746139063.log.GISAat" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 36d (7s) == sanity-flr test 37: mirror I/O API verification ======= 18:37:47 (1746139067) create test files with size 15743882 .. 565cb6ec3c19e074cd885f69b9136fd8 - d1c6068897f6ff6e78816e9e30be80c3 - ef175a9eaab0bacb08bee4d6a9c32d61 - 3 Verifying mirror read .. lfs mirror mirror: fail to pread 12582912-4194304 of mirror 1: Invalid argument (22) lfs read: fail to read data from mirror 1: Invalid argument sanity-flr test_37: @@@@@@ FAIL: 1: mismatch: \'565cb6ec3c19e074cd885f69b9136fd8 -\' vs. \'b911b4caf86ade3ff3dcc943d71cbe47 -\' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:1730:test_37() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:1759:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_37.*.1746139072.log rsync: chown "/tmp/testlogs/.sanity-flr.test_37.debug_log.oleg422-server.1746139072.log.UTrgp4" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_37.dmesg.oleg422-server.1746139072.log.MqKAbX" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 37 (8s) == sanity-flr test 38: resync ============================ 18:37:55 (1746139075) 4 mirror IDs: 1 2 3 4 lfs mirror mirror: fail to pread 4194304-4194304 of mirror 1: Invalid argument (22) lfs read: fail to read data from mirror 1: Invalid argument lfs mirror mirror: fail to pread 4194304-4194304 of mirror 4: Invalid argument (22) lfs read: fail to read data from mirror 4: Invalid argument sanity-flr test_38: @@@@@@ FAIL: valid mirror doesn't exist Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:1811:test_38() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:1839:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_38.*.1746139079.log rsync: chown "/tmp/testlogs/.sanity-flr.test_38.debug_log.oleg422-server.1746139079.log.2Dyx5S" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_38.dmesg.oleg422-server.1746139079.log.3vHRW2" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 38 (7s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 18:38:02 (1746139082) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 18:38:05 (1746139085) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0660101 s, 15.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x11d lmm_fid: 0x200000402:0x11d:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x35:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x11d lmm_fid: 0x200000402:0x11d:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x36:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x11d lmm_fid: 0x200000402:0x11d:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x11d lmm_fid: 0x200000402:0x11d:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x39:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x11d lmm_fid: 0x200000402:0x11d:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x11d lmm_fid: 0x200000402:0x11d:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x11d lmm_fid: 0x200000402:0x11d:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0170705 s, 61.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x11f lmm_fid: 0x200000402:0x11f:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x11f lmm_fid: 0x200000402:0x11f:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x11f lmm_fid: 0x200000402:0x11f:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x11f lmm_fid: 0x200000402:0x11f:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x37:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x11f lmm_fid: 0x200000402:0x11f:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x11f lmm_fid: 0x200000402:0x11f:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x11f lmm_fid: 0x200000402:0x11f:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (5s) == sanity-flr test 41: lfs mirror resync check =========== 18:38:10 (1746139090) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0317965 s, 66.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.157786 s, 26.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0713428 s, 29.4 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (5s) == sanity-flr test 42: lfs mirror verify ================= 18:38:15 (1746139095) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 18:38:16 (1746139096) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 18:38:18 (1746139098) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.019058 s, 55.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 18:38:22 (1746139102) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.181495 s, 17.3 MB/s PASS 44a (5s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 18:38:27 (1746139107) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x13:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x13:0x0] } split mirror_id 1 id 65537 sanity-flr test_44b: @@@@@@ FAIL: split --mirror-id=1 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr should succeed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:2290:test_44b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:2308:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_44b.*.1746139109.log rsync: chown "/tmp/testlogs/.sanity-flr.test_44b.debug_log.oleg422-server.1746139109.log.PTwa6I" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_44b.dmesg.oleg422-server.1746139109.log.KCbpH8" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 18:38:32 (1746139112) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.14475 s, 72.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x12d:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg422-client.virtnet /mnt/lustre (opts:) Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 18:38:44 (1746139124) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff88012fd27000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0941599 s, 111 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg422-client.virtnet /mnt/lustre (opts:) Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800a98b6800.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 18:38:49 (1746139129) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (15s) == sanity-flr test 46: Verify setstripe --copy option ==== 18:39:04 (1746139144) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (14s) == sanity-flr test 47: Verify mirror obj alloc =========== 18:39:18 (1746139158) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 18:39:19 (1746139159) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.11557 s, 27.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0697409 s, 45.1 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 363097e1359315cf361b5ae49757aeae - mirror 1 checksum: 2be7dccabd046d2ea96c157a7b42b1dd - mirror 2 checksum: 363097e1359315cf361b5ae49757aeae - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu May 1 18:39:21 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0xb lmm_fid: [0x200000404:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu May 1 18:39:21 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0xb lmm_fid: [0x200000404:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 363097e1359315cf361b5ae49757aeae - mirror 1 checksum: 2be7dccabd046d2ea96c157a7b42b1dd - mirror 2 checksum: 2be7dccabd046d2ea96c157a7b42b1dd - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu May 1 18:39:21 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0xb lmm_fid: [0x200000404:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu May 1 18:39:21 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0xb lmm_fid: [0x200000404:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } PASS 48 (4s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 18:39:23 (1746139163) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0140829 s, 74.5 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (3s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 18:39:26 (1746139166) Failing mds1 on oleg422-server Stopping /mnt/lustre-mds1 (opts:) on oleg422-server 18:39:35 (1746139175) shut down facet: mds1 facet_host: oleg422-server facet_failover_host: oleg422-server Failover mds1 to oleg422-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-MDT0000 18:39:51 (1746139191) targets are mounted 18:39:51 (1746139191) facet_failover done oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x11:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0136419 s, 76.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } PASS 50A (34s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 18:40:00 (1746139200) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0209443 s, 978 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00680238 s, 3.0 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror lfs mirror mirror: fail to pread 0-4194304 of mirror 2: Invalid argument (22) lfs read: fail to read data from mirror 2: Invalid argument sanity-flr test_50a: @@@@@@ FAIL: data mismatch: \'e77f515b491f4db2743c19af08739cfc -\' vs. \'d41d8cd98f00b204e9800998ecf8427e -\' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:2846:test_50a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:2895:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_50a.*.1746139202.log rsync: chown "/tmp/testlogs/.sanity-flr.test_50a.debug_log.oleg422-server.1746139202.log.0cgSoR" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_50a.dmesg.oleg422-server.1746139202.log.Q5ADzd" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 18:40:04 (1746139204) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0286355 s, 715 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 lfs mirror mirror: fail to pread 0-4194304 of mirror 2: Invalid argument (22) lfs read: fail to read data from mirror 2: Invalid argument sanity-flr test_50b: @@@@@@ FAIL: data mismatch: \'7b63b54edd900422cd2e120f663a6aae -\' vs. \'d41d8cd98f00b204e9800998ecf8427e -\' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:2939:test_50b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:2983:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_50b.*.1746139207.log rsync: chown "/tmp/testlogs/.sanity-flr.test_50b.debug_log.oleg422-server.1746139207.log.VZV8Kf" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_50b.dmesg.oleg422-server.1746139207.log.y5OtgM" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 18:40:08 (1746139208) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00847354 s, 1.9 MB/s PASS 50c (5s) == sanity-flr test 50d: mirror rsync keep holes ========== 18:40:13 (1746139213) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0345895 s, 592 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 lfs mirror mirror: fail to pread 1048576-20480 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 1048576-20480 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/d50d.sanity-flr/f50d.sanity-flr': Invalid argument (22) lfs mirror mirror: fail to pread 0-4194304 of mirror 2: Invalid argument (22) lfs read: fail to read data from mirror 2: Invalid argument sanity-flr test_50d: @@@@@@ FAIL: data mismatch: \'a7ed1ef90410d7e8e4b3902ae358e0e5 -\' vs. \'d41d8cd98f00b204e9800998ecf8427e -\' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3054:test_50d() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3066:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_50d.*.1746139215.log rsync: chown "/tmp/testlogs/.sanity-flr.test_50d.debug_log.oleg422-server.1746139215.log.a0Ddq9" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_50d.dmesg.oleg422-server.1746139215.log.0wful2" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 18:40:17 (1746139217) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00417323 s, 981 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0264567 s, 155 kB/s PASS 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 18:40:20 (1746139220) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (10s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 18:40:30 (1746139230) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps lfs migrate: error reading src bytes 0-67108864: Invalid argument (22) error: lfs migrate: /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr: data copy failed: Invalid argument sanity-flr test_61a: @@@@@@ FAIL: cannot migrate /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3189:test_61a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3196:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_61a.*.1746139249.log rsync: chown "/tmp/testlogs/.sanity-flr.test_61a.debug_log.oleg422-server.1746139249.log.m8M6kp" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_61a.dmesg.oleg422-server.1746139249.log.7ZpCvJ" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 FAIL 61a (21s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 18:40:51 (1746139251) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4c:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (20s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 18:41:11 (1746139271) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps lfs mirror mirror: fail to pread 0-4096 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 0-4096 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr': Invalid argument (22) sanity-flr test_61c: @@@@@@ FAIL: cannot resync mirror /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3261:test_61c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3271:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_61c.*.1746139288.log rsync: chown "/tmp/testlogs/.sanity-flr.test_61c.debug_log.oleg422-server.1746139288.log.d3RM59" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_61c.dmesg.oleg422-server.1746139288.log.64fhR7" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 18:41:30 (1746139290) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 18:41:33 (1746139293) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 26171 mirror split pid 26172 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 26171 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 26172 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (64s) == sanity-flr test 70a: flr mode fsx test ================ 18:42:37 (1746139357) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 9358 skipping zero length punch hole truncating to largest ever: 0x19ce4 000002 1746139358.881508 trunc from 00000000 to 0x019ce3 (0x19ce4 bytes) truncating to largest ever: 0x33bef 000003 1746139358.893368 trunc from 0x019ce4 to 0x033bee (0x19f0b bytes) 000004 1746139358.903297 trunc from 0x002515 to 0x00ec67 (0xc753 bytes) 000005 1746139358.906754 trunc from 0x033bef to 0x002591 (0x3165d bytes) 000007 1746139358.920180 fallocate 0x01bc54 thru 0x02b5b1 (0xf95e bytes) 000008 1746139358.925037 mapwrite 0x00f67a thru 0x01e759 (0xf0e0 bytes) 000009 1746139359.025898 mapread 0x015c45 thru 0x019d52 (0x410e bytes) 000011 1746139359.027986 punch from 0x010699 to 0x016214 (0x5b7c bytes) 000013 1746139359.044434 mapwrite 0x00727d thru 0x01343e (0xc1c2 bytes) 000014 1746139359.075516 read 0x005ae5 thru 0x00d284 (0x77a0 bytes) 000015 1746139359.076819 trunc from 0x01251f to 0x0168a8 (0x438a bytes) 000017 1746139359.081809 trunc from 0x00be8b to 0x013228 (0x739e bytes) 000019 1746139359.085885 mapread 0x010508 thru 0x01c2be (0xbdb7 bytes) 000020 1746139359.093718 fallocate 0x013f13 thru 0x01578a (0x1878 bytes) 000021 1746139359.099799 read 0x0099b9 thru 0x00eef7 (0x553f bytes) 000022 1746139359.106604 mapwrite 0x016642 thru 0x01d0b0 (0x6a6f bytes) 000023 1746139359.128350 read 0x00f2d4 thru 0x0114ab (0x21d8 bytes) 000024 1746139359.129200 trunc from 0x001f2d to 0x00c992 (0xaa66 bytes) 000025 1746139359.134899 mapread 0x0053cd thru 0x00f63e (0xa272 bytes) 000027 1746139359.144720 read 0x019a7d thru 0x01d08c (0x3610 bytes) 000028 1746139359.145199 trunc from 0x0116a5 to 0x01e759 (0xd0b5 bytes) 000029 1746139359.156478 read 0x00bcf6 thru 0x016ce1 (0xafec bytes) 000030 1746139359.169856 mapwrite 0x02c763 thru 0x031d9f (0x563d bytes) 000032 1746139359.211690 write 0x022959 thru 0x02f4fd (0xcba5 bytes) 000033 1746139359.212834 trunc from 0x031da0 to 0x018faa (0x18df5 bytes) 000034 1746139359.230218 write 0x033e22 thru 0x03e7fc (0xa9db bytes) 000035 1746139359.231604 punch from 0x014897 to 0x019437 (0x4ba1 bytes) 000036 1746139359.238316 trunc from 0x03e7fd to 0x0144ed (0x2a30f bytes) 000037 1746139359.256433 trunc from 0x0144ee to 0x032f6b (0x1ea7e bytes) 000038 1746139359.276751 punch from 0x01a89d to 0x02961b (0xed7f bytes) 000039 1746139359.283089 trunc from 0x025eea to 0x03002c (0xa143 bytes) 000041 1746139359.290092 trunc from 0x032f6c to 0x029efb (0x9070 bytes) 000042 1746139359.306279 trunc from 0x017c03 to 0x01c709 (0x4b07 bytes) 000043 1746139359.313504 mapwrite 0x011c6c thru 0x01bdb5 (0xa14a bytes) 000044 1746139359.370909 trunc from 0x014e41 to 0x0153a1 (0x561 bytes) 000045 1746139359.375845 trunc from 0x029efc to 0x018e70 (0x1108b bytes) truncating to largest ever: 0x38733 000046 1746139359.391333 trunc from 0x018e71 to 0x038732 (0x1f8c2 bytes) 000047 1746139359.403580 mapread 0x002fb2 thru 0x00fe21 (0xce70 bytes) 000048 1746139359.405174 mapwrite 0x021df6 thru 0x0295bc (0x77c7 bytes) 000049 1746139359.453981 mapread 0x0170d7 thru 0x01cb6a (0x5a94 bytes) 000050 1746139359.468482 mapwrite 0x00e763 thru 0x0122e8 (0x3b86 bytes) 000051 1746139359.486556 trunc from 0x006203 to 0x00aa08 (0x4806 bytes) 000053 1746139359.493196 punch from 0x01db64 to 0x02a49f (0xc93c bytes) 000056 1746139359.498307 read 0x003f07 thru 0x00c7d3 (0x88cd bytes) 000057 1746139359.528241 read 0x002582 thru 0x00cce6 (0xa765 bytes) 000058 1746139359.529332 read 0x01a825 thru 0x01e5f1 (0x3dcd bytes) 000059 1746139359.529788 write 0x003b05 thru 0x01059b (0xca97 bytes) 000060 1746139359.530824 mapwrite 0x01cdd4 thru 0x01f8ff (0x2b2c bytes) 000062 1746139359.552580 punch from 0x011ba9 to 0x020a0c (0xee64 bytes) 000063 1746139359.561549 mapwrite 0x016cee thru 0x0187ec (0x1aff bytes) 000066 1746139359.577723 write 0x001198 thru 0x00755f (0x63c8 bytes) 000067 1746139359.578593 fallocate 0x00878e thru 0x00dbe7 (0x545a bytes) 000068 1746139359.584360 punch from 0x030420 to 0x038732 (0x8313 bytes) 000069 1746139359.589016 punch from 0x024596 to 0x02a482 (0x5eed bytes) 000070 1746139359.591914 mapwrite 0x02e60f thru 0x0381f2 (0x9be4 bytes) 000071 1746139359.630403 punch from 0x01e869 to 0x023a0f (0x51a7 bytes) fallocating to largest ever: 0x3a129 000072 1746139359.634375 fallocate 0x036304 thru 0x03a128 (0x3e25 bytes) 000074 1746139359.637497 trunc from 0x023c42 to 0x027892 (0x3c51 bytes) 000075 1746139359.642320 read 0x0234c7 thru 0x02a54a (0x7084 bytes) 000076 1746139359.648997 mapread 0x00ff0b thru 0x01f9ba (0xfab0 bytes) 000077 1746139359.650035 trunc from 0x03762b to 0x038732 (0x1108 bytes) 000078 1746139359.656859 write 0x0021d2 thru 0x010a86 (0xe8b5 bytes) 000079 1746139359.658954 punch from 0x002956 to 0x0082fc (0x59a7 bytes) 000081 1746139359.676089 punch from 0x0311c2 to 0x038732 (0x7571 bytes) 000082 1746139359.684301 read 0x02809a thru 0x034dc7 (0xcd2e bytes) 000083 1746139359.690879 write 0x037f9c thru 0x03f737 (0x779c bytes) 000085 1746139359.693761 trunc from 0x03f738 to 0x02cd50 (0x129e7 bytes) 000086 1746139359.719481 trunc from 0x02cd51 to 0x0288cc (0x4484 bytes) 000087 1746139359.737764 trunc from 0x0288cd to 0x00e086 (0x1a846 bytes) 000088 1746139359.762571 trunc from 0x00e087 to 0x01c0c8 (0xe042 bytes) 000090 1746139359.796030 trunc from 0x004625 to 0x010d50 (0xc72c bytes) 000092 1746139359.802643 mapread 0x018d91 thru 0x01c0c8 (0x3338 bytes) 000094 1746139359.834707 punch from 0x01b36b to 0x01c0c8 (0xd5e bytes) 000095 1746139359.847205 trunc from 0x0084aa to 0x010f7d (0x8ad4 bytes) 000096 1746139359.859588 MIRROR_EXTEND to 2 mirrors lfs mirror mirror: fail to pread 0-4194304 of mirror 2: Invalid argument (22) lfs verify: failed to read data from mirror 2: Invalid argument. mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 22 lfs mirror mirror: fail to pread 0-4194304 of mirror 1: Invalid argument (22) lfs verify: failed to read data from mirror 1: Invalid argument. lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c0c9: skipped LOG DUMP (76 total operations): 1: 1746139358.877328 SKIPPED (no operation) 2: 1746139358.881508 TRUNC_UP 0x00000 to 0x19ce4 3: 1746139358.893368 TRUNC_UP 0x19ce4 to 0x33bef 4: 1746139358.903297 ZERO 0x02515 thru 0x0ec67 (0x0c753 bytes) 5: 1746139358.906754 TRUNC_DN 0x33bef to 0x02592 6: 1746139358.920180 FALLOC 0x1bc54 thru 0x2b5b2 (0x0f95e bytes)PAST_EOF 7: 1746139358.925037 MAPWRITE 0x0f67a thru 0x1e759 (0x0f0e0 bytes) 8: 1746139359.025898 MAPREAD 0x15c45 thru 0x19d52 (0x0410e bytes) 9: 1746139359.027986 PUNCH 0x10699 thru 0x16214 (0x05b7c bytes) 10: 1746139359.044434 MAPWRITE 0x0727d thru 0x1343e (0x0c1c2 bytes) 11: 1746139359.075516 READ 0x05ae5 thru 0x0d284 (0x077a0 bytes) 12: 1746139359.076819 ZERO 0x1251f thru 0x168a8 (0x0438a bytes) 13: 1746139359.081809 ZERO 0x0be8b thru 0x13228 (0x0739e bytes) 14: 1746139359.085885 MAPREAD 0x10508 thru 0x1c2be (0x0bdb7 bytes) 15: 1746139359.093718 FALLOC 0x13f13 thru 0x1578b (0x01878 bytes)INTERIOR 16: 1746139359.099799 READ 0x099b9 thru 0x0eef7 (0x0553f bytes) 17: 1746139359.106604 MAPWRITE 0x16642 thru 0x1d0b0 (0x06a6f bytes) 18: 1746139359.128350 READ 0x0f2d4 thru 0x114ab (0x021d8 bytes) 19: 1746139359.129200 ZERO 0x01f2d thru 0x0c992 (0x0aa66 bytes) 20: 1746139359.134899 MAPREAD 0x053cd thru 0x0f63e (0x0a272 bytes) 21: 1746139359.144720 READ 0x19a7d thru 0x1d08c (0x03610 bytes) 22: 1746139359.145199 ZERO 0x116a5 thru 0x1e759 (0x0d0b5 bytes) 23: 1746139359.156478 READ 0x0bcf6 thru 0x16ce1 (0x0afec bytes) 24: 1746139359.169856 MAPWRITE 0x2c763 thru 0x31d9f (0x0563d bytes) 25: 1746139359.211690 WRITE 0x22959 thru 0x2f4fd (0x0cba5 bytes) 26: 1746139359.212834 TRUNC_DN 0x31da0 to 0x18fab 27: 1746139359.230218 WRITE 0x33e22 thru 0x3e7fc (0x0a9db bytes) HOLE 28: 1746139359.231604 PUNCH 0x14897 thru 0x19437 (0x04ba1 bytes) 29: 1746139359.238316 TRUNC_DN 0x3e7fd to 0x144ee 30: 1746139359.256433 TRUNC_UP 0x144ee to 0x32f6c 31: 1746139359.276751 PUNCH 0x1a89d thru 0x2961b (0x0ed7f bytes) 32: 1746139359.283089 ZERO 0x25eea thru 0x3002c (0x0a143 bytes) 33: 1746139359.290092 TRUNC_DN 0x32f6c to 0x29efc 34: 1746139359.306279 ZERO 0x17c03 thru 0x1c709 (0x04b07 bytes) 35: 1746139359.313504 MAPWRITE 0x11c6c thru 0x1bdb5 (0x0a14a bytes) 36: 1746139359.370909 ZERO 0x14e41 thru 0x153a1 (0x00561 bytes) 37: 1746139359.375845 TRUNC_DN 0x29efc to 0x18e71 38: 1746139359.391333 TRUNC_UP 0x18e71 to 0x38733 39: 1746139359.403580 MAPREAD 0x02fb2 thru 0x0fe21 (0x0ce70 bytes) 40: 1746139359.405174 MAPWRITE 0x21df6 thru 0x295bc (0x077c7 bytes) 41: 1746139359.453981 MAPREAD 0x170d7 thru 0x1cb6a (0x05a94 bytes) 42: 1746139359.468482 MAPWRITE 0x0e763 thru 0x122e8 (0x03b86 bytes) 43: 1746139359.486556 ZERO 0x06203 thru 0x0aa08 (0x04806 bytes) 44: 1746139359.493196 PUNCH 0x1db64 thru 0x2a49f (0x0c93c bytes) 45: 1746139359.498307 READ 0x03f07 thru 0x0c7d3 (0x088cd bytes) 46: 1746139359.528241 READ 0x02582 thru 0x0cce6 (0x0a765 bytes) 47: 1746139359.529332 READ 0x1a825 thru 0x1e5f1 (0x03dcd bytes) 48: 1746139359.529788 WRITE 0x03b05 thru 0x1059b (0x0ca97 bytes) 49: 1746139359.530824 MAPWRITE 0x1cdd4 thru 0x1f8ff (0x02b2c bytes) 50: 1746139359.552580 PUNCH 0x11ba9 thru 0x20a0c (0x0ee64 bytes) 51: 1746139359.561549 MAPWRITE 0x16cee thru 0x187ec (0x01aff bytes) 52: 1746139359.577723 WRITE 0x01198 thru 0x0755f (0x063c8 bytes) 53: 1746139359.578593 FALLOC 0x0878e thru 0x0dbe8 (0x0545a bytes)INTERIOR 54: 1746139359.584360 PUNCH 0x30420 thru 0x38732 (0x08313 bytes) 55: 1746139359.589016 PUNCH 0x24596 thru 0x2a482 (0x05eed bytes) 56: 1746139359.591914 MAPWRITE 0x2e60f thru 0x381f2 (0x09be4 bytes) 57: 1746139359.630403 PUNCH 0x1e869 thru 0x23a0f (0x051a7 bytes) 58: 1746139359.634375 FALLOC 0x36304 thru 0x3a129 (0x03e25 bytes)PAST_EOF 59: 1746139359.637497 ZERO 0x23c42 thru 0x27892 (0x03c51 bytes) 60: 1746139359.642320 READ 0x234c7 thru 0x2a54a (0x07084 bytes) 61: 1746139359.648997 MAPREAD 0x0ff0b thru 0x1f9ba (0x0fab0 bytes) 62: 1746139359.650035 ZERO 0x3762b thru 0x38732 (0x01108 bytes) 63: 1746139359.656859 WRITE 0x021d2 thru 0x10a86 (0x0e8b5 bytes) 64: 1746139359.658954 PUNCH 0x02956 thru 0x082fc (0x059a7 bytes) 65: 1746139359.676089 PUNCH 0x311c2 thru 0x38732 (0x07571 bytes) 66: 1746139359.684301 READ 0x2809a thru 0x34dc7 (0x0cd2e bytes) 67: 1746139359.690879 WRITE 0x37f9c thru 0x3f737 (0x0779c bytes) EXTEND 68: 1746139359.693761 TRUNC_DN 0x3f738 to 0x2cd51 69: 1746139359.719481 TRUNC_DN 0x2cd51 to 0x288cd 70: 1746139359.737764 TRUNC_DN 0x288cd to 0x0e087 71: 1746139359.762571 TRUNC_UP 0x0e087 to 0x1c0c9 72: 1746139359.796030 ZERO 0x04625 thru 0x10d50 (0x0c72c bytes) 73: 1746139359.802643 MAPREAD 0x18d91 thru 0x1c0c8 (0x03338 bytes) 74: 1746139359.834707 PUNCH 0x1b36b thru 0x1c0c8 (0x00d5e bytes) 75: 1746139359.847205 ZERO 0x084aa thru 0x10f7d (0x08ad4 bytes) 76: 1746139359.859588 MIRROR_EXTEND to 2 mirrors Using seed 9358 Correct content saved for comparison (maybe hexdump "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr" vs "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr.fsxgood") sanity-flr test_70a: @@@@@@ FAIL: fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3351:test_70a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3353:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_70a.*.1746139361.log rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.debug_log.oleg422-server.1746139361.log.S4PjMb" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.dmesg.oleg422-server.1746139361.log.b4xBbj" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 70a (7s) == sanity-flr test 71: check mirror extend parent fid ==== 18:42:44 (1746139364) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000403:0x3a9:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (6s) == sanity-flr test 200a: stress test ===================== 18:42:50 (1746139370) Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre2 Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 714912 .. Extending file size to 4031232 .. done Extending file size to 7958880 .. Extending file size to 5430368 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8716864 .. Extending file size to 8873088 .. Extending file size to 9122400 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9386368 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9049088 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 30496 30497 30498 30499 30500 failed 192.168.204.122@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg422-client.virtnet /mnt/lustre2 (opts:) 192.168.204.122@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg422-client.virtnet /mnt/lustre3 (opts:) lfs mirror mirror: fail to pread 12288-9375744 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 12288-9375744 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/f200a.sanity-flr': Invalid argument (22) sanity-flr test_200a: @@@@@@ FAIL: final resync failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3534:test_200a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3546:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_200a.*.1746139437.log rsync: chown "/tmp/testlogs/.sanity-flr.test_200a.debug_log.oleg422-server.1746139437.log.q7kjhI" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_200a.dmesg.oleg422-server.1746139437.log.QAxdz1" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 200a (70s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 18:44:00 (1746139440) Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre2 Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6309056 .. Extending file size to 8673888 .. Extending file size to 8778816 .. failed Extending file size to 8889920 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8928864 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8933568 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9028512 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 1112 1113 1115 1116 1118 failed 192.168.204.122@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg422-client.virtnet /mnt/lustre2 (opts:) 192.168.204.122@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg422-client.virtnet /mnt/lustre3 (opts:) lfs mirror mirror: fail to pread 4096-9027584 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 4096-9027584 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/f200b.sanity-flr': Invalid argument (22) PID TTY STAT TIME COMMAND 1 ? Ss 0:05 /usr/lib/systemd/systemd --switched-root --system --deserialize 22 2 ? S 0:00 [kthreadd] 4 ? S< 0:00 [kworker/0:0H] 5 ? S 0:00 [kworker/u8:0] 6 ? S 0:02 [ksoftirqd/0] 7 ? S 0:03 [migration/0] 8 ? S 0:00 [rcu_bh] 9 ? S 0:32 [rcu_sched] 10 ? S 0:00 [rcuob/0] 11 ? S 0:11 [rcuos/0] 12 ? S< 0:00 [lru-add-drain] 13 ? S 0:00 [watchdog/0] 14 ? S 0:00 [watchdog/1] 15 ? S 0:02 [migration/1] 16 ? S 0:00 [ksoftirqd/1] 17 ? S 0:02 [kworker/1:0] 18 ? S< 0:00 [kworker/1:0H] 19 ? S 0:00 [rcuob/1] 20 ? S 0:10 [rcuos/1] 21 ? S 0:00 [watchdog/2] 22 ? S 0:02 [migration/2] 23 ? S 0:00 [ksoftirqd/2] 24 ? S 0:00 [kworker/2:0] 25 ? S< 0:00 [kworker/2:0H] 26 ? S 0:00 [rcuob/2] 27 ? S 0:11 [rcuos/2] 28 ? S 0:00 [watchdog/3] 29 ? S 0:02 [migration/3] 30 ? S 0:00 [ksoftirqd/3] 32 ? S< 0:00 [kworker/3:0H] 33 ? S 0:00 [rcuob/3] 34 ? S 0:11 [rcuos/3] 35 ? S 0:00 [kdevtmpfs] 36 ? S< 0:00 [netns] 37 ? S 0:00 [khungtaskd] 38 ? S< 0:00 [writeback] 39 ? S< 0:00 [kintegrityd] 40 ? S< 0:00 [bioset] 41 ? S< 0:00 [bioset] 42 ? S< 0:00 [bioset] 43 ? S< 0:00 [kblockd] 44 ? S< 0:00 [md] 45 ? S< 0:00 [watchdogd] 46 ? S 0:00 [kworker/2:1] 47 ? S< 0:00 [rpciod] 48 ? S< 0:00 [xprtiod] 51 ? S 0:00 [kswapd0] 52 ? SN 0:00 [ksmd] 53 ? SN 0:00 [khugepaged] 54 ? S< 0:00 [nfsiod] 55 ? S< 0:00 [crypto] 63 ? S< 0:00 [kthrotld] 65 ? S< 0:00 [kmpath_rdacd] 66 ? S< 0:00 [kpsmoused] 67 ? S< 0:00 [ipv6_addrconf] 80 ? S< 0:00 [deferwq] 273 ? S< 0:00 [ata_sff] 277 ? S 0:00 [scsi_eh_0] 278 ? S< 0:00 [scsi_tmf_0] 279 ? S 0:00 [scsi_eh_1] 281 ? S< 0:00 [scsi_tmf_1] 302 ? S 0:01 [kworker/u8:4] 445 ? Ss 0:01 @bd-client --systemd-mark 192.168.200.253 -N centos7 /dev/nbd0 -p -b4096 447 ? S< 0:00 [nbd0] 475 ? S 0:00 [jbd2/nbd0-8] 476 ? S< 0:00 [ext4-rsv-conver] 560 ? Ss 0:02 /usr/lib/systemd/systemd-journald 594 ? Ss 0:00 /usr/lib/systemd/systemd-udevd 677 ? S 0:01 [kworker/0:2] 709 ? S< 0:00 [kvm-irqfd-clean] 722 ? S< 0:00 [edac-poller] 731 ? S< 0:00 [kworker/0:1H] 763 ? S 0:00 [nfsv4.1-svc] 783 ? S 0:00 [kworker/3:2] 790 ? S 0:00 [kworker/1:2] 799 ? Ss 0:01 /usr/lib/systemd/systemd-logind 801 ? Ss 0:03 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation 804 ? Ssl 0:00 /usr/sbin/gssproxy -D 812 ? Ssl 0:01 /usr/sbin/NetworkManager --no-daemon 819 ? Ss 0:00 /usr/sbin/sshd -D 837 ttyS1 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS1 vt220 838 tty1 Ss+ 0:00 /sbin/agetty --noclear tty1 linux 839 ttyS0 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS0 vt220 851 ? S 0:00 [kworker/u8:1] 852 ? Ssl 0:01 /usr/lib/polkit-1/polkitd --no-debug 856 ? S 0:00 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-7ba69b54-82d0-4b54-bb1b-f37e48dd76dd-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0 868 pts/0 S+ 0:00 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 870 pts/0 R+ 0:00 tee -i /tmp/testlogs//sanity-flr.test_200b.test_log.oleg422-client.log 887 ? Ssl 0:01 /usr/sbin/rsyslogd -n 1064 ? S< 0:00 [kworker/3:1H] 1122 ? Ss 0:00 sshd: root@pts/0 1161 pts/0 Ss+ 0:01 /bin/bash /home/green/git/lustre-release/lustre/tests/auster -D /tmp/testlogs/ -r -k sanity-flr 1990 ? S 0:00 [ptlrpc_hr00_000] 1991 ? S 0:00 [ptlrpc_hr01_000] 1992 ? S 0:00 [socknal_cd00] 1993 ? S 0:00 [socknal_cd01] 1994 ? S 0:00 [socknal_cd02] 1995 ? S 0:00 [socknal_cd03] 1996 ? S 0:00 [socknal_reaper] 1997 ? S 0:16 [socknal_sd00_00] 1998 ? S 0:16 [socknal_sd00_01] 1999 ? S 0:00 [socknal_sd01_00] 2000 ? S 0:00 [socknal_sd01_01] 2001 ? S 0:00 [acceptor_001] 2002 ? S 0:00 [monitor_thread] 2003 ? S 0:00 [lnet_discovery] 2004 ? S 0:00 [ptlrpcd_rcv] 2005 ? S 0:03 [ptlrpcd_00_00] 2006 ? S 0:03 [ptlrpcd_00_01] 2007 ? S 0:06 [ptlrpcd_01_00] 2008 ? S 0:06 [ptlrpcd_01_01] 2400 ? S< 0:00 [kworker/1:1H] 2984 ? S 0:00 /usr/lib/systemd/systemd-udevd 3006 pts/0 R+ 0:00 ps ax 6375 ? S< 0:00 [kworker/2:1H] 7404 ? S 0:00 [kworker/u8:2] 7650 pts/0 S+ 0:01 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 8516 ? S 0:02 [ldlm_cb00_000] 8517 ? S 0:01 [ldlm_cb00_001] 8518 ? S 0:00 [ldlm_cb01_000] 8519 ? S 0:00 [ldlm_cb01_001] 8520 ? S 0:00 [ldlm_cn00_000] 8521 ? S 0:00 [ldlm_cn00_001] 8522 ? S 0:00 [ldlm_cn00_002] 8523 ? S 0:00 [ldlm_cn01_000] 8524 ? S 0:00 [ldlm_cn01_001] 8525 ? S 0:00 [ldlm_cn01_002] 8526 ? S 0:01 [ldlm_bl_01] 8527 ? S 0:01 [ldlm_bl_02] 8528 ? S 0:00 [ldlm_elt] 8529 ? S 0:00 [ll_cfg_requeue] 16300 ? S 0:01 [ldlm_bl_03] 19123 ? S 0:02 [ldlm_cb00_002] 25991 ? S 0:00 [kworker/0:0] 26200 ? S 0:02 [kworker/0:1] 26714 ? S 0:00 [ldlm_cb01_002] 26752 ? S 0:00 [kworker/3:1] 26898 ? S 0:00 [ldlm_bl_04] 27903 ? S 0:00 [ldlm_bl_05] 28204 ? S 0:01 [kworker/1:1] 28698 ? S 0:00 [kworker/2:2] 29621 ? S 0:00 [kworker/3:0] 30670 ? S 0:00 [ldlm_bl_06] 30769 ? S 0:00 [ldlm_bl_07] 30770 ? S 0:00 [ldlm_bl_08] sanity-flr test_200b: @@@@@@ FAIL: final resync failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3606:test_200b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3618:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_200b.*.1746139505.log rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.debug_log.oleg422-server.1746139505.log.33L0aB" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.dmesg.oleg422-server.1746139505.log.rRa3FO" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 200b (69s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 18:45:09 (1746139509) Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2364 1285324 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12388 3594632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18028 3588992 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 30416 7183624 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2364 1285324 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 12388 3594632 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18028 3588992 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 30416 7183624 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0600816 s, 34.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0934864 s, 22.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.122@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg422-client.virtnet /mnt/lustre2 (opts:) PASS 200c (8s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 18:45:18 (1746139518) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0544089 s, 38.5 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 18:45:21 (1746139521) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x65:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } PASS 203 (4s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 18:45:25 (1746139525) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0516589 s, 40.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x413:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x414:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x415:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (4s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 18:45:29 (1746139529) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0580671 s, 36.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } PASS 204b (4s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 18:45:34 (1746139534) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0494535 s, 42.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x91:0x0] } PASS 204c (26s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 18:46:00 (1746139560) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg422-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.035812 s, 29.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x414:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x417:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x416:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x414:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x417:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x416:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x415:0x0] } PASS 204d (36s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 18:46:35 (1746139595) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg422-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0558176 s, 37.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x92:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x92:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (39s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 18:47:14 (1746139634) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg422-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg422-server: Pool lustre.test_204f created Adding targets to pool oleg422-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg422-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0597168 s, 35.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x93:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x93:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg422-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg422-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg422-server: Pool lustre.test_204f destroyed PASS 204f (51s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 18:48:05 (1746139685) PASS 205a (4s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 18:48:09 (1746139689) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 18:48:12 (1746139692) oleg422-server: Pool lustre.test_206 created Adding targets to pool oleg422-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg422-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg422-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg422-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg422-server: Pool lustre.test_206 destroyed PASS 206 (19s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 18:48:31 (1746139711) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0451152 s, 23.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0181248 s, 57.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x61d3b3ab Mirror 3: 0x61d3b3ab PASS 207 (6s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 18:48:37 (1746139717) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 18:48:39 (1746139719) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 18:48:40 (1746139720) Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0326395 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1c20000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xb80000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xb80000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xb80000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1490000 Read this (4 bytes): BBBB 192.168.204.122@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg422-client.virtnet /mnt/lustre2 (opts:) PASS 209a (24s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 18:49:04 (1746139744) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.01439 s, 569 kB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 18:49:08 (1746139748) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0345407 s, 30.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x86 lmm_fid: [0x200000404:0x86:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x86 lmm_fid: [0x200000404:0x86:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x9e:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834054 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-01 18:49:08.000000000 -0400 Modify: 2025-05-01 18:49:08.000000000 -0400 Change: 2025-05-01 18:49:09.000000000 -0400 Birth: - PASS 210a (4s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 18:49:12 (1746139752) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0298341 s, 35.1 MB/s oleg422-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg422-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg422-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg422-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg422-server: O/280000401/d31/bf: File not found by ext2_lookup PASS 210b (17s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 18:49:29 (1746139769) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.132243 s, 310 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0323785 s, 127 kB/s size after second write -rw-r--r-- 1 root root 4096 May 1 18:49 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x8b:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 May 1 18:49 /mnt/lustre/f211.sanity-flr PASS 211 (5s) == sanity-flr test complete, duration 954 sec ============ 18:49:34 (1746139774) sanity-flr: FAIL: test_0g error resync-ing file '/mnt/lustre/f0g.sanity-flr' sanity-flr: FAIL: test_0h error resync-ing file '/mnt/lustre/d0h.sanity-flr/f0h.sanity-flr' sanity-flr: FAIL: test_0j resync /mnt/lustre/f0j.sanity-flr failed sanity-flr: FAIL: test_36d mirror resync /mnt/lustre/f36d.sanity-flr failed sanity-flr: FAIL: test_37 1: mismatch: \'565cb6ec3c19e074cd885f69b9136fd8 -\' vs. \'b911b4caf86ade3ff3dcc943d71cbe47 -\' sanity-flr: FAIL: test_38 valid mirror doesn't exist sanity-flr: FAIL: test_44b split --mirror-id=1 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr should succeed sanity-flr: FAIL: test_50a data mismatch: \'e77f515b491f4db2743c19af08739cfc -\' vs. \'d41d8cd98f00b204e9800998ecf8427e -\' sanity-flr: FAIL: test_50b data mismatch: \'7b63b54edd900422cd2e120f663a6aae -\' vs. \'d41d8cd98f00b204e9800998ecf8427e -\' sanity-flr: FAIL: test_50d data mismatch: \'a7ed1ef90410d7e8e4b3902ae358e0e5 -\' vs. \'d41d8cd98f00b204e9800998ecf8427e -\' sanity-flr: FAIL: test_61a cannot migrate /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr sanity-flr: FAIL: test_61c cannot resync mirror /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sanity-flr: FAIL: test_70a fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed sanity-flr: FAIL: test_200a final resync failed sanity-flr: FAIL: test_200b final resync failed === sanity-flr: start cleanup 18:49:35 (1746139775) === === sanity-flr: finish cleanup 18:49:39 (1746139779) ===