-----============= acceptance-small: sanity-flr ============----- Mon Aug 17 06:02:05 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg343-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 06:02:26 (1786960946) === oleg343-client.virtnet: executing check_config_client /mnt/lustre oleg343-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg343-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff919a829bc000.idle_timeout=debug osc.lustre-OST0001-osc-ffff919a829bc000.idle_timeout=debug disable quota as required oleg343-server: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 06:02:59 (1786960979) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7646] == sanity-flr test 0a: lfs mirror create with -N option == 06:03:01 (1786960981) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 06:03:16 (1786960996) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 06:03:19 (1786960999) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 06:03:21 (1786961001) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 06:03:45 (1786961025) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 06:03:48 (1786961028) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 06:03:51 (1786961031) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff919a829bc000.stats=clear osc.lustre-OST0001-osc-ffff919a829bc000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 06:03:58 (1786961038) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 06:04:08 (1786961048) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 06:04:16 (1786961056) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 06:04:19 (1786961059) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 06:04:27 (1786961067) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 06:04:36 (1786961076) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 06:04:43 (1786961083) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 06:04:47 (1786961087) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 06:04:54 (1786961094) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 06:05:04 (1786961104) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 06:05:11 (1786961111) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 1.67379 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120288 s, 8.7 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 06:05:22 (1786961122) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.440783 s, 30.9 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 06:05:30 (1786961130) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 06:05:40 (1786961140) Stopping /mnt/lustre-ost1 (opts:) on oleg343-server oleg343-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff919a829bc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff919a829bc000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg343-server: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg343-server: oleg343-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg343-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg343-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg343-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg343-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg343-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg343-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg343-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg343-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg343-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg343-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg343-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg343-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg343-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg343-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg343-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg343-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg343-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg343-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg343-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg343-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg343-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg343-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg343-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg343-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg343-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg343-server oleg343-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff919a829bc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff919a829bc000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg343-server: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg343-server: oleg343-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg343-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (130s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 06:07:50 (1786961270) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131177 s, 16.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg343-server oleg343-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff919a829bc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff919a829bc000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg343-server: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg343-server: oleg343-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg343-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (45s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 06:08:35 (1786961315) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg343-server oleg343-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff919a829bc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff919a829bc000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg343-server: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg343-server: oleg343-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg343-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg343-server oleg343-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff919a829bc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff919a829bc000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg343-server: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg343-server: oleg343-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg343-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (76s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 06:09:52 (1786961392) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.58834 s, 9.4 MB/s -rw-r--r-- 1 root root 52428800 Aug 17 06:10 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.235s user 0m0.002s sys 0m0.172s Stopping /mnt/lustre-ost1 (opts:) on oleg343-server oleg343-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff919a829bc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff919a829bc000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.205s user 0m0.002s sys 0m0.174s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg343-server: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg343-server: oleg343-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg343-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg343-server oleg343-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff919a829bc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff919a829bc000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.326s user 0m0.003s sys 0m0.183s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg343-server: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg343-server: oleg343-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg343-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg343-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff919a829bc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff919a829bc000.ost_server_uuid in FULL state after 0 sec PASS 33b (96s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 06:11:27 (1786961487) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 06:11:30 (1786961490) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 06:11:33 (1786961493) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 06:11:36 (1786961496) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 06:11:43 (1786961503) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.557826 s, 22.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 06:11:55 (1786961515) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188756 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396803 s, 26.4 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 06:12:02 (1786961522) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 06:12:10 (1786961530) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00104275 s, 982 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00326766 s, 313 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00302309 s, 339 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00127911 s, 801 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 06:12:20 (1786961540) create test files with size 15750691 .. cba21c1ec8956ebdb1ec372065c0adbe - e893dc0c80b3866db22bc68d88415f1f - 6ad972ea37ee9837a7f777615819fe6e - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg343-client.virtnet /mnt/lustre (opts:) Starting client: oleg343-client.virtnet: -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 06:12:44 (1786961564) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (21s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 06:13:05 (1786961585) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 06:13:12 (1786961592) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817637 s, 12.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xae:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xaf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xed:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037415 s, 28.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xee:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xef:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 06:13:20 (1786961600) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.165803 s, 12.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.414314 s, 10.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.161195 s, 13.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 06:13:38 (1786961618) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 06:13:43 (1786961623) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 06:13:46 (1786961626) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325898 s, 32.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 06:13:54 (1786961634) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 06:13:58 (1786961638) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xf6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 06:14:09 (1786961649) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.399141 s, 26.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg343-client.virtnet /mnt/lustre (opts:) Starting client: oleg343-client.virtnet: -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 06:14:23 (1786961663) llite.lustre-ffff919a98b86000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.234354 s, 44.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg343-client.virtnet /mnt/lustre (opts:) Starting client: oleg343-client.virtnet: -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff919a929ca800.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 06:14:37 (1786961677) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (39s) == sanity-flr test 46: Verify setstripe --copy option ==== 06:15:16 (1786961716) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (33s) == sanity-flr test 47: Verify mirror obj alloc =========== 06:15:49 (1786961749) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 06:15:53 (1786961753) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.254997 s, 12.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.179294 s, 17.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xace3d9d2 (nosync,stale) Mirror 2: 0xace3d9d2 (nosync,stale) ** verify mirror 2 doesn't change original checksum: f6b9306726ed339015c1d1fde4f7d456 - mirror 1 checksum: 13a32c780e871827b27903b8dad301b6 - mirror 2 checksum: f6b9306726ed339015c1d1fde4f7d456 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-17T06:15:57-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-17T06:15:57-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: f6b9306726ed339015c1d1fde4f7d456 - mirror 1 checksum: 13a32c780e871827b27903b8dad301b6 - mirror 2 checksum: 13a32c780e871827b27903b8dad301b6 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-17T06:15:57-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-17T06:15:57-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcd:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 06:16:07 (1786961767) Failing mds1 on oleg343-server Stopping /mnt/lustre-mds1 (opts:) on oleg343-server 06:16:15 (1786961775) shut down facet: mds1 facet_host: oleg343-server facet_failover_host: oleg343-server Failover mds1 to oleg343-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg343-server: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-MDT0000 06:16:39 (1786961799) targets are mounted 06:16:39 (1786961799) facet_failover done oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258465 s, 40.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xce:0x0] } PASS 50A (47s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 06:16:55 (1786961815) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 06:16:58 (1786961818) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 06:17:01 (1786961821) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00661637 s, 2.5 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 06:17:08 (1786961828) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 06:17:11 (1786961831) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0287741 s, 142 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0713234 s, 57.4 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 06:17:19 (1786961839) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 06:17:41 (1786961861) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 06:18:09 (1786961889) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 06:18:33 (1786961913) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 06:18:59 (1786961939) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 06:19:06 (1786961946) mirror create pid 54343 mirror split pid 54344 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54343 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54344 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 06:20:14 (1786962014) Seed set to 2016 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1786962016.620368 write 0x014723 thru 0x01acb8 (0x6596 bytes) truncating to largest ever: 0x164ef 000003 1786962016.626468 trunc from 0x01acb9 to 0x0164ee (0x47ca bytes) 000004 1786962016.640661 write 0x00f5c7 thru 0x01e4ea (0xef24 bytes) 000005 1786962016.648574 mapread 0x00b0ab thru 0x01ae3f (0xfd95 bytes) 000006 1786962016.669033 mapwrite 0x034a83 thru 0x034f07 (0x485 bytes) 000008 1786962016.805484 write 0x0318f9 thru 0x039af7 (0x81ff bytes) 000009 1786962016.806768 write 0x037bf5 thru 0x03b462 (0x386e bytes) 000010 1786962016.819827 read 0x00097b thru 0x0079fe (0x7084 bytes) 000013 1786962016.845750 write 0x03a53c thru 0x03ffff (0x5ac4 bytes) 000014 1786962016.848477 mapread 0x028e34 thru 0x037d5d (0xef2a bytes) 000015 1786962016.877952 write 0x03aeaa thru 0x03ffff (0x5156 bytes) 000017 1786962016.881410 mapread 0x00ac6a thru 0x016339 (0xb6d0 bytes) 000018 1786962016.882641 write 0x0197a0 thru 0x021a19 (0x827a bytes) 000019 1786962016.884838 mapread 0x03f560 thru 0x03ffff (0xaa0 bytes) 000020 1786962016.886442 read 0x004171 thru 0x00ea04 (0xa894 bytes) 000022 1786962016.887815 write 0x01b6a2 thru 0x0229f4 (0x7353 bytes) 000023 1786962016.892918 read 0x0343d8 thru 0x03463e (0x267 bytes) 000025 1786962016.894723 mapread 0x016507 thru 0x01b40e (0x4f08 bytes) 000026 1786962016.895956 mapread 0x011d64 thru 0x01351a (0x17b7 bytes) 000031 1786962016.900313 trunc from 0x040000 to 0x00322c (0x3cdd3 bytes) 000034 1786962016.956127 mapread 0x00157b thru 0x00322c (0x1cb2 bytes) truncating to largest ever: 0x1d83e 000036 1786962016.962120 trunc from 0x00322d to 0x01d83d (0x1a611 bytes) 000038 1786962016.993877 write 0x025b63 thru 0x026711 (0xbaf bytes) 000039 1786962016.996900 mapwrite 0x0120c0 thru 0x01bc5d (0x9b9e bytes) 000040 1786962017.089254 mapread 0x01de72 thru 0x025727 (0x78b6 bytes) 000041 1786962017.091566 mapread 0x0216b9 thru 0x023cc5 (0x260d bytes) 000042 1786962017.095721 trunc from 0x026712 to 0x01599a (0x10d77 bytes) 000044 1786962017.140033 read 0x00255c thru 0x010f46 (0xe9eb bytes) 000045 1786962017.144003 trunc from 0x01599b to 0x010067 (0x5933 bytes) 000046 1786962017.176590 write 0x008cad thru 0x00c076 (0x33ca bytes) 000047 1786962017.179005 write 0x03e074 thru 0x03ffff (0x1f8c bytes) 000048 1786962017.181360 trunc from 0x040000 to 0x015a90 (0x2a56f bytes) 000049 1786962017.217138 mapwrite 0x01b771 thru 0x029603 (0xde93 bytes) 000051 1786962017.314803 mapwrite 0x03366b thru 0x037b7c (0x4512 bytes) 000052 1786962017.393127 trunc from 0x037b7d to 0x014800 (0x2337c bytes) 000055 1786962017.414890 mapread 0x00368f thru 0x009f08 (0x687a bytes) 000056 1786962017.416419 trunc from 0x014801 to 0x000fec (0x13814 bytes) 000057 1786962017.438399 mapread 0x00021e thru 0x000fec (0xdcf bytes) 000058 1786962017.439410 read 0x000dcf thru 0x000fec (0x21e bytes) 000062 1786962017.443504 read 0x0007f9 thru 0x000e88 (0x690 bytes) 000064 1786962017.444910 write 0x01e020 thru 0x0206da (0x26bb bytes) truncating to largest ever: 0x363e3 000065 1786962017.446762 trunc from 0x0206db to 0x0363e2 (0x15d08 bytes) 000069 1786962017.463684 MIRROR_EXTEND to 2 mirrors 000070 1786962018.228254 write 0x00c6aa thru 0x00eef4 (0x284b bytes) 000072 1786962018.313790 trunc from 0x0363e3 to 0x00c4fe (0x29ee4 bytes) 000073 1786962018.330822 mapread 0x009d4e thru 0x00c4fe (0x27b1 bytes) 000074 1786962018.347922 mapwrite 0x017372 thru 0x01fd31 (0x89c0 bytes) 000075 1786962018.410596 write 0x0221d0 thru 0x02ef23 (0xcd54 bytes) 000077 1786962018.412692 mapwrite 0x006606 thru 0x00783b (0x1236 bytes) 000080 1786962018.457733 read 0x00961c thru 0x00d425 (0x3e0a bytes) 000083 1786962018.464929 trunc from 0x02ef24 to 0x024037 (0xaeec bytes) 000084 1786962018.481142 read 0x01973a thru 0x024037 (0xa8fe bytes) 000085 1786962018.493980 write 0x014467 thru 0x01eb20 (0xa6ba bytes) 000087 1786962018.500568 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) 000090 1786962018.680718 MIRROR_EXTEND to 2 mirrors 000094 1786962019.679105 trunc from 0x024038 to 0x027e9f (0x3e68 bytes) 000095 1786962019.809329 trunc from 0x027ea0 to 0x01ec2e (0x9271 bytes) 000096 1786962019.847039 mapwrite 0x027ccf thru 0x02c3ac (0x46de bytes) 000097 1786962019.945874 mapread 0x01634e thru 0x02278b (0xc43e bytes) 000098 1786962019.963012 read 0x000943 thru 0x0022e6 (0x19a4 bytes) 000099 1786962019.976706 read 0x0134a5 thru 0x019ad4 (0x6630 bytes) 000100 1786962019.979919 read 0x024f01 thru 0x02c3ac (0x74ac bytes) 000106 1786962019.985255 mapwrite 0x02ec9a thru 0x03b97e (0xcce5 bytes) 000109 1786962020.128107 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b97f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b97f): Mirror 2: 0xd86df572 (stale) CRC-32 checksum value for chunk [0, 0x3b97f): Mirror 1: 0xc331116d Mirror 3: 0xc331116d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b97f: skipped 000110 1786962021.112634 mapread 0x007afc thru 0x01758b (0xfa90 bytes) 000111 1786962021.143233 mapwrite 0x01056c thru 0x0200dc (0xfb71 bytes) truncating to largest ever: 0x37154 000112 1786962021.372306 trunc from 0x03b97f to 0x037153 (0x482b bytes) 000113 1786962021.430814 trunc from 0x037154 to 0x01ec4c (0x18507 bytes) 000116 1786962021.453481 trunc from 0x01ec4d to 0x029271 (0xa625 bytes) 000118 1786962021.475196 read 0x0132b4 thru 0x01f783 (0xc4d0 bytes) 000120 1786962021.484131 mapwrite 0x00c0e4 thru 0x00c118 (0x35 bytes) 000122 1786962021.545298 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29272): Mirror 3: 0x99346f69 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29272): Mirror 3: 0x99346f69 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29272: skipped 000123 1786962021.836027 mapread 0x018bb4 thru 0x01c124 (0x3571 bytes) truncating to largest ever: 0x3b065 000124 1786962021.855539 trunc from 0x029272 to 0x03b064 (0x11df3 bytes) 000125 1786962021.875259 MIRROR_RESYNC 2 mirrors 000127 1786962022.747140 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) 000128 1786962022.948708 mapread 0x0125c6 thru 0x01e3e8 (0xbe23 bytes) 000132 1786962022.982012 trunc from 0x03b065 to 0x022608 (0x18a5c bytes) 000133 1786962023.007876 mapread 0x01a870 thru 0x01fb64 (0x52f5 bytes) 000134 1786962023.016585 read 0x00fd70 thru 0x016feb (0x727c bytes) 000135 1786962023.022689 read 0x00df30 thru 0x018a88 (0xab59 bytes) 000137 1786962023.036136 mapwrite 0x00dd43 thru 0x016c07 (0x8ec5 bytes) 000141 1786962023.088676 trunc from 0x022609 to 0x00bc8a (0x1697e bytes) 000142 1786962023.113936 mapwrite 0x008bd9 thru 0x0090cd (0x4f5 bytes) 000143 1786962023.159833 read 0x00a78b thru 0x00bc8a (0x1500 bytes) 000144 1786962023.160492 write 0x0262c1 thru 0x0359fe (0xf73e bytes) 000145 1786962023.162095 mapread 0x00590a thru 0x006553 (0xc4a bytes) 000149 1786962023.167227 trunc from 0x0359ff to 0x0320b9 (0x3945 bytes) 000150 1786962023.181845 mapwrite 0x028118 thru 0x033741 (0xb62a bytes) 000152 1786962023.316471 trunc from 0x033742 to 0x0347f0 (0x10af bytes) 000153 1786962023.358829 write 0x00d777 thru 0x018908 (0xb192 bytes) 000155 1786962023.363499 write 0x023af8 thru 0x02b8e1 (0x7dea bytes) 000156 1786962023.365951 mapread 0x020649 thru 0x025fdc (0x5994 bytes) 000157 1786962023.370219 mapwrite 0x034eab thru 0x03737d (0x24d3 bytes) 000159 1786962023.528376 write 0x023f24 thru 0x02f6e8 (0xb7c5 bytes) 000160 1786962023.536747 MIRROR_EXTEND to 2 mirrors 000161 1786962024.120262 read 0x017753 thru 0x024c0c (0xd4ba bytes) 000162 1786962024.136608 write 0x03dac6 thru 0x03f6ae (0x1be9 bytes) 000163 1786962024.234542 read 0x039a21 thru 0x03f6ae (0x5c8e bytes) 000164 1786962024.274790 mapread 0x003dbd thru 0x010696 (0xc8da bytes) 000165 1786962024.287983 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) 000166 1786962024.486873 mapread 0x0105d0 thru 0x014b02 (0x4533 bytes) 000167 1786962024.518688 write 0x001e35 thru 0x00f6e6 (0xd8b2 bytes) 000169 1786962024.527018 mapread 0x002fa0 thru 0x004712 (0x1773 bytes) 000171 1786962024.530809 mapread 0x01a099 thru 0x0281c7 (0xe12f bytes) 000173 1786962024.560725 read 0x038383 thru 0x039877 (0x14f5 bytes) 000174 1786962024.569170 mapread 0x038e5d thru 0x03acad (0x1e51 bytes) 000175 1786962024.589131 mapwrite 0x011c0e thru 0x0175dc (0x59cf bytes) 000176 1786962024.651600 mapwrite 0x016fbf thru 0x01c260 (0x52a2 bytes) 000181 1786962024.723233 MIRROR_EXTEND to 2 mirrors 000182 1786962025.580965 mapread 0x01b147 thru 0x01f7d3 (0x468d bytes) 000183 1786962025.599627 write 0x016bfc thru 0x01a132 (0x3537 bytes) 000185 1786962025.708512 MIRROR_RESYNC 2 mirrors 000186 1786962026.495180 trunc from 0x03f6af to 0x021142 (0x1e56c bytes) 000187 1786962026.657967 write 0x010f29 thru 0x01ddcf (0xcea7 bytes) 000189 1786962026.681203 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) 000192 1786962026.888755 trunc from 0x021143 to 0x01fbae (0x1594 bytes) 000193 1786962026.908312 write 0x0142a0 thru 0x01afe0 (0x6d41 bytes) 000194 1786962026.918224 mapread 0x016c10 thru 0x01e4e7 (0x78d8 bytes) 000195 1786962026.939617 read 0x000f4d thru 0x007d96 (0x6e4a bytes) 000199 1786962026.949174 mapwrite 0x004dd3 thru 0x009b97 (0x4dc5 bytes) 000200 1786962027.019810 mapwrite 0x030697 thru 0x032890 (0x21fa bytes) 000203 1786962027.110392 mapwrite 0x00c9a9 thru 0x01b4d6 (0xeb2e bytes) 000204 1786962027.177216 mapwrite 0x033abc thru 0x039c3b (0x6180 bytes) 000207 1786962027.262521 read 0x010969 thru 0x01179b (0xe33 bytes) 000211 1786962027.265793 mapwrite 0x03f2b9 thru 0x03ffff (0xd47 bytes) 000216 1786962027.338571 mapread 0x00a1e9 thru 0x012274 (0x808c bytes) 000217 1786962027.340552 trunc from 0x040000 to 0x005e18 (0x3a1e7 bytes) 000218 1786962027.390073 trunc from 0x005e19 to 0x03a390 (0x34578 bytes) 000219 1786962027.437730 mapread 0x01f629 thru 0x020161 (0xb39 bytes) 000224 1786962027.457332 read 0x012ad5 thru 0x0200bc (0xd5e8 bytes) 000225 1786962027.470134 mapwrite 0x0182ff thru 0x025e83 (0xdb85 bytes) 000226 1786962027.524872 mapread 0x035fa5 thru 0x03a390 (0x43ec bytes) 000228 1786962027.527312 mapwrite 0x01c58d thru 0x02412e (0x7ba2 bytes) 000230 1786962027.584818 mapread 0x012374 thru 0x0191f2 (0x6e7f bytes) 000231 1786962027.586624 trunc from 0x03a391 to 0x010ab8 (0x298d8 bytes) 000232 1786962027.614150 mapwrite 0x007540 thru 0x0129e6 (0xb4a7 bytes) 000235 1786962027.728208 mapread 0x00fdfb thru 0x0129e6 (0x2bec bytes) 000237 1786962027.730736 write 0x03b2c2 thru 0x03ffff (0x4d3e bytes) 000238 1786962027.732130 read 0x004d88 thru 0x00f7a1 (0xaa1a bytes) 000240 1786962027.734079 trunc from 0x040000 to 0x01da5e (0x225a1 bytes) 000246 1786962027.785459 MIRROR_EXTEND to 2 mirrors 000247 1786962028.624509 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) 000248 1786962028.744564 mapwrite 0x03c7ae thru 0x03ffff (0x3852 bytes) 000250 1786962028.860743 read 0x0104f5 thru 0x01de7b (0xd987 bytes) 000254 1786962028.884863 write 0x036b33 thru 0x03ffff (0x94cd bytes) 000255 1786962028.894395 mapread 0x010e42 thru 0x01b018 (0xa1d7 bytes) 000257 1786962028.897725 mapwrite 0x029129 thru 0x030517 (0x73ef bytes) 000258 1786962028.970961 mapread 0x02b247 thru 0x037581 (0xc33b bytes) 000260 1786962028.983715 mapwrite 0x03e0f8 thru 0x03f09c (0xfa5 bytes) 000261 1786962029.042656 mapread 0x03b749 thru 0x03ce09 (0x16c1 bytes) 000262 1786962029.043941 mapwrite 0x003e7b thru 0x012b44 (0xecca bytes) 000264 1786962029.163911 write 0x03d810 thru 0x03ffff (0x27f0 bytes) 000266 1786962029.165807 trunc from 0x040000 to 0x00d775 (0x3288a bytes) 000267 1786962029.211074 write 0x02b894 thru 0x039481 (0xdbee bytes) 000268 1786962029.214170 trunc from 0x039482 to 0x034e04 (0x467d bytes) truncating to largest ever: 0x3e7cc 000274 1786962029.240320 trunc from 0x034e05 to 0x03e7cb (0x99c7 bytes) 000275 1786962029.261740 trunc from 0x03e7cc to 0x02cb15 (0x11cb6 bytes) 000278 1786962029.280326 mapread 0x01c352 thru 0x01de68 (0x1b17 bytes) 000279 1786962029.290022 read 0x027f04 thru 0x02cb15 (0x4c12 bytes) 000281 1786962029.300177 read 0x026b9c thru 0x02cb15 (0x5f7a bytes) 000282 1786962029.300883 trunc from 0x02cb16 to 0x032ee6 (0x63d1 bytes) 000283 1786962029.315580 mapwrite 0x02060d thru 0x02cd17 (0xc70b bytes) 000286 1786962029.363714 write 0x01ee0b thru 0x0258fd (0x6af3 bytes) 000287 1786962029.364959 trunc from 0x032ee7 to 0x020d50 (0x12196 bytes) 000288 1786962029.386103 trunc from 0x020d51 to 0x019e09 (0x6f47 bytes) 000290 1786962029.399378 mapwrite 0x011ab9 thru 0x0164f6 (0x4a3e bytes) 000292 1786962029.440144 write 0x03d59c thru 0x03ffff (0x2a64 bytes) 000293 1786962029.441444 read 0x024e09 thru 0x02f84f (0xaa47 bytes) 000294 1786962029.448916 write 0x03d8c5 thru 0x03ffff (0x273b bytes) 000295 1786962029.450098 write 0x027042 thru 0x036951 (0xf910 bytes) 000296 1786962029.451715 trunc from 0x040000 to 0x00e70f (0x318f0 bytes) 000297 1786962029.465714 mapwrite 0x017b8d thru 0x024cc3 (0xd137 bytes) 000299 1786962029.651957 read 0x0170a4 thru 0x01fee2 (0x8e3f bytes) 000300 1786962029.652647 mapwrite 0x023708 thru 0x029f0d (0x6806 bytes) 000307 1786962029.803080 mapread 0x01563d thru 0x021a63 (0xc427 bytes) 000308 1786962029.804448 trunc from 0x029f0e to 0x01e607 (0xb906 bytes) 000311 1786962029.860655 write 0x03b101 thru 0x03ffff (0x4eff bytes) 000313 1786962029.877265 mapwrite 0x00d1b2 thru 0x015e24 (0x8c73 bytes) 000314 1786962029.991791 trunc from 0x040000 to 0x009828 (0x367d7 bytes) 000316 1786962030.025521 write 0x03d517 thru 0x03ffff (0x2ae9 bytes) 000317 1786962030.029340 write 0x023b36 thru 0x0294ca (0x5995 bytes) 000320 1786962030.067357 write 0x02776e thru 0x0351f7 (0xda8a bytes) 000321 1786962030.072657 mapwrite 0x02ab62 thru 0x03971b (0xebba bytes) 000323 1786962030.225275 mapread 0x038845 thru 0x03d0dc (0x4898 bytes) 000324 1786962030.234629 write 0x020ea2 thru 0x0290bf (0x821e bytes) 000326 1786962030.257136 trunc from 0x040000 to 0x00fe6c (0x30193 bytes) 000331 1786962030.300240 write 0x030c9c thru 0x03662e (0x5993 bytes) 000332 1786962030.302871 mapwrite 0x029088 thru 0x030c92 (0x7c0b bytes) 000334 1786962030.434192 mapread 0x022f12 thru 0x031ede (0xefcd bytes) 000336 1786962030.537268 MIRROR_EXTEND to 2 mirrors 000337 1786962031.908710 write 0x02c846 thru 0x035f79 (0x9734 bytes) 000339 1786962032.081503 trunc from 0x03662f to 0x03bad4 (0x54a6 bytes) 000340 1786962032.139524 read 0x014ef7 thru 0x0217b7 (0xc8c1 bytes) 000343 1786962032.168008 write 0x025f0b thru 0x030faf (0xb0a5 bytes) 000345 1786962032.175748 trunc from 0x03bad5 to 0x00118e (0x3a946 bytes) 000349 1786962032.219138 read 0x0007c6 thru 0x00118e (0x9c9 bytes) 000350 1786962032.233137 write 0x00cb4d thru 0x01af49 (0xe3fd bytes) 000354 1786962032.249946 trunc from 0x01af4a to 0x02b50b (0x105c2 bytes) 000359 1786962032.308532 write 0x02ab79 thru 0x033502 (0x898a bytes) 000360 1786962032.319950 trunc from 0x033503 to 0x02e774 (0x4d8e bytes) 000361 1786962032.344622 mapwrite 0x01d0d5 thru 0x027697 (0xa5c3 bytes) 000363 1786962032.456595 trunc from 0x02e775 to 0x028700 (0x6074 bytes) 000364 1786962032.483833 write 0x03eaac thru 0x03ffff (0x1554 bytes) 000365 1786962032.486275 read 0x00ee22 thru 0x01b8a6 (0xca85 bytes) 000367 1786962032.489806 mapread 0x02913d thru 0x030edb (0x7d9f bytes) 000368 1786962032.501036 mapwrite 0x000bf9 thru 0x00fa0d (0xee15 bytes) 000371 1786962032.580207 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xab626073 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa21a86e8 Mirror 3: 0xa21a86e8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000374 1786962033.311716 mapread 0x028f7f thru 0x0367aa (0xd82c bytes) 000376 1786962033.341600 trunc from 0x040000 to 0x037468 (0x8b97 bytes) 000377 1786962033.455739 MIRROR_RESYNC 3 mirrors 000382 1786962034.220430 mapwrite 0x033d60 thru 0x039b62 (0x5e03 bytes) 000388 1786962034.354645 read 0x024878 thru 0x031ea2 (0xd62b bytes) 000390 1786962034.367891 MIRROR_RESYNC 3 mirrors 000392 1786962035.383314 MIRROR_EXTEND to 4 mirrors 000393 1786962036.251724 read 0x02ce28 thru 0x036681 (0x985a bytes) 000394 1786962036.260387 read 0x012f4d thru 0x01cc44 (0x9cf8 bytes) 000395 1786962036.270638 MIRROR_RESYNC 4 mirrors 000398 1786962036.713160 write 0x03221f thru 0x03ffff (0xdde1 bytes) 000399 1786962036.791861 trunc from 0x040000 to 0x006df9 (0x39206 bytes) 000400 1786962036.812421 read 0x003c08 thru 0x006df9 (0x31f2 bytes) 000405 1786962036.845789 MIRROR_RESYNC 4 mirrors 000406 1786962037.790876 write 0x032dc6 thru 0x03786c (0x4aa7 bytes) 000408 1786962037.868642 mapwrite 0x00e6c1 thru 0x01ce65 (0xe7a5 bytes) 000410 1786962038.082466 mapwrite 0x0298b7 thru 0x03293f (0x9089 bytes) 000411 1786962038.155430 write 0x000c8e thru 0x005e9c (0x520f bytes) 000420 1786962038.180885 trunc from 0x03786d to 0x026448 (0x11424 bytes) 000421 1786962038.208184 MIRROR_RESYNC 4 mirrors 000422 1786962039.273928 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) 000423 1786962039.592764 read 0x017182 thru 0x01c1f2 (0x5071 bytes) 000424 1786962039.600435 trunc from 0x026449 to 0x00dc6c (0x187dc bytes) 000425 1786962039.701183 write 0x007cb8 thru 0x00c69a (0x49e3 bytes) 000426 1786962039.725108 mapwrite 0x010264 thru 0x01edd3 (0xeb70 bytes) 000427 1786962039.846158 MIRROR_RESYNC 3 mirrors 000429 1786962040.640248 read 0x0181a2 thru 0x01e198 (0x5ff7 bytes) 000430 1786962040.647278 mapread 0x018b76 thru 0x01edd3 (0x625e bytes) 000431 1786962040.652109 MIRROR_RESYNC 3 mirrors 000432 1786962040.973922 mapread 0x0061b9 thru 0x00c5f5 (0x643d bytes) 000433 1786962041.006708 mapread 0x00f686 thru 0x01cde3 (0xd75e bytes) 000434 1786962041.019766 write 0x03a1e1 thru 0x03ffff (0x5e1f bytes) 000435 1786962041.082775 read 0x012fa0 thru 0x017477 (0x44d8 bytes) 000439 1786962041.095325 mapread 0x018484 thru 0x0223ed (0x9f6a bytes) 000440 1786962041.113677 read 0x001f0e thru 0x0111fd (0xf2f0 bytes) 000441 1786962041.129942 mapread 0x008085 thru 0x0157e1 (0xd75d bytes) 000442 1786962041.131308 mapwrite 0x01ce5b thru 0x0237c7 (0x696d bytes) 000443 1786962041.202143 mapread 0x005575 thru 0x00d4b3 (0x7f3f bytes) 000444 1786962041.203853 write 0x0100bb thru 0x012630 (0x2576 bytes) 000448 1786962041.211424 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xad3c1f8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xad3c1f8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000450 1786962041.501987 trunc from 0x040000 to 0x0052de (0x3ad21 bytes) 000452 1786962041.522603 mapwrite 0x001d1e thru 0x007530 (0x5813 bytes) 000453 1786962041.611234 mapread 0x001b49 thru 0x007530 (0x59e8 bytes) 000455 1786962041.615265 mapwrite 0x00e7f3 thru 0x012395 (0x3ba3 bytes) 000456 1786962041.753609 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) 000457 1786962041.906775 trunc from 0x012396 to 0x00405c (0xe339 bytes) 000460 1786962041.928464 mapread 0x0014dd thru 0x00405c (0x2b80 bytes) 000462 1786962041.948217 mapread 0x003cf3 thru 0x00405c (0x36a bytes) 000467 1786962041.958773 mapwrite 0x005c9d thru 0x00b7b4 (0x5b18 bytes) 000468 1786962042.068694 mapread 0x001a6b thru 0x0026a3 (0xc39 bytes) 000469 1786962042.074275 read 0x0004d0 thru 0x00b7b4 (0xb2e5 bytes) 000470 1786962042.084311 read 0x003086 thru 0x00b7b4 (0x872f bytes) 000473 1786962042.091709 write 0x010a9d thru 0x011312 (0x876 bytes) 000475 1786962042.104167 write 0x028b19 thru 0x032716 (0x9bfe bytes) 000476 1786962042.113166 mapwrite 0x006183 thru 0x009cfc (0x3b7a bytes) 000482 1786962042.219720 mapread 0x01bd3c thru 0x01f2b8 (0x357d bytes) 000483 1786962042.242511 mapread 0x014aaa thru 0x01b1fb (0x6752 bytes) 000485 1786962042.247670 trunc from 0x032717 to 0x0368db (0x41c5 bytes) 000486 1786962042.273206 read 0x012787 thru 0x017c78 (0x54f2 bytes) 000487 1786962042.274278 read 0x01779c thru 0x021353 (0x9bb8 bytes) 000490 1786962042.279223 mapread 0x0030c0 thru 0x00fcad (0xcbee bytes) 000491 1786962042.284148 read 0x009d8f thru 0x01644b (0xc6bd bytes) 000493 1786962042.288096 mapwrite 0x00110a thru 0x00d9e5 (0xc8dc bytes) 000494 1786962042.359988 mapread 0x01de4a thru 0x0224d2 (0x4689 bytes) 000495 1786962042.367617 MIRROR_EXTEND to 2 mirrors 000496 1786962043.245430 mapread 0x019352 thru 0x02894d (0xf5fc bytes) 000497 1786962043.290894 mapread 0x01073a thru 0x0116cb (0xf92 bytes) 000499 1786962043.304150 read 0x0333b1 thru 0x0368db (0x352b bytes) truncating to largest ever: 0x3ece7 000500 1786962043.310966 trunc from 0x0368dc to 0x03ece6 (0x840b bytes) 000503 1786962043.472902 trunc from 0x03ece7 to 0x00bf90 (0x32d56 bytes) 000504 1786962043.499305 mapread 0x00996f thru 0x00bf90 (0x2622 bytes) 000505 1786962043.509102 mapread 0x004233 thru 0x00bf90 (0x7d5e bytes) 000508 1786962043.513002 write 0x03ba97 thru 0x03ea90 (0x2ffa bytes) 000510 1786962043.517515 write 0x03410d thru 0x03afc4 (0x6eb8 bytes) 000512 1786962043.543646 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) 000513 1786962043.746425 mapread 0x00f3b4 thru 0x011354 (0x1fa1 bytes) 000514 1786962043.783925 MIRROR_EXTEND to 2 mirrors 000517 1786962045.066371 mapwrite 0x0080d2 thru 0x015b8e (0xdabd bytes) 000519 1786962045.387471 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) 000525 1786962045.625714 mapread 0x034554 thru 0x03ea90 (0xa53d bytes) 000530 1786962045.680747 read 0x01372a thru 0x01546f (0x1d46 bytes) 000532 1786962045.689640 trunc from 0x03ea91 to 0x03d9fe (0x1092 bytes) 000533 1786962045.732240 read 0x010294 thru 0x017045 (0x6db2 bytes) 000536 1786962045.759407 mapwrite 0x03898c thru 0x038b7c (0x1f1 bytes) 000537 1786962045.866968 read 0x02e3c5 thru 0x030aec (0x2728 bytes) 000538 1786962045.870975 mapwrite 0x00940a thru 0x014792 (0xb389 bytes) 000539 1786962045.984714 mapread 0x00778a thru 0x01400c (0xc883 bytes) 000540 1786962045.992746 write 0x02d1d4 thru 0x038af1 (0xb91e bytes) 000544 1786962046.005954 read 0x0138f3 thru 0x013dbd (0x4cb bytes) 000546 1786962046.014796 mapwrite 0x00ee92 thru 0x01b8b9 (0xca28 bytes) 000548 1786962046.149669 mapwrite 0x034a5a thru 0x03e4bb (0x9a62 bytes) 000554 1786962046.346653 mapread 0x01df2f thru 0x024de0 (0x6eb2 bytes) 000555 1786962046.348176 trunc from 0x03e4bc to 0x02b5d5 (0x12ee6 bytes) 000557 1786962046.367356 read 0x0278ae thru 0x02b5d5 (0x3d28 bytes) 000559 1786962046.369282 trunc from 0x02b5d6 to 0x00a576 (0x2105f bytes) 000564 1786962046.406483 mapwrite 0x00d5ca thru 0x015284 (0x7cbb bytes) 000565 1786962046.544487 mapwrite 0x024215 thru 0x02c586 (0x8372 bytes) 000567 1786962046.780599 mapread 0x00ab6d thru 0x018276 (0xd70a bytes) 000568 1786962046.796184 MIRROR_EXTEND to 2 mirrors 000570 1786962047.457651 mapread 0x01bbd9 thru 0x01ed23 (0x314b bytes) 000571 1786962047.478538 trunc from 0x02c587 to 0x012ef1 (0x19695 bytes) 000572 1786962047.547520 write 0x029e9a thru 0x02de74 (0x3fdb bytes) 000573 1786962047.554307 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2de75): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2de75): Mirror 2: 0x42b2c9d9 (stale) CRC-32 checksum value for chunk [0, 0x2de75): Mirror 1: 0xfcf77d47 Mirror 3: 0xfcf77d47 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2de75: skipped 000574 1786962048.486522 MIRROR_RESYNC 3 mirrors 000575 1786962049.255045 mapread 0x024ce2 thru 0x02de74 (0x9193 bytes) 000577 1786962049.277576 write 0x0292e1 thru 0x02f3d7 (0x60f7 bytes) 000578 1786962049.347614 mapread 0x011c68 thru 0x018d35 (0x70ce bytes) 000579 1786962049.362555 read 0x01330b thru 0x01b63d (0x8333 bytes) 000580 1786962049.373116 mapwrite 0x018c1c thru 0x018c74 (0x59 bytes) 000581 1786962049.454813 mapwrite 0x0070d7 thru 0x01489d (0xd7c7 bytes) 000582 1786962049.563810 mapread 0x007ddd thru 0x00d97b (0x5b9f bytes) 000583 1786962049.566420 trunc from 0x02f3d8 to 0x01fa0c (0xf9cb bytes) 000585 1786962049.598824 mapread 0x01b85f thru 0x01fa0c (0x41ae bytes) 000586 1786962049.608223 mapwrite 0x029207 thru 0x037dad (0xeba7 bytes) 000589 1786962049.758299 read 0x0042ff thru 0x006d70 (0x2a72 bytes) 000591 1786962049.785164 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37dae): Mirror 3: 0x6d68738f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37dae): Mirror 3: 0x6d68738f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37dae: skipped 000592 1786962050.238532 mapwrite 0x02329d thru 0x025c5d (0x29c1 bytes) 000596 1786962050.393158 mapwrite 0x00f6a8 thru 0x01a9b0 (0xb309 bytes) 000597 1786962050.449009 write 0x003ad2 thru 0x00d836 (0x9d65 bytes) 000600 1786962050.458404 mapread 0x02200a thru 0x023d88 (0x1d7f bytes) 000605 1786962050.465975 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37dae): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37dae): Mirror 3: 0x6d68738f (stale) CRC-32 checksum value for chunk [0, 0x37dae): Mirror 1: 0xf312f6d4 Mirror 4: 0xf312f6d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37dae: skipped 000606 1786962051.265836 read 0x006bd7 thru 0x00998b (0x2db5 bytes) 000607 1786962051.281424 trunc from 0x037dae to 0x008aea (0x2f2c3 bytes) 000608 1786962051.443620 MIRROR_RESYNC 3 mirrors 000609 1786962052.102479 write 0x014452 thru 0x01c548 (0x80f7 bytes) 000611 1786962052.166572 trunc from 0x01c549 to 0x01a1fd (0x234b bytes) 000612 1786962052.188742 mapwrite 0x015335 thru 0x01a84c (0x5518 bytes) 000614 1786962052.272924 mapwrite 0x01b2b7 thru 0x01ffd2 (0x4d1c bytes) 000616 1786962052.481681 mapwrite 0x02bda6 thru 0x02e2e8 (0x2543 bytes) 000617 1786962052.556302 mapread 0x027caa thru 0x02e2e8 (0x663f bytes) 000618 1786962052.562641 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e2e9): Mirror 4: 0xf0d736fe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e2e9): Mirror 4: 0xf0d736fe (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e2e9: skipped 000619 1786962052.866408 mapread 0x00caa2 thru 0x01a52b (0xda8a bytes) 000620 1786962052.906200 mapread 0x01c7bd thru 0x01cfda (0x81e bytes) 000621 1786962052.907284 write 0x03a6fd thru 0x03ffff (0x5903 bytes) 000622 1786962052.911694 mapwrite 0x03cd93 thru 0x03ffff (0x326d bytes) 000623 1786962052.985264 mapwrite 0x021411 thru 0x03069b (0xf28b bytes) 000625 1786962053.103789 write 0x018e89 thru 0x01a8c4 (0x1a3c bytes) 000628 1786962053.106585 read 0x008803 thru 0x0150c3 (0xc8c1 bytes) 000630 1786962053.108716 read 0x00c978 thru 0x015674 (0x8cfd bytes) 000631 1786962053.109702 write 0x003415 thru 0x00386d (0x459 bytes) 000633 1786962053.119765 mapwrite 0x01a85f thru 0x01ac18 (0x3ba bytes) 000635 1786962053.163775 mapwrite 0x0091e2 thru 0x011d56 (0x8b75 bytes) 000636 1786962053.207786 read 0x02e5a2 thru 0x0358d0 (0x732f bytes) 000637 1786962053.208512 mapwrite 0x02a949 thru 0x03a282 (0xf93a bytes) 000638 1786962053.262562 mapread 0x02c14b thru 0x033ce4 (0x7b9a bytes) 000639 1786962053.268015 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) 000641 1786962053.372681 write 0x03288a thru 0x03a5be (0x7d35 bytes) 000642 1786962053.384649 write 0x00e534 thru 0x0147e6 (0x62b3 bytes) 000646 1786962053.397001 write 0x02247e thru 0x02f218 (0xcd9b bytes) 000647 1786962053.405317 trunc from 0x040000 to 0x037b1f (0x84e0 bytes) 000648 1786962053.422851 read 0x01499e thru 0x01674f (0x1db2 bytes) 000650 1786962053.428125 mapwrite 0x026baf thru 0x027608 (0xa5a bytes) 000651 1786962053.463449 write 0x0009d6 thru 0x00bdf6 (0xb421 bytes) 000652 1786962053.472759 mapread 0x024988 thru 0x03445a (0xfad3 bytes) 000653 1786962053.483229 trunc from 0x037b20 to 0x03ca09 (0x4eea bytes) 000655 1786962053.502550 mapread 0x01d508 thru 0x028424 (0xaf1d bytes) 000657 1786962053.516013 MIRROR_EXTEND to 2 mirrors 000658 1786962054.106905 write 0x0179da thru 0x02168e (0x9cb5 bytes) 000662 1786962054.330858 trunc from 0x03ca0a to 0x03593d (0x70cc bytes) 000663 1786962054.400410 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) 000665 1786962054.631335 MIRROR_EXTEND to 2 mirrors 000666 1786962055.207148 mapread 0x01746f thru 0x026ab8 (0xf64a bytes) 000667 1786962055.248612 MIRROR_RESYNC 2 mirrors 000670 1786962055.645812 mapread 0x0202f1 thru 0x02686e (0x657e bytes) 000675 1786962055.655941 read 0x029a5e thru 0x03593d (0xbee0 bytes) 000676 1786962055.666871 mapread 0x008e60 thru 0x01366d (0xa80e bytes) 000677 1786962055.690884 mapread 0x030106 thru 0x0353b2 (0x52ad bytes) 000683 1786962055.693643 mapwrite 0x02727a thru 0x02bacf (0x4856 bytes) 000685 1786962055.850446 read 0x03349c thru 0x03593d (0x24a2 bytes) 000691 1786962055.860847 mapread 0x01b981 thru 0x025aca (0xa14a bytes) 000692 1786962055.870451 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) 000696 1786962056.029716 MIRROR_EXTEND to 2 mirrors 000698 1786962056.655772 mapwrite 0x00d08f thru 0x0103ba (0x332c bytes) 000700 1786962056.858720 trunc from 0x03593e to 0x018db6 (0x1cb87 bytes) 000701 1786962056.892677 mapread 0x00f5f2 thru 0x018db6 (0x97c5 bytes) 000702 1786962056.906213 MIRROR_RESYNC 2 mirrors 000703 1786962057.783539 mapwrite 0x02134e thru 0x02f831 (0xe4e4 bytes) 000706 1786962057.974664 mapwrite 0x02d1da thru 0x031c38 (0x4a5f bytes) 000707 1786962058.064753 write 0x03fe04 thru 0x03ffff (0x1fc bytes) 000708 1786962058.067874 read 0x00779e thru 0x01505a (0xd8bd bytes) 000709 1786962058.095766 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) 000710 1786962058.311972 trunc from 0x040000 to 0x01b007 (0x24ff8 bytes) 000711 1786962058.340746 trunc from 0x01b008 to 0x00f1d5 (0xbe32 bytes) 000712 1786962058.371747 mapread 0x000a26 thru 0x00d340 (0xc91b bytes) 000713 1786962058.402298 trunc from 0x00f1d6 to 0x00665b (0x8b7a bytes) 000714 1786962058.418903 mapwrite 0x007c90 thru 0x017593 (0xf904 bytes) 000717 1786962058.538731 mapwrite 0x016ed5 thru 0x024fb0 (0xe0dc bytes) 000724 1786962058.690630 read 0x016a90 thru 0x01b807 (0x4d78 bytes) 000725 1786962058.694371 mapread 0x01b436 thru 0x024fb0 (0x9b7b bytes) 000726 1786962058.696509 trunc from 0x024fb1 to 0x005cff (0x1f2b1 bytes) 000727 1786962058.732313 trunc from 0x005d00 to 0x010370 (0xa671 bytes) 000728 1786962058.783399 trunc from 0x010371 to 0x00ef48 (0x1428 bytes) 000729 1786962058.850107 mapwrite 0x013867 thru 0x01b461 (0x7bfb bytes) 000731 1786962058.995449 read 0x010f9c thru 0x01b461 (0xa4c6 bytes) 000732 1786962058.996665 mapread 0x00b521 thru 0x00eaa5 (0x3585 bytes) 000733 1786962059.030401 trunc from 0x01b462 to 0x02dbf7 (0x12796 bytes) 000735 1786962059.061560 read 0x02ada7 thru 0x02dbf7 (0x2e51 bytes) 000736 1786962059.072480 mapread 0x028e28 thru 0x02dbf7 (0x4dd0 bytes) 000737 1786962059.073775 mapwrite 0x007ff7 thru 0x00a83e (0x2848 bytes) 000738 1786962059.122828 mapread 0x02a909 thru 0x02dbf7 (0x32ef bytes) 000739 1786962059.123969 write 0x0191ee thru 0x021fd0 (0x8de3 bytes) 000740 1786962059.126572 write 0x004056 thru 0x01109d (0xd048 bytes) 000741 1786962059.128916 mapwrite 0x01d9ae thru 0x01fa9c (0x20ef bytes) 000742 1786962059.189386 trunc from 0x02dbf8 to 0x039863 (0xbc6c bytes) 000743 1786962059.207747 mapwrite 0x0018f1 thru 0x007224 (0x5934 bytes) 000744 1786962059.272799 write 0x002897 thru 0x00e198 (0xb902 bytes) 000746 1786962059.278895 trunc from 0x039864 to 0x0371a7 (0x26bc bytes) 000747 1786962059.306595 mapwrite 0x0162a8 thru 0x0260e0 (0xfe39 bytes) 000748 1786962059.397267 mapwrite 0x014bcb thru 0x023999 (0xedcf bytes) 000752 1786962059.438384 mapread 0x0169b5 thru 0x020256 (0x98a2 bytes) 000754 1786962059.440521 read 0x000e0b thru 0x00f37e (0xe574 bytes) 000756 1786962059.442239 trunc from 0x0371a8 to 0x037eaf (0xd08 bytes) 000758 1786962059.480388 trunc from 0x037eb0 to 0x02b487 (0xca28 bytes) 000762 1786962059.512927 write 0x024b0b thru 0x025852 (0xd48 bytes) 000763 1786962059.515263 read 0x0164d0 thru 0x01d6a4 (0x71d5 bytes) 000764 1786962059.516896 write 0x03c265 thru 0x03f187 (0x2f23 bytes) 000766 1786962059.522245 mapread 0x009ef9 thru 0x00c216 (0x231e bytes) 000767 1786962059.524651 mapread 0x02adf1 thru 0x02edbc (0x3fcc bytes) 000768 1786962059.536838 write 0x034494 thru 0x03b63a (0x71a7 bytes) 000772 1786962059.549843 trunc from 0x03f188 to 0x00c614 (0x32b73 bytes) 000775 1786962059.579774 mapwrite 0x00647f thru 0x00f721 (0x92a3 bytes) 000776 1786962059.682638 read 0x00dc33 thru 0x00f721 (0x1aef bytes) 000777 1786962059.684023 mapwrite 0x0330dc thru 0x03ffff (0xcf24 bytes) 000779 1786962059.877889 read 0x032c86 thru 0x03f5d1 (0xc94c bytes) 000781 1786962059.881680 mapwrite 0x007fa4 thru 0x00ad51 (0x2dae bytes) 000782 1786962059.943657 mapwrite 0x0357ce thru 0x038089 (0x28bc bytes) 000783 1786962059.982946 write 0x02d741 thru 0x032eef (0x57af bytes) 000784 1786962059.987160 mapwrite 0x000709 thru 0x003fd4 (0x38cc bytes) 000788 1786962060.066641 trunc from 0x040000 to 0x018950 (0x276af bytes) 000789 1786962060.122353 mapread 0x001fd0 thru 0x00ddb1 (0xbde2 bytes) 000790 1786962060.125869 mapwrite 0x0354c6 thru 0x03fe0a (0xa945 bytes) 000792 1786962060.276231 read 0x02188e thru 0x02af87 (0x96fa bytes) 000793 1786962060.279718 read 0x019714 thru 0x0258bc (0xc1a9 bytes) 000795 1786962060.284779 mapread 0x03b455 thru 0x03fe0a (0x49b6 bytes) 000796 1786962060.287202 write 0x0165df thru 0x01830e (0x1d30 bytes) 000799 1786962060.291292 mapwrite 0x0073e1 thru 0x010748 (0x9368 bytes) 000800 1786962060.364884 mapwrite 0x027f95 thru 0x02e6ed (0x6759 bytes) 000801 1786962060.499695 mapread 0x016d96 thru 0x02696d (0xfbd8 bytes) 000802 1786962060.501132 trunc from 0x03fe0b to 0x0186be (0x2774c bytes) 000804 1786962060.551729 mapwrite 0x003314 thru 0x00d86b (0xa558 bytes) 000811 1786962060.708968 mapwrite 0x03e13c thru 0x03ffff (0x1ec4 bytes) 000812 1786962060.827510 write 0x01916b thru 0x01ec85 (0x5b1b bytes) 000815 1786962060.835657 mapread 0x032efe thru 0x03ffff (0xd102 bytes) 000817 1786962060.839548 read 0x006c86 thru 0x007f0c (0x1287 bytes) 000818 1786962060.840605 mapwrite 0x0312c9 thru 0x03d90b (0xc643 bytes) 000819 1786962060.909808 write 0x0011f2 thru 0x00c156 (0xaf65 bytes) 000820 1786962060.916257 read 0x02e47f thru 0x0358e5 (0x7467 bytes) 000824 1786962060.939013 MIRROR_EXTEND to 2 mirrors 000825 1786962062.270960 write 0x003194 thru 0x008786 (0x55f3 bytes) 000826 1786962062.437859 mapread 0x015c4b thru 0x018d9a (0x3150 bytes) 000828 1786962062.502595 trunc from 0x040000 to 0x027ac8 (0x18537 bytes) 000831 1786962062.557766 mapwrite 0x0075be thru 0x01731e (0xfd61 bytes) 000832 1786962062.737681 trunc from 0x027ac9 to 0x03aab1 (0x12fe9 bytes) 000834 1786962062.769832 mapread 0x019d87 thru 0x01bf6a (0x21e4 bytes) 000836 1786962062.784641 read 0x028685 thru 0x0352bc (0xcc38 bytes) 000837 1786962062.811447 mapwrite 0x006b4b thru 0x009290 (0x2746 bytes) 000842 1786962062.991170 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) 000843 1786962063.310939 read 0x0113d3 thru 0x015350 (0x3f7e bytes) 000845 1786962063.357390 mapread 0x022c64 thru 0x02b1e3 (0x8580 bytes) 000850 1786962063.419842 read 0x031906 thru 0x037c43 (0x633e bytes) 000852 1786962063.450635 trunc from 0x03aab2 to 0x00f766 (0x2b34b bytes) 000854 1786962063.493433 read 0x00723f thru 0x00de1e (0x6be0 bytes) 000857 1786962063.497362 write 0x034e88 thru 0x0387b4 (0x392d bytes) 000861 1786962063.503404 mapwrite 0x0033c8 thru 0x011801 (0xe43a bytes) 000862 1786962063.608820 write 0x00a993 thru 0x0153be (0xaa2c bytes) 000864 1786962063.617247 read 0x02fc38 thru 0x0350c2 (0x548b bytes) 000865 1786962063.629694 MIRROR_EXTEND to 2 mirrors 000866 1786962064.941457 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) 000867 1786962065.232079 mapwrite 0x037aaa thru 0x03ffff (0x8556 bytes) 000869 1786962065.533399 read 0x0048ca thru 0x00c876 (0x7fad bytes) 000870 1786962065.553447 read 0x00dc36 thru 0x01a53b (0xc906 bytes) 000871 1786962065.596533 trunc from 0x040000 to 0x037a1e (0x85e1 bytes) 000877 1786962065.655814 trunc from 0x037a1f to 0x03de9d (0x647f bytes) 000880 1786962065.741412 mapread 0x03192e thru 0x03de9d (0xc570 bytes) 000884 1786962065.815908 mapwrite 0x039070 thru 0x03ffff (0x6f90 bytes) 000887 1786962065.952456 read 0x031018 thru 0x03e35c (0xd345 bytes) 000888 1786962065.955043 mapwrite 0x003c2c thru 0x00add8 (0x71ad bytes) 000889 1786962066.054633 write 0x0347e4 thru 0x03e655 (0x9e72 bytes) 000891 1786962066.059557 trunc from 0x040000 to 0x00b8f2 (0x3470d bytes) 000894 1786962066.097021 mapwrite 0x02e99c thru 0x03b349 (0xc9ae bytes) 000898 1786962066.239229 mapwrite 0x01657d thru 0x01d59d (0x7021 bytes) 000899 1786962066.365698 MIRROR_EXTEND to 2 mirrors 000900 1786962067.030941 write 0x01c6c1 thru 0x020462 (0x3da2 bytes) 000901 1786962067.112375 trunc from 0x03b34a to 0x01cff7 (0x1e352 bytes) 000902 1786962067.152347 mapread 0x014601 thru 0x01cff7 (0x89f7 bytes) 000903 1786962067.170439 mapwrite 0x00f803 thru 0x0172ad (0x7aab bytes) 000905 1786962067.228894 mapwrite 0x0101d6 thru 0x01757e (0x73a9 bytes) 000906 1786962067.287995 MIRROR_RESYNC 2 mirrors 000908 1786962067.883830 write 0x00b623 thru 0x01903e (0xda1c bytes) 000910 1786962067.958900 mapread 0x00664e thru 0x00f1c5 (0x8b78 bytes) 000911 1786962067.977400 mapwrite 0x03fed5 thru 0x03ffff (0x12b bytes) 000912 1786962068.173422 write 0x01481f thru 0x01e92f (0xa111 bytes) 000913 1786962068.185951 trunc from 0x040000 to 0x03e3d6 (0x1c29 bytes) 000915 1786962068.207029 write 0x02a916 thru 0x038607 (0xdcf2 bytes) 000916 1786962068.214927 read 0x024328 thru 0x02fb78 (0xb851 bytes) 000924 1786962068.235641 read 0x01ef0a thru 0x02dace (0xebc5 bytes) 000925 1786962068.257608 trunc from 0x03e3d7 to 0x00546c (0x38f6a bytes) 000928 1786962068.320692 mapread 0x00452e thru 0x00546c (0xf3f bytes) 000929 1786962068.323628 read 0x003bcb thru 0x00546c (0x18a2 bytes) 000930 1786962068.337855 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x546d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x546d): Mirror 2: 0x972f5c1d (stale) CRC-32 checksum value for chunk [0, 0x546d): Mirror 1: 0x972f5c1d Mirror 3: 0x972f5c1d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x546d: skipped 000931 1786962069.258430 mapread 0x002f5d thru 0x004a81 (0x1b25 bytes) 000932 1786962069.300355 write 0x035e56 thru 0x0369e8 (0xb93 bytes) 000934 1786962069.389944 mapread 0x031164 thru 0x0322f3 (0x1190 bytes) 000936 1786962069.403967 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x369e9): Mirror 2: 0xed57901d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x369e9): Mirror 2: 0xed57901d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x369e9: skipped 000938 1786962069.739722 mapwrite 0x00f1a0 thru 0x00f318 (0x179 bytes) 000939 1786962069.809845 trunc from 0x0369e9 to 0x039794 (0x2dac bytes) 000940 1786962069.857452 mapwrite 0x0013bf thru 0x008604 (0x7246 bytes) 000941 1786962069.919600 read 0x022582 thru 0x02c6d9 (0xa158 bytes) 000943 1786962069.931579 read 0x012c8d thru 0x01f9a9 (0xcd1d bytes) 000944 1786962069.959773 read 0x032ffb thru 0x039794 (0x679a bytes) 000946 1786962069.962631 write 0x019044 thru 0x01cfc1 (0x3f7e bytes) 000948 1786962069.965439 write 0x000cde thru 0x002c3e (0x1f61 bytes) 000949 1786962069.966483 mapread 0x026ad9 thru 0x02803c (0x1564 bytes) 000950 1786962069.967498 trunc from 0x039795 to 0x00c3e9 (0x2d3ab bytes) 000954 1786962069.989684 mapwrite 0x002ed7 thru 0x008206 (0x5330 bytes) 000955 1786962070.042823 mapwrite 0x02b755 thru 0x02e14f (0x29fb bytes) 000956 1786962070.117911 write 0x018228 thru 0x0266a8 (0xe481 bytes) 000958 1786962070.121507 write 0x02785c thru 0x0375a1 (0xfd46 bytes) 000963 1786962070.127674 write 0x03d708 thru 0x03ffff (0x28f8 bytes) 000966 1786962070.130588 mapread 0x00b09d thru 0x00d511 (0x2475 bytes) 000968 1786962070.133391 mapwrite 0x019137 thru 0x02630f (0xd1d9 bytes) 000970 1786962070.202667 mapread 0x02903f thru 0x034de8 (0xbdaa bytes) 000974 1786962070.207949 trunc from 0x040000 to 0x020759 (0x1f8a6 bytes) 000975 1786962070.231512 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2075a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2075a): Mirror 2: 0x2254321c (stale) CRC-32 checksum value for chunk [0, 0x2075a): Mirror 1: 0xad456a18 Mirror 3: 0xad456a18 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2075a: skipped 000977 1786962071.070634 read 0x01b413 thru 0x01eb02 (0x36f0 bytes) 000978 1786962071.077087 mapwrite 0x017782 thru 0x024755 (0xcfd4 bytes) 000980 1786962071.241038 read 0x01a77e thru 0x02221b (0x7a9e bytes) 000981 1786962071.241723 mapread 0x00fd54 thru 0x019fa5 (0xa252 bytes) 000982 1786962071.253460 read 0x022c94 thru 0x024755 (0x1ac2 bytes) 000983 1786962071.254412 trunc from 0x024756 to 0x00e15d (0x165f8 bytes) 000984 1786962071.281564 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xe15e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe15e): Mirror 2: 0xa60d6e16 (stale) Mirror 3: 0xc8d2ff5a (stale) CRC-32 checksum value for chunk [0, 0xe15e): Mirror 1: 0xc8d2ff5a Mirror 4: 0xc8d2ff5a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe15e: skipped 000985 1786962072.318376 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xe15e): Mirror 3: 0xc8d2ff5a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe15e): Mirror 3: 0xc8d2ff5a (stale) CRC-32 checksum value for chunk [0, 0xe15e): Mirror 1: 0xc8d2ff5a Mirror 4: 0xc8d2ff5a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe15e: skipped 000986 1786962073.056322 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xe15e): Mirror 3: 0xc8d2ff5a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe15e): Mirror 3: 0xc8d2ff5a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe15e: skipped 000989 1786962073.490558 write 0x01401e thru 0x022aac (0xea8f bytes) 000990 1786962073.508656 mapwrite 0x0226fe thru 0x0274bb (0x4dbe bytes) 000991 1786962073.640824 trunc from 0x0274bc to 0x03c9f2 (0x15537 bytes) 000992 1786962073.672629 trunc from 0x03c9f3 to 0x008c26 (0x33dcc bytes) 000993 1786962073.705080 write 0x036559 thru 0x039fd1 (0x3a79 bytes) 000996 1786962073.714879 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) 000999 1786962073.952733 write 0x03d2ba thru 0x03ffff (0x2d46 bytes) 001000 1786962073.959507 mapread 0x027e2a thru 0x029a8a (0x1c61 bytes) All operations completed A-OK! PASS 70a (66s) == sanity-flr test 71: check mirror extend parent fid ==== 06:21:20 (1786962080) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 06:21:24 (1786962084) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.344441 s, 6.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ba:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318562 s, 3.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ba:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ba:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bb:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (12s) == sanity-flr test 200a: stress test ===================== 06:21:36 (1786962096) Starting client: oleg343-client.virtnet: -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre2 Starting client: oleg343-client.virtnet: -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1690080 .. Extending file size to 2378656 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4668896 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6227904 .. Extending file size to 3879744 .. Extending file size to 8201376 .. Extending file size to 6542272 .. Extending file size to 8665728 .. Extending file size to 6975488 .. Extending file size to 7614112 .. Extending file size to 7718912 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8818080 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9263904 .. Waiting 57366 57367 57370 57371 57372 done 192.168.203.143@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg343-client.virtnet /mnt/lustre2 (opts:) 192.168.203.143@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg343-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 06:22:58 (1786962178) Starting client: oleg343-client.virtnet: -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre2 Starting client: oleg343-client.virtnet: -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6289280 .. Extending file size to 3521664 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 3959840 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 4219552 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7876736 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8164608 .. done Extending file size to 8188928 .. Extending file size to 8336416 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8480608 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8225120 .. Extending file size to 9138688 .. Extending file size to 8328672 .. Extending file size to 8590208 .. Extending file size to 8842464 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 58746 58747 58748 58750 58751 failed 192.168.203.143@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg343-client.virtnet /mnt/lustre2 (opts:) 192.168.203.143@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg343-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 06:24:18 (1786962258) Starting client: oleg343-client.virtnet: -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0968627 s, 21.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.293157 s, 7.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xde2abdec (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.143@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg343-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 06:24:36 (1786962276) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.161936 s, 13.0 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 06:24:44 (1786962284) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f4:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } PASS 203 (10s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 06:24:54 (1786962294) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.396366 s, 5.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 06:25:04 (1786962304) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.216584 s, 9.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 06:25:13 (1786962313) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.202527 s, 10.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 06:25:46 (1786962346) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg343-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103418 s, 10.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } PASS 204d (67s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 06:26:53 (1786962413) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg343-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.408684 s, 5.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (60s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 06:27:53 (1786962473) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg343-server mds-ost sync done. Creating new pool test_204f oleg343-server: Pool lustre.test_204f created Adding targets to pool oleg343-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg343-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Updated after 2s: want 'lustre-OST0000_UUID lustre-OST0001_UUID ' got 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.206639 s, 10.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg343-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg343-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg343-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 06:29:14 (1786962554) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 06:29:23 (1786962563) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 06:29:31 (1786962571) oleg343-server: Pool lustre.test_206 created Adding targets to pool oleg343-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg343-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg343-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg343-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg343-server: Pool lustre.test_206 destroyed PASS 206 (24s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 06:29:55 (1786962595) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748937 s, 14.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461024 s, 22.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x105a9625 Mirror 3: 0x105a9625 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 06:30:06 (1786962606) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 06:30:11 (1786962611) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 06:30:14 (1786962614) Starting client: oleg343-client.virtnet: -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0752801 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2450000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2050000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2050000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2050000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xc70000 Read this (4 bytes): BBBB 192.168.203.143@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg343-client.virtnet /mnt/lustre2 (opts:) PASS 209a (34s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 06:30:48 (1786962648) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0415437 s, 197 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 06:30:56 (1786962656) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319862 s, 32.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x219 lmm_fid: [0x200000404:0x219:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x219 lmm_fid: [0x200000404:0x219:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834457 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-17 06:30:58.000000000 -0400 Modify: 2026-08-17 06:30:58.000000000 -0400 Change: 2026-08-17 06:30:59.000000000 -0400 Birth: 2026-08-17 06:30:58.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 06:31:03 (1786962663) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 06:31:06 (1786962666) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.482057 s, 85.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.12583 s, 32.6 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 17 06:31 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21b:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 17 06:31 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 06:31:13 (1786962673) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 06:31:20 (1786962680) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1762 sec =========== 06:31:29 (1786962689) === sanity-flr: start cleanup 06:31:30 (1786962690) === === sanity-flr: finish cleanup 06:31:36 (1786962696) ===