-----============= acceptance-small: sanity-flr ============----- Fri Jul 3 23:58:58 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg260-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 23:59:18 (1783137558) === oleg260-client.virtnet: executing check_config_client /mnt/lustre oleg260-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg260-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9631029ec000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9631029ec000.idle_timeout=debug disable quota as required oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 23:59:50 (1783137590) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7614] == sanity-flr test 0a: lfs mirror create with -N option == 23:59:52 (1783137592) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 00:00:10 (1783137610) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 00:00:13 (1783137613) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 00:00:17 (1783137617) PASS 0d (22s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 00:00:39 (1783137639) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 00:00:43 (1783137643) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 00:00:46 (1783137646) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9631029ec000.stats=clear osc.lustre-OST0001-osc-ffff9631029ec000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 00:00:54 (1783137654) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 00:01:06 (1783137666) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 00:01:14 (1783137674) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 00:01:17 (1783137677) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 00:01:25 (1783137685) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 00:01:34 (1783137694) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 00:01:40 (1783137700) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 00:01:43 (1783137703) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 00:01:50 (1783137710) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 00:02:01 (1783137721) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 00:02:08 (1783137728) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.277574 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141228 s, 7.4 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 00:02:18 (1783137738) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0738456 s, 28.4 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 00:02:25 (1783137745) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 00:02:35 (1783137755) Stopping /mnt/lustre-ost1 (opts:) on oleg260-server oleg260-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9631029ec000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9631029ec000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg260-server: oleg260-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg260-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg260-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg260-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg260-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg260-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg260-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg260-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg260-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg260-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg260-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg260-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg260-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg260-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg260-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg260-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg260-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg260-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg260-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg260-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg260-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg260-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg260-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg260-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg260-server oleg260-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9631029ec000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9631029ec000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg260-server: oleg260-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg260-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (128s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 00:04:44 (1783137884) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.440568 s, 16.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg260-server oleg260-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9631029ec000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9631029ec000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg260-server: oleg260-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg260-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 00:05:27 (1783137927) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg260-server oleg260-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9631029ec000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9631029ec000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg260-server: oleg260-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg260-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg260-server oleg260-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9631029ec000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9631029ec000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg260-server: oleg260-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg260-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (79s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 00:06:45 (1783138005) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.49617 s, 9.5 MB/s -rw-r--r-- 1 root root 52428800 Jul 4 00:06 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.070s user 0m0.001s sys 0m0.171s Stopping /mnt/lustre-ost1 (opts:) on oleg260-server oleg260-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9631029ec000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9631029ec000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.480s user 0m0.001s sys 0m0.178s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg260-server: oleg260-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg260-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg260-server oleg260-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9631029ec000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9631029ec000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.312s user 0m0.001s sys 0m0.178s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg260-server: oleg260-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg260-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg260-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9631029ec000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9631029ec000.ost_server_uuid in FULL state after 0 sec PASS 33b (109s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 00:08:35 (1783138115) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (6s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 00:08:40 (1783138120) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 00:08:44 (1783138124) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 00:08:47 (1783138127) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 00:08:54 (1783138134) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.606174 s, 20.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 00:09:07 (1783138147) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420723 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548954 s, 19.1 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 00:09:14 (1783138154) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 00:09:23 (1783138163) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000782823 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00177574 s, 577 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00162202 s, 631 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0019009 s, 539 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 00:09:32 (1783138172) create test files with size 15733809 .. 813f6f7335f8dafc0cab91638453f6e3 - 82b7f546288f5b26b04641a42564c047 - 053d6201568f8704b5f005f004e8d4ed - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 00:09:57 (1783138197) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 00:10:11 (1783138211) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 00:10:18 (1783138218) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892124 s, 11.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308584 s, 34.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbb:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xda:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 00:10:27 (1783138227) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.111444 s, 18.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.396344 s, 10.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.151907 s, 13.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 00:10:41 (1783138241) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 00:10:45 (1783138245) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 00:10:48 (1783138248) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458473 s, 22.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 00:10:56 (1783138256) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 00:11:00 (1783138260) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe1:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe1:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 00:11:10 (1783138270) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.312361 s, 33.6 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 00:11:21 (1783138281) llite.lustre-ffff963112f0f000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.276246 s, 38.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff963118757000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 00:11:36 (1783138296) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (47s) == sanity-flr test 46: Verify setstripe --copy option ==== 00:12:23 (1783138343) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (47s) == sanity-flr test 47: Verify mirror obj alloc =========== 00:13:10 (1783138390) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (5s) == sanity-flr test 48: Verify snapshot mirror ============ 00:13:15 (1783138395) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.402371 s, 7.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.153429 s, 20.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x4c3dbac2 (nosync,stale) Mirror 2: 0x4c3dbac2 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 72105e6c4a0147efa21cc79924157183 - mirror 1 checksum: c70d967c7f2d67834d1c6fc4f06950a9 - mirror 2 checksum: 72105e6c4a0147efa21cc79924157183 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-04T00:13:18-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-04T00:13:18-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 72105e6c4a0147efa21cc79924157183 - mirror 1 checksum: c70d967c7f2d67834d1c6fc4f06950a9 - mirror 2 checksum: c70d967c7f2d67834d1c6fc4f06950a9 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-04T00:13:18-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-04T00:13:18-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } PASS 48 (9s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 00:13:26 (1783138406) Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server 00:13:32 (1783138412) shut down facet: mds1 facet_host: oleg260-server facet_failover_host: oleg260-server Failover mds1 to oleg260-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 00:13:55 (1783138435) targets are mounted 00:13:55 (1783138435) facet_failover done oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456791 s, 23.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } PASS 50A (48s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 00:14:14 (1783138454) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 00:14:17 (1783138457) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 00:14:21 (1783138461) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00715629 s, 2.3 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 00:14:29 (1783138469) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 00:14:34 (1783138474) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (5s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 00:14:38 (1783138478) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 00:15:01 (1783138501) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 00:15:31 (1783138531) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 00:15:55 (1783138555) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 00:16:20 (1783138580) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 00:16:27 (1783138587) mirror create pid 53781 mirror split pid 53782 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53781 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53782 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 00:17:36 (1783138656) Seed set to 8658 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read skipping zero size read 000013 1783138659.017675 write 0x006339 thru 0x009c9a (0x3962 bytes) 000014 1783138659.024488 read 0x0009df thru 0x0082cf (0x78f1 bytes) 000016 1783138659.044171 mapwrite 0x0344cf thru 0x03ffff (0xbb31 bytes) 000019 1783138659.225983 mapread 0x013e21 thru 0x023496 (0xf676 bytes) truncating to largest ever: 0x2b380 000020 1783138659.275089 trunc from 0x040000 to 0x02b37f (0x14c80 bytes) 000021 1783138659.295373 write 0x03bdff thru 0x03ffff (0x4201 bytes) 000022 1783138659.298107 trunc from 0x040000 to 0x009140 (0x36ebf bytes) 000023 1783138659.332988 write 0x01fd0d thru 0x02264d (0x2941 bytes) 000024 1783138659.335475 mapread 0x020951 thru 0x02264d (0x1cfd bytes) 000028 1783138659.344920 read 0x01704a thru 0x01ccc3 (0x5c7a bytes) 000029 1783138659.356140 mapwrite 0x00eaf3 thru 0x018565 (0x9a73 bytes) 000030 1783138659.440625 write 0x0348f5 thru 0x036a21 (0x212d bytes) 000031 1783138659.444007 mapread 0x008a43 thru 0x01811e (0xf6dc bytes) 000032 1783138659.446522 mapwrite 0x02e74e thru 0x031546 (0x2df9 bytes) 000033 1783138659.514465 mapread 0x0096ed thru 0x017d45 (0xe659 bytes) 000036 1783138659.519367 mapread 0x025c3f thru 0x02c906 (0x6cc8 bytes) 000037 1783138659.522643 read 0x0285ba thru 0x02bb46 (0x358d bytes) 000039 1783138659.526369 mapwrite 0x00b248 thru 0x0103cd (0x5186 bytes) 000040 1783138659.606101 mapread 0x023942 thru 0x0319b3 (0xe072 bytes) 000041 1783138659.621511 MIRROR_EXTEND to 2 mirrors 000042 1783138660.700925 trunc from 0x036a22 to 0x0085f3 (0x2e42e bytes) 000043 1783138660.874616 mapwrite 0x02ec88 thru 0x034505 (0x587e bytes) 000045 1783138661.089646 MIRROR_RESYNC 2 mirrors 000046 1783138661.748020 mapwrite 0x018c8a thru 0x01ffee (0x7365 bytes) 000049 1783138661.903755 trunc from 0x034506 to 0x011c18 (0x228ed bytes) 000056 1783138661.934303 write 0x025fef thru 0x02b24d (0x525f bytes) 000057 1783138661.936505 write 0x00f8b7 thru 0x01df06 (0xe650 bytes) truncating to largest ever: 0x3e4dc 000059 1783138661.950489 trunc from 0x02b24e to 0x03e4db (0x1328e bytes) 000060 1783138661.975293 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) 000063 1783138662.210854 mapwrite 0x01628f thru 0x0204a0 (0xa212 bytes) 000064 1783138662.288608 trunc from 0x03e4dc to 0x001de5 (0x3c6f6 bytes) 000066 1783138662.306640 mapwrite 0x00e1fe thru 0x01a7eb (0xc5ee bytes) 000067 1783138662.421908 mapread 0x005908 thru 0x00d8e3 (0x7fdc bytes) 000068 1783138662.436444 write 0x037b37 thru 0x03ffff (0x84c9 bytes) 000070 1783138662.440683 read 0x02f2be thru 0x03497b (0x56be bytes) 000071 1783138662.450267 write 0x01b3ba thru 0x02869a (0xd2e1 bytes) 000072 1783138662.464492 write 0x02aefc thru 0x031380 (0x6485 bytes) 000073 1783138662.472605 read 0x0324fa thru 0x038f52 (0x6a59 bytes) 000076 1783138662.490379 MIRROR_EXTEND to 2 mirrors 000079 1783138663.312956 read 0x02c69c thru 0x039abe (0xd423 bytes) 000082 1783138663.331435 write 0x02efb6 thru 0x02f743 (0x78e bytes) 000084 1783138663.447604 trunc from 0x040000 to 0x0082fe (0x37d01 bytes) 000087 1783138663.517735 write 0x02bd0d thru 0x033cad (0x7fa1 bytes) 000088 1783138663.521057 read 0x00a431 thru 0x0189d6 (0xe5a6 bytes) 000089 1783138663.538143 mapread 0x0047d7 thru 0x01127b (0xcaa5 bytes) 000090 1783138663.540441 trunc from 0x033cae to 0x02f60c (0x46a1 bytes) 000092 1783138663.569977 mapwrite 0x02050d thru 0x02d116 (0xcc0a bytes) 000093 1783138663.637151 write 0x011c6d thru 0x021615 (0xf9a9 bytes) 000094 1783138663.638633 trunc from 0x02f60d to 0x0057f8 (0x29e14 bytes) 000095 1783138663.667715 trunc from 0x0057f9 to 0x033663 (0x2de6b bytes) 000098 1783138663.693967 mapwrite 0x00784d thru 0x0174aa (0xfc5e bytes) 000099 1783138663.871764 mapread 0x0218db thru 0x0274b0 (0x5bd6 bytes) 000101 1783138663.912306 write 0x026750 thru 0x02f094 (0x8945 bytes) 000103 1783138663.922036 write 0x00c761 thru 0x01348b (0x6d2b bytes) 000104 1783138663.931959 MIRROR_RESYNC 2 mirrors 000107 1783138664.830551 mapread 0x022435 thru 0x027e21 (0x59ed bytes) 000108 1783138664.851359 mapread 0x01f82b thru 0x026e93 (0x7669 bytes) 000110 1783138664.852574 mapwrite 0x03d7f5 thru 0x03ffff (0x280b bytes) 000111 1783138665.051586 mapread 0x004f76 thru 0x00d25d (0x82e8 bytes) 000112 1783138665.066805 mapwrite 0x021ea6 thru 0x02421e (0x2379 bytes) 000113 1783138665.114371 trunc from 0x040000 to 0x00755d (0x38aa2 bytes) 000115 1783138665.137415 mapread 0x0013d4 thru 0x00755d (0x618a bytes) 000116 1783138665.138774 read 0x003591 thru 0x00755d (0x3fcd bytes) 000117 1783138665.140328 trunc from 0x00755e to 0x01a4b9 (0x12f5c bytes) 000118 1783138665.162980 read 0x0074d5 thru 0x009ad8 (0x2604 bytes) 000119 1783138665.173003 read 0x00a46d thru 0x01472f (0xa2c3 bytes) 000123 1783138665.178841 write 0x01e038 thru 0x021938 (0x3901 bytes) 000125 1783138665.181367 trunc from 0x021939 to 0x026540 (0x4c08 bytes) 000126 1783138665.194571 write 0x02fe81 thru 0x03822b (0x83ab bytes) 000127 1783138665.196398 trunc from 0x03822c to 0x03166e (0x6bbd bytes) 000130 1783138665.218703 mapwrite 0x01113a thru 0x014182 (0x3049 bytes) 000131 1783138665.302089 mapwrite 0x03b585 thru 0x03ffff (0x4a7b bytes) 000133 1783138665.495183 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) 000135 1783138665.704994 read 0x005792 thru 0x0086ba (0x2f29 bytes) 000138 1783138665.727893 write 0x00d468 thru 0x01b6da (0xe273 bytes) 000139 1783138665.739008 trunc from 0x040000 to 0x00acf4 (0x3530b bytes) 000140 1783138665.766907 mapwrite 0x0182ac thru 0x0270af (0xee04 bytes) 000142 1783138665.990938 write 0x000224 thru 0x00cdb6 (0xcb93 bytes) 000143 1783138666.009305 mapwrite 0x01d007 thru 0x01d13a (0x134 bytes) 000144 1783138666.049413 write 0x032fca thru 0x03360b (0x642 bytes) 000145 1783138666.051366 write 0x0261e3 thru 0x0283d4 (0x21f2 bytes) 000146 1783138666.056029 write 0x031dd5 thru 0x03eec5 (0xd0f1 bytes) 000150 1783138666.063444 trunc from 0x03eec6 to 0x02bd11 (0x131b4 bytes) 000151 1783138666.081879 mapwrite 0x0079f6 thru 0x010560 (0x8b6b bytes) 000153 1783138666.131955 mapwrite 0x00d92c thru 0x0172b1 (0x9986 bytes) 000154 1783138666.202451 trunc from 0x02bd12 to 0x00e256 (0x1dabb bytes) 000157 1783138666.226139 write 0x037a37 thru 0x03ffff (0x85c9 bytes) 000164 1783138666.238596 read 0x0345d1 thru 0x03b6c7 (0x70f7 bytes) 000165 1783138666.247340 write 0x01ab42 thru 0x021fd5 (0x7494 bytes) 000167 1783138666.250363 trunc from 0x040000 to 0x016115 (0x29eea bytes) 000168 1783138666.289312 mapwrite 0x037211 thru 0x03d4ea (0x62da bytes) 000170 1783138666.411916 trunc from 0x03d4eb to 0x02f85f (0xdc8b bytes) 000172 1783138666.432515 mapwrite 0x031f7f thru 0x037e78 (0x5efa bytes) 000173 1783138666.577934 mapwrite 0x028450 thru 0x033a73 (0xb624 bytes) 000175 1783138666.698382 mapwrite 0x037a58 thru 0x03dc37 (0x61e0 bytes) 000177 1783138666.810521 mapread 0x02d8cf thru 0x036d24 (0x9456 bytes) 000178 1783138666.821028 MIRROR_EXTEND to 2 mirrors 000182 1783138667.879921 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) 000183 1783138668.016864 mapread 0x03a78c thru 0x03dc37 (0x34ac bytes) 000184 1783138668.044882 read 0x02121d thru 0x029cbd (0x8aa1 bytes) 000185 1783138668.057957 mapread 0x00bd66 thru 0x015110 (0x93ab bytes) 000186 1783138668.079061 write 0x004995 thru 0x00debb (0x9527 bytes) 000189 1783138668.108821 mapread 0x03a332 thru 0x03dc37 (0x3906 bytes) 000190 1783138668.119559 mapwrite 0x018062 thru 0x018e57 (0xdf6 bytes) 000191 1783138668.197398 mapread 0x002854 thru 0x00be52 (0x95ff bytes) 000193 1783138668.219801 trunc from 0x03dc38 to 0x020b06 (0x1d131 bytes) 000196 1783138668.235795 read 0x01dd71 thru 0x020b06 (0x2d96 bytes) 000197 1783138668.236790 write 0x03b719 thru 0x03e629 (0x2f11 bytes) 000199 1783138668.239267 trunc from 0x03e62a to 0x0077ad (0x36e7c bytes) 000200 1783138668.269416 mapread 0x0013ce thru 0x0077ad (0x63e0 bytes) 000202 1783138668.278653 MIRROR_EXTEND to 2 mirrors 000203 1783138669.117512 read 0x003cad thru 0x0077ad (0x3b01 bytes) 000205 1783138669.158233 MIRROR_EXTEND to 3 mirrors 000206 1783138670.038874 mapwrite 0x0190d0 thru 0x024f68 (0xbe99 bytes) 000207 1783138670.339797 mapread 0x023b9c thru 0x024f68 (0x13cd bytes) 000208 1783138670.340792 trunc from 0x024f69 to 0x039756 (0x147ee bytes) 000210 1783138670.384362 write 0x03b977 thru 0x03ffff (0x4689 bytes) 000213 1783138670.389045 write 0x020a58 thru 0x029800 (0x8da9 bytes) 000215 1783138670.400926 trunc from 0x040000 to 0x01dce6 (0x22319 bytes) 000216 1783138670.424760 read 0x01700e thru 0x019edf (0x2ed2 bytes) 000217 1783138670.425618 mapread 0x00a4aa thru 0x01682d (0xc384 bytes) 000218 1783138670.467993 read 0x009343 thru 0x00d8aa (0x4568 bytes) 000219 1783138670.478929 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1dce7): Mirror 2: 0xb7e1a5ef (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dce7): Mirror 2: 0xb7e1a5ef (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dce7: skipped 000220 1783138671.037221 trunc from 0x01dce7 to 0x01efec (0x1306 bytes) 000221 1783138671.063427 MIRROR_RESYNC 2 mirrors 000224 1783138671.721684 mapread 0x00e6fd thru 0x019672 (0xaf76 bytes) 000225 1783138671.769048 write 0x03a3ff thru 0x03ffff (0x5c01 bytes) 000226 1783138671.821266 read 0x0339fa thru 0x035343 (0x194a bytes) 000228 1783138671.828932 write 0x00b174 thru 0x019333 (0xe1c0 bytes) 000230 1783138671.839352 write 0x030c12 thru 0x03140a (0x7f9 bytes) 000232 1783138671.847957 read 0x035c75 thru 0x03b3f1 (0x577d bytes) 000233 1783138671.862210 write 0x02cfba thru 0x030eaa (0x3ef1 bytes) 000234 1783138671.867408 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4af93914 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x958ff212 Mirror 3: 0x958ff212 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000237 1783138672.662674 mapread 0x009e9a thru 0x016bc4 (0xcd2b bytes) 000238 1783138672.708003 write 0x03c983 thru 0x03ffff (0x367d bytes) 000239 1783138672.858360 mapread 0x00ba21 thru 0x011810 (0x5df0 bytes) 000240 1783138672.878785 mapwrite 0x038082 thru 0x03ffea (0x7f69 bytes) 000242 1783138672.934040 MIRROR_RESYNC 3 mirrors 000243 1783138673.494707 mapread 0x0060d9 thru 0x009dba (0x3ce2 bytes) 000246 1783138673.519713 trunc from 0x040000 to 0x03d300 (0x2cff bytes) 000247 1783138673.598367 mapwrite 0x032fd6 thru 0x0361f4 (0x321f bytes) 000248 1783138673.689729 write 0x005ead thru 0x01095c (0xaab0 bytes) 000249 1783138673.710946 mapread 0x0322fe thru 0x03d300 (0xb003 bytes) 000251 1783138673.738918 trunc from 0x03d301 to 0x028c9b (0x14665 bytes) 000252 1783138673.771526 trunc from 0x028c9c to 0x022ffb (0x5ca0 bytes) 000253 1783138673.791665 mapread 0x01d438 thru 0x022043 (0x4c0c bytes) 000254 1783138673.805231 write 0x00ca47 thru 0x0173af (0xa969 bytes) 000256 1783138673.807205 mapwrite 0x027d2d thru 0x028e3c (0x1110 bytes) 000257 1783138673.885615 trunc from 0x028e3d to 0x018b56 (0x102e6 bytes) 000259 1783138673.914015 write 0x00a81e thru 0x0151ce (0xa9b1 bytes) 000260 1783138673.915335 mapwrite 0x018603 thru 0x022ef3 (0xa8f1 bytes) 000261 1783138674.020698 read 0x00ca99 thru 0x00e442 (0x19aa bytes) 000265 1783138674.025970 mapwrite 0x038378 thru 0x03ffff (0x7c88 bytes) 000266 1783138674.121377 write 0x01f7b8 thru 0x02a3af (0xabf8 bytes) 000271 1783138674.140707 write 0x023689 thru 0x032781 (0xf0f9 bytes) 000272 1783138674.150944 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x51f1f08c (stale) Mirror 3: 0x51f1f08c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6f57401e Mirror 4: 0x6f57401e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000273 1783138674.965159 mapwrite 0x005dd5 thru 0x00785a (0x1a86 bytes) 000275 1783138675.232948 mapread 0x018d8e thru 0x01b9d6 (0x2c49 bytes) 000276 1783138675.252363 trunc from 0x040000 to 0x00979a (0x36865 bytes) 000278 1783138675.381388 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x979b): Mirror 3: 0x341267d6 (stale) Mirror 4: 0xa88633b5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x979b): Mirror 3: 0x341267d6 (stale) Mirror 4: 0xa88633b5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x979b: skipped 000279 1783138675.893751 mapwrite 0x017470 thru 0x01b0bf (0x3c50 bytes) 000281 1783138676.074689 mapread 0x01a4cf thru 0x01b0bf (0xbf1 bytes) 000283 1783138676.091596 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b0c0): Mirror 4: 0xf608ba95 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b0c0): Mirror 4: 0xf608ba95 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b0c0: skipped 000287 1783138676.390711 MIRROR_RESYNC 2 mirrors 000288 1783138677.440551 mapread 0x01906f thru 0x01b0bf (0x2051 bytes) 000292 1783138677.459837 write 0x002ce0 thru 0x007e8a (0x51ab bytes) 000293 1783138677.548046 write 0x00bd3c thru 0x018630 (0xc8f5 bytes) 000294 1783138677.557878 mapwrite 0x006f9b thru 0x010b6e (0x9bd4 bytes) 000297 1783138677.599737 read 0x002b8f thru 0x00ded9 (0xb34b bytes) 000298 1783138677.601221 mapread 0x013d56 thru 0x01b0bf (0x736a bytes) 000299 1783138677.602827 trunc from 0x01b0c0 to 0x01ed5e (0x3c9f bytes) 000304 1783138677.634839 read 0x01888d thru 0x01ed5e (0x64d2 bytes) 000306 1783138677.645607 read 0x00a866 thru 0x0107c6 (0x5f61 bytes) 000309 1783138677.651484 mapread 0x013fe4 thru 0x01ed5e (0xad7b bytes) 000310 1783138677.654889 mapread 0x002b63 thru 0x0061c4 (0x3662 bytes) 000311 1783138677.660884 MIRROR_RESYNC 2 mirrors 000312 1783138678.472313 write 0x024b66 thru 0x026a9a (0x1f35 bytes) 000313 1783138678.571811 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) 000315 1783138678.718797 write 0x029f0f thru 0x0336bc (0x97ae bytes) 000317 1783138678.722506 mapwrite 0x018f17 thru 0x019b3d (0xc27 bytes) 000320 1783138678.788625 trunc from 0x0336bd to 0x028a24 (0xac98 bytes) 000321 1783138678.825137 trunc from 0x028a25 to 0x0120bc (0x16968 bytes) 000322 1783138678.853782 mapread 0x00dfe8 thru 0x0120bc (0x40d5 bytes) 000323 1783138678.865575 mapwrite 0x003ed7 thru 0x00a15f (0x6289 bytes) 000324 1783138678.939208 read 0x007f46 thru 0x010314 (0x83cf bytes) 000325 1783138678.940156 mapwrite 0x001b02 thru 0x010cca (0xf1c9 bytes) 000328 1783138678.994217 mapread 0x000698 thru 0x00c20b (0xbb74 bytes) 000329 1783138679.001786 write 0x004515 thru 0x00a7fc (0x62e8 bytes) 000331 1783138679.004788 write 0x00e3f5 thru 0x01d5a6 (0xf1b2 bytes) 000333 1783138679.008858 write 0x01eb04 thru 0x026fad (0x84aa bytes) 000337 1783138679.014675 write 0x032bb8 thru 0x038cee (0x6137 bytes) 000339 1783138679.017870 mapread 0x01d23e thru 0x0288da (0xb69d bytes) 000340 1783138679.027442 write 0x028661 thru 0x035f39 (0xd8d9 bytes) 000342 1783138679.030783 write 0x010122 thru 0x01a817 (0xa6f6 bytes) 000343 1783138679.032879 write 0x0106d8 thru 0x011add (0x1406 bytes) 000344 1783138679.034447 write 0x011116 thru 0x019c94 (0x8b7f bytes) 000346 1783138679.037798 read 0x016f7b thru 0x026098 (0xf11e bytes) 000347 1783138679.040031 trunc from 0x038cef to 0x021fd0 (0x16d1e bytes) 000348 1783138679.064611 MIRROR_EXTEND to 2 mirrors 000350 1783138679.691859 mapwrite 0x033ca0 thru 0x034613 (0x974 bytes) 000351 1783138679.828585 mapwrite 0x0263f2 thru 0x02a775 (0x4384 bytes) 000353 1783138679.886539 trunc from 0x034614 to 0x015594 (0x1f07f bytes) 000356 1783138679.930848 read 0x005d6c thru 0x00a16a (0x43ff bytes) 000357 1783138679.951646 trunc from 0x015595 to 0x02c0f4 (0x16b60 bytes) 000359 1783138680.005775 trunc from 0x02c0f5 to 0x024bbe (0x7536 bytes) 000361 1783138680.025975 write 0x00f78e thru 0x0172b0 (0x7b23 bytes) 000362 1783138680.036500 read 0x01c0f2 thru 0x01e087 (0x1f96 bytes) 000363 1783138680.044385 read 0x00e40d thru 0x0174d3 (0x90c7 bytes) 000364 1783138680.045062 trunc from 0x024bbf to 0x02530f (0x751 bytes) 000366 1783138680.065420 write 0x0080bb thru 0x009330 (0x1276 bytes) 000367 1783138680.067038 trunc from 0x025310 to 0x012f3c (0x123d3 bytes) 000369 1783138680.086758 write 0x021878 thru 0x02285a (0xfe3 bytes) 000370 1783138680.088730 trunc from 0x02285b to 0x023a00 (0x11a6 bytes) 000372 1783138680.106829 write 0x00c792 thru 0x01879f (0xc00e bytes) 000374 1783138680.114948 write 0x02efad thru 0x03c0c8 (0xd11c bytes) 000376 1783138680.118562 write 0x01fba6 thru 0x02b974 (0xbdcf bytes) 000377 1783138680.129678 mapwrite 0x035935 thru 0x03ffff (0xa6cb bytes) 000379 1783138680.203887 trunc from 0x040000 to 0x01f725 (0x208da bytes) 000381 1783138680.221761 mapwrite 0x01c115 thru 0x02673c (0xa628 bytes) 000382 1783138680.320834 write 0x01e109 thru 0x025413 (0x730b bytes) 000385 1783138680.326639 mapwrite 0x034676 thru 0x03901a (0x49a5 bytes) 000387 1783138680.417541 mapread 0x02e860 thru 0x03901a (0xa7bb bytes) 000388 1783138680.446747 mapread 0x00e80c thru 0x0149af (0x61a4 bytes) 000389 1783138680.447967 trunc from 0x03901b to 0x031146 (0x7ed4 bytes) 000392 1783138680.469563 read 0x0188d0 thru 0x027687 (0xedb8 bytes) 000393 1783138680.481920 read 0x006609 thru 0x0097f5 (0x31ed bytes) 000395 1783138680.483735 mapread 0x0142be thru 0x01f10a (0xae4d bytes) 000396 1783138680.485006 read 0x001699 thru 0x00a78d (0x90f5 bytes) 000398 1783138680.486825 read 0x009e2c thru 0x01535c (0xb531 bytes) 000401 1783138680.489128 mapwrite 0x00212b thru 0x009367 (0x723d bytes) 000402 1783138680.529284 mapread 0x006384 thru 0x011fd1 (0xbc4e bytes) 000404 1783138680.531257 read 0x02ef93 thru 0x031146 (0x21b4 bytes) 000406 1783138680.537424 MIRROR_RESYNC 2 mirrors 000408 1783138680.984761 MIRROR_EXTEND to 3 mirrors 000409 1783138681.565989 mapwrite 0x01d938 thru 0x025753 (0x7e1c bytes) 000411 1783138681.773442 mapread 0x007cc9 thru 0x00a43e (0x2776 bytes) 000412 1783138681.787802 read 0x01d868 thru 0x02509c (0x7835 bytes) 000414 1783138681.790820 write 0x0088b9 thru 0x00d966 (0x50ae bytes) 000416 1783138681.794846 write 0x003f9d thru 0x00f11f (0xb183 bytes) 000417 1783138681.803011 mapread 0x029987 thru 0x031146 (0x77c0 bytes) 000418 1783138681.817999 read 0x000496 thru 0x00d851 (0xd3bc bytes) 000419 1783138681.834640 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31147): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31147): Mirror 2: 0xa2cfcf6 (stale) Mirror 3: 0xa2cfcf6 (stale) CRC-32 checksum value for chunk [0, 0x31147): Mirror 1: 0xbbbbba29 Mirror 4: 0xbbbbba29 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31147: skipped 000421 1783138682.763891 MIRROR_RESYNC 4 mirrors 000422 1783138683.400803 mapwrite 0x014911 thru 0x02212e (0xd81e bytes) 000423 1783138683.563261 MIRROR_RESYNC 4 mirrors 000424 1783138684.348452 MIRROR_RESYNC 4 mirrors 000425 1783138684.731809 MIRROR_RESYNC 4 mirrors 000426 1783138685.271272 read 0x02f290 thru 0x031146 (0x1eb7 bytes) 000427 1783138685.293141 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) 000428 1783138685.707600 trunc from 0x031147 to 0x02b363 (0x5de3 bytes) 000430 1783138685.857619 write 0x013567 thru 0x014f34 (0x19ce bytes) 000431 1783138685.872836 read 0x018f6a thru 0x021b04 (0x8b9b bytes) 000441 1783138685.906764 trunc from 0x02b364 to 0x03cd0b (0x119a8 bytes) 000442 1783138685.969146 read 0x018302 thru 0x01e66d (0x636c bytes) 000443 1783138685.969833 write 0x0331b1 thru 0x03bc02 (0x8a52 bytes) 000444 1783138685.992438 mapread 0x023597 thru 0x026b27 (0x3591 bytes) 000446 1783138685.999552 write 0x0119f6 thru 0x0191dd (0x77e8 bytes) 000447 1783138686.004474 read 0x0387bb thru 0x03cd0b (0x4551 bytes) 000448 1783138686.013407 write 0x036bd9 thru 0x03ffff (0x9427 bytes) 000450 1783138686.020983 trunc from 0x040000 to 0x00631d (0x39ce2 bytes) 000451 1783138686.064851 trunc from 0x00631e to 0x02f059 (0x28d3c bytes) 000452 1783138686.115545 mapread 0x007e6b thru 0x017489 (0xf61f bytes) 000454 1783138686.128537 mapread 0x0288c6 thru 0x02f059 (0x6794 bytes) 000455 1783138686.141483 MIRROR_RESYNC 3 mirrors 000457 1783138687.295895 mapread 0x01c854 thru 0x01e2dc (0x1a89 bytes) 000458 1783138687.317653 MIRROR_RESYNC 3 mirrors 000462 1783138687.681781 write 0x0078f3 thru 0x00a5a1 (0x2caf bytes) 000466 1783138687.818139 mapread 0x005598 thru 0x00b1d3 (0x5c3c bytes) 000470 1783138687.865891 write 0x008c04 thru 0x0154f2 (0xc8ef bytes) 000473 1783138687.880632 write 0x023146 thru 0x030df5 (0xdcb0 bytes) 000476 1783138687.896886 read 0x020f53 thru 0x023424 (0x24d2 bytes) 000477 1783138687.905061 mapread 0x00737b thru 0x016280 (0xef06 bytes) 000478 1783138687.906769 write 0x02b7ec thru 0x03399e (0x81b3 bytes) 000481 1783138687.911723 write 0x01d77e thru 0x01fb47 (0x23ca bytes) 000483 1783138687.914744 mapwrite 0x01403c thru 0x02060d (0xc5d2 bytes) 000484 1783138687.975648 trunc from 0x03399f to 0x03c51b (0x8b7d bytes) 000487 1783138688.033527 read 0x03ac3f thru 0x03c51b (0x18dd bytes) 000488 1783138688.049426 trunc from 0x03c51c to 0x007798 (0x34d83 bytes) 000490 1783138688.082723 trunc from 0x007799 to 0x02483c (0x1d0a4 bytes) 000491 1783138688.106016 mapread 0x002392 thru 0x011aa9 (0xf718 bytes) 000492 1783138688.173697 trunc from 0x02483d to 0x018bcf (0xbc6d bytes) 000493 1783138688.197633 MIRROR_RESYNC 3 mirrors 000494 1783138689.065441 mapread 0x001327 thru 0x008e9e (0x7b78 bytes) 000497 1783138689.123306 MIRROR_RESYNC 3 mirrors 000498 1783138689.703791 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) 000499 1783138690.096773 trunc from 0x018bd0 to 0x033a7e (0x1aeaf bytes) 000500 1783138690.257735 mapwrite 0x0093e8 thru 0x011a7c (0x8695 bytes) 000501 1783138690.361729 mapwrite 0x019c35 thru 0x01ba1d (0x1de9 bytes) 000502 1783138690.430152 mapwrite 0x007980 thru 0x015a2d (0xe0ae bytes) 000505 1783138690.563787 trunc from 0x033a7f to 0x027427 (0xc657 bytes) 000507 1783138690.606524 write 0x024ddf thru 0x02f303 (0xa525 bytes) 000508 1783138690.637603 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2f304): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f304): Mirror 2: 0xb57489c2 (stale) CRC-32 checksum value for chunk [0, 0x2f304): Mirror 1: 0x4c994cd5 Mirror 3: 0x4c994cd5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f304: skipped 000510 1783138692.024562 trunc from 0x02f304 to 0x03c0d3 (0xcdd0 bytes) 000511 1783138692.170351 read 0x00f40f thru 0x01b598 (0xc18a bytes) 000512 1783138692.187818 read 0x00e8d2 thru 0x011515 (0x2c44 bytes) 000513 1783138692.215850 mapread 0x03b30f thru 0x03c0d3 (0xdc5 bytes) 000515 1783138692.217987 write 0x03df07 thru 0x03ffff (0x20f9 bytes) 000516 1783138692.220307 trunc from 0x040000 to 0x03a627 (0x59d8 bytes) 000517 1783138692.242807 write 0x01ce42 thru 0x02224f (0x540e bytes) 000519 1783138692.246017 read 0x02c803 thru 0x0310a3 (0x48a1 bytes) 000520 1783138692.247333 read 0x038cb2 thru 0x03a627 (0x1976 bytes) 000522 1783138692.248892 write 0x010ba7 thru 0x01d2aa (0xc704 bytes) 000525 1783138692.253631 mapread 0x031152 thru 0x03a627 (0x94d6 bytes) 000526 1783138692.255690 trunc from 0x03a628 to 0x01ca5d (0x1dbca bytes) 000527 1783138692.284777 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1ca5e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ca5e): Mirror 2: 0x6dce85de (stale) Mirror 3: 0x82e323a8 (stale) CRC-32 checksum value for chunk [0, 0x1ca5e): Mirror 1: 0x2a6733ed Mirror 4: 0x2a6733ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ca5e: skipped 000528 1783138693.113651 read 0x007595 thru 0x00baa1 (0x450d bytes) 000530 1783138693.135580 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1ca5e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ca5e): Mirror 2: 0x6dce85de (stale) Mirror 3: 0x82e323a8 (stale) CRC-32 checksum value for chunk [0, 0x1ca5e): Mirror 1: 0x2a6733ed Mirror 4: 0x2a6733ed Mirror 5: 0x2a6733ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ca5e: skipped 000532 1783138694.012992 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ca5e): Mirror 2: 0x6dce85de (stale) Mirror 3: 0x82e323a8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ca5e): Mirror 2: 0x6dce85de (stale) Mirror 3: 0x82e323a8 (stale) CRC-32 checksum value for chunk [0, 0x1ca5e): Mirror 1: 0x2a6733ed Mirror 4: 0x2a6733ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ca5e: skipped 000533 1783138694.543211 mapread 0x00951b thru 0x014356 (0xae3c bytes) 000534 1783138694.634774 read 0x0043a1 thru 0x00a4ee (0x614e bytes) 000536 1783138694.636292 mapwrite 0x03d10f thru 0x03ffff (0x2ef1 bytes) 000540 1783138694.897859 mapread 0x037ce5 thru 0x03dae8 (0x5e04 bytes) 000542 1783138694.907519 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe88d1a1 (stale) Mirror 3: 0x6bb0e5f0 (stale) Mirror 4: 0xa86d0797 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa30b6409 Mirror 5: 0xa30b6409 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000543 1783138695.675857 mapwrite 0x011ddf thru 0x01d0ea (0xb30c bytes) 000544 1783138695.843111 trunc from 0x040000 to 0x02a33d (0x15cc2 bytes) 000545 1783138695.869737 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a33e): Mirror 2: 0x8c3fa698 (stale) Mirror 3: 0xd40088f3 (stale) Mirror 5: 0x9080dc1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a33e): Mirror 2: 0x8c3fa698 (stale) Mirror 3: 0xd40088f3 (stale) Mirror 5: 0x9080dc1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a33e: skipped 000549 1783138696.202455 write 0x0108ad thru 0x01b15c (0xa8b0 bytes) 000553 1783138696.216511 write 0x02e26e thru 0x034666 (0x63f9 bytes) 000554 1783138696.217979 trunc from 0x034667 to 0x022cf0 (0x11976 bytes) 000558 1783138696.243285 MIRROR_RESYNC 4 mirrors 000560 1783138697.035465 mapread 0x0052cd thru 0x00d2b1 (0x7fe5 bytes) 000563 1783138697.055182 write 0x008fb5 thru 0x014979 (0xb9c5 bytes) 000564 1783138697.109388 mapwrite 0x032b05 thru 0x038376 (0x5872 bytes) 000565 1783138697.178918 mapread 0x02be22 thru 0x02da5f (0x1c3e bytes) 000566 1783138697.180423 write 0x03574c thru 0x03ffff (0xa8b4 bytes) 000567 1783138697.182241 write 0x00e8fe thru 0x018979 (0xa07c bytes) 000568 1783138697.188557 read 0x022675 thru 0x0283a7 (0x5d33 bytes) 000569 1783138697.195959 read 0x021497 thru 0x02b4f4 (0xa05e bytes) 000570 1783138697.206045 write 0x00758b thru 0x0109d2 (0x9448 bytes) 000571 1783138697.207992 mapwrite 0x02079e thru 0x021f1b (0x177e bytes) 000572 1783138697.259651 mapwrite 0x03b9f3 thru 0x03f66c (0x3c7a bytes) 000573 1783138697.306602 MIRROR_RESYNC 4 mirrors 000576 1783138697.971568 mapwrite 0x017859 thru 0x01b671 (0x3e19 bytes) 000578 1783138698.127730 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xbd15d62e (stale) Mirror 3: 0xbd15d62e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xbd15d62e (stale) Mirror 3: 0xbd15d62e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000581 1783138698.521960 read 0x013a59 thru 0x0185ad (0x4b55 bytes) 000582 1783138698.534385 write 0x0150ed thru 0x0195a7 (0x44bb bytes) 000584 1783138698.541680 write 0x02808c thru 0x033a60 (0xb9d5 bytes) 000588 1783138698.563295 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xbd15d62e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xbd15d62e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000589 1783138698.889887 read 0x029074 thru 0x037908 (0xe895 bytes) 000591 1783138698.912705 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) 000592 1783138699.050712 trunc from 0x040000 to 0x019d54 (0x262ab bytes) 000594 1783138699.101575 trunc from 0x019d55 to 0x00127f (0x18ad5 bytes) 000595 1783138699.123559 mapwrite 0x006664 thru 0x00bd11 (0x56ae bytes) 000598 1783138699.217541 trunc from 0x00bd12 to 0x01883e (0xcb2d bytes) 000599 1783138699.240406 write 0x01a3b1 thru 0x020aac (0x66fc bytes) 000600 1783138699.243750 mapwrite 0x00e1ed thru 0x01e1dd (0xfff1 bytes) 000604 1783138699.318403 write 0x01ad76 thru 0x01ce03 (0x208e bytes) 000605 1783138699.319716 mapread 0x003bc6 thru 0x006a57 (0x2e92 bytes) 000606 1783138699.324622 trunc from 0x020aad to 0x01b3db (0x56d1 bytes) 000610 1783138699.345512 trunc from 0x01b3dc to 0x00c495 (0xef46 bytes) 000616 1783138699.374915 read 0x00401c thru 0x00c495 (0x847a bytes) 000617 1783138699.375980 read 0x002bcd thru 0x003bb2 (0xfe6 bytes) 000619 1783138699.377729 trunc from 0x00c496 to 0x027c0a (0x1b775 bytes) 000620 1783138699.391590 write 0x03524f thru 0x03ffff (0xadb1 bytes) 000623 1783138699.396424 mapread 0x01f60b thru 0x0279f1 (0x83e7 bytes) 000624 1783138699.407786 mapread 0x031164 thru 0x039d69 (0x8c06 bytes) 000625 1783138699.409727 read 0x026013 thru 0x030661 (0xa64f bytes) 000626 1783138699.410669 mapwrite 0x02238f thru 0x02284d (0x4bf bytes) 000629 1783138699.462664 read 0x0118ae thru 0x020812 (0xef65 bytes) 000630 1783138699.464360 trunc from 0x040000 to 0x02e746 (0x118b9 bytes) 000631 1783138699.493550 trunc from 0x02e747 to 0x022d8e (0xb9b8 bytes) 000634 1783138699.523009 mapwrite 0x0156e9 thru 0x02069b (0xafb3 bytes) 000636 1783138699.574501 read 0x00d278 thru 0x019d89 (0xcb12 bytes) 000638 1783138699.576781 mapread 0x00c2a6 thru 0x01a5c1 (0xe31c bytes) 000640 1783138699.582571 trunc from 0x022d8f to 0x00a04c (0x18d42 bytes) 000641 1783138699.603485 trunc from 0x00a04d to 0x021671 (0x17625 bytes) 000642 1783138699.631489 mapread 0x01408d thru 0x016c4f (0x2bc3 bytes) 000643 1783138699.644017 trunc from 0x021672 to 0x01cadd (0x4b94 bytes) 000645 1783138699.664322 read 0x017be9 thru 0x01cadd (0x4ef5 bytes) 000646 1783138699.670305 MIRROR_EXTEND to 2 mirrors 000653 1783138700.220468 MIRROR_EXTEND to 3 mirrors 000654 1783138700.853538 mapread 0x013c74 thru 0x01cadd (0x8e6a bytes) 000655 1783138700.880187 read 0x011f59 thru 0x012d9d (0xe45 bytes) 000656 1783138700.883906 write 0x01daef thru 0x025a02 (0x7f14 bytes) 000657 1783138700.968807 mapread 0x0128f8 thru 0x014db7 (0x24c0 bytes) 000658 1783138700.987959 write 0x013313 thru 0x022dcd (0xfabb bytes) 000661 1783138700.993576 mapwrite 0x023732 thru 0x023d45 (0x614 bytes) 000662 1783138701.071019 write 0x011634 thru 0x0152d1 (0x3c9e bytes) 000663 1783138701.072194 mapwrite 0x02012b thru 0x02938b (0x9261 bytes) 000665 1783138701.148269 read 0x005512 thru 0x0099c4 (0x44b3 bytes) 000666 1783138701.158113 trunc from 0x02938c to 0x0057a9 (0x23be2 bytes) 000667 1783138701.180573 trunc from 0x0057aa to 0x03a463 (0x34cba bytes) 000669 1783138701.197710 read 0x011b6c thru 0x01a73f (0x8bd4 bytes) 000670 1783138701.234291 mapread 0x020f4b thru 0x02cf43 (0xbff9 bytes) 000671 1783138701.239975 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a464): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a464): Mirror 2: 0xd1d6c668 (stale) Mirror 3: 0xd1d6c668 (stale) CRC-32 checksum value for chunk [0, 0x3a464): Mirror 1: 0x5e85570e Mirror 4: 0x5e85570e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a464: skipped 000673 1783138702.169233 mapwrite 0x026ea0 thru 0x02b6cf (0x4830 bytes) 000675 1783138702.430619 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a464): Mirror 2: 0xd1d6c668 (stale) Mirror 4: 0x5e85570e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a464): Mirror 2: 0xd1d6c668 (stale) Mirror 4: 0x5e85570e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a464: skipped 000676 1783138702.696931 mapwrite 0x03e1f7 thru 0x03ffff (0x1e09 bytes) 000677 1783138702.906463 trunc from 0x040000 to 0x02eb45 (0x114ba bytes) 000678 1783138702.928773 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2eb46): Mirror 4: 0x4c47f638 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2eb46): Mirror 4: 0x4c47f638 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2eb46: skipped 000679 1783138703.208875 trunc from 0x02eb46 to 0x0007bd (0x2e388 bytes) 000680 1783138703.226652 write 0x02f869 thru 0x03f7fa (0xff92 bytes) 000681 1783138703.229971 write 0x025ed8 thru 0x029d75 (0x3e9e bytes) 000683 1783138703.240651 write 0x0087c0 thru 0x017742 (0xef83 bytes) 000684 1783138703.249780 read 0x013b27 thru 0x0148e1 (0xdbb bytes) 000685 1783138703.250664 mapwrite 0x0142de thru 0x018f5c (0x4c7f bytes) 000686 1783138703.303680 trunc from 0x03f7fb to 0x01a5f0 (0x2520a bytes) 000687 1783138703.324516 trunc from 0x01a5f1 to 0x00d7a8 (0xce48 bytes) 000689 1783138703.341694 trunc from 0x00d7a9 to 0x020fe3 (0x1383b bytes) 000690 1783138703.356461 mapwrite 0x021a94 thru 0x022bbc (0x1129 bytes) 000692 1783138703.429541 mapwrite 0x031d3a thru 0x038410 (0x66d7 bytes) 000695 1783138703.508896 mapwrite 0x03bfed thru 0x03dd7e (0x1d92 bytes) 000697 1783138703.575803 mapwrite 0x00817b thru 0x011591 (0x9417 bytes) 000698 1783138703.642683 read 0x00f2cf thru 0x01904a (0x9d7c bytes) 000700 1783138703.658880 write 0x01c12f thru 0x02b28d (0xf15f bytes) 000701 1783138703.661379 trunc from 0x03dd7f to 0x002a04 (0x3b37a bytes) 000703 1783138703.682988 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) 000705 1783138703.792607 mapwrite 0x029438 thru 0x03442c (0xaff5 bytes) 000706 1783138703.865598 write 0x009aa1 thru 0x01587d (0xbddd bytes) 000707 1783138703.881662 write 0x00c38a thru 0x01abcd (0xe844 bytes) 000708 1783138703.883462 mapwrite 0x019c1e thru 0x023a2b (0x9e0e bytes) 000713 1783138703.933297 read 0x005e5c thru 0x00f229 (0x93ce bytes) 000714 1783138703.952138 read 0x00dbcf thru 0x016d8c (0x91be bytes) 000715 1783138703.952830 read 0x009dcc thru 0x01575c (0xb991 bytes) 000716 1783138703.953510 write 0x039966 thru 0x03bc01 (0x229c bytes) 000718 1783138703.974314 MIRROR_EXTEND to 2 mirrors 000719 1783138704.663459 read 0x024ef9 thru 0x027f7b (0x3083 bytes) 000720 1783138704.667936 mapwrite 0x01fb21 thru 0x0283ad (0x888d bytes) 000721 1783138704.847530 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) 000723 1783138705.063000 write 0x003ece thru 0x005e72 (0x1fa5 bytes) 000724 1783138705.075659 read 0x001136 thru 0x0067d4 (0x569f bytes) 000726 1783138705.084743 mapwrite 0x002420 thru 0x00b70d (0x92ee bytes) 000727 1783138705.158202 mapwrite 0x030664 thru 0x0365fc (0x5f99 bytes) 000731 1783138705.220289 read 0x0089ea thru 0x0090be (0x6d5 bytes) 000733 1783138705.222998 trunc from 0x03bc02 to 0x008559 (0x336a8 bytes) 000734 1783138705.242550 mapwrite 0x03244c thru 0x034543 (0x20f8 bytes) 000735 1783138705.304518 trunc from 0x034544 to 0x02fb6b (0x49d8 bytes) 000736 1783138705.326653 write 0x028af6 thru 0x02dace (0x4fd9 bytes) 000741 1783138705.330267 read 0x0117e7 thru 0x01aa0a (0x9224 bytes) 000744 1783138705.332776 write 0x006061 thru 0x00f749 (0x96e9 bytes) 000745 1783138705.336701 mapwrite 0x024da2 thru 0x0278ce (0x2b2d bytes) 000746 1783138705.415157 trunc from 0x02fb6c to 0x03bda9 (0xc23e bytes) 000747 1783138705.443437 write 0x03979a thru 0x03ffff (0x6866 bytes) 000750 1783138705.466693 mapread 0x01e46c thru 0x021649 (0x31de bytes) 000751 1783138705.470442 trunc from 0x040000 to 0x02de4f (0x121b0 bytes) 000755 1783138705.507621 mapwrite 0x01cfa6 thru 0x022293 (0x52ee bytes) 000756 1783138705.565695 read 0x0041c3 thru 0x008251 (0x408f bytes) 000757 1783138705.574046 mapwrite 0x00fb62 thru 0x0103c9 (0x868 bytes) 000762 1783138705.616052 read 0x003f05 thru 0x012651 (0xe74d bytes) 000763 1783138705.622331 MIRROR_EXTEND to 2 mirrors 000764 1783138706.422641 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) 000765 1783138706.595616 write 0x015290 thru 0x01b547 (0x62b8 bytes) 000766 1783138706.611670 trunc from 0x02de50 to 0x007039 (0x26e16 bytes) 000767 1783138706.628473 write 0x009bf3 thru 0x00fc5d (0x606b bytes) 000768 1783138706.631228 read 0x007e2a thru 0x0086e2 (0x8b9 bytes) 000770 1783138706.636949 mapwrite 0x00d587 thru 0x01075a (0x31d4 bytes) 000774 1783138706.733062 write 0x036cca thru 0x036f2b (0x262 bytes) 000775 1783138706.735019 trunc from 0x036f2c to 0x01ec37 (0x182f4 bytes) 000776 1783138706.755984 mapwrite 0x0289e7 thru 0x02a58c (0x1ba6 bytes) 000777 1783138706.869587 write 0x03dfd8 thru 0x03ffff (0x2028 bytes) 000778 1783138706.870776 read 0x026fbe thru 0x02af35 (0x3f78 bytes) 000779 1783138706.872765 read 0x012f15 thru 0x01fa31 (0xcb1d bytes) 000780 1783138706.906157 write 0x037582 thru 0x039533 (0x1fb2 bytes) 000781 1783138706.907183 read 0x02de32 thru 0x02e603 (0x7d2 bytes) 000783 1783138706.915563 mapwrite 0x01c11b thru 0x01ec00 (0x2ae6 bytes) 000787 1783138707.023218 mapread 0x02e8ae thru 0x03325d (0x49b0 bytes) 000788 1783138707.024532 mapread 0x00d98e thru 0x01725d (0x98d0 bytes) 000789 1783138707.032229 mapread 0x01dcb0 thru 0x0238d3 (0x5c24 bytes) 000790 1783138707.033464 read 0x031c0d thru 0x03700e (0x5402 bytes) 000791 1783138707.037294 write 0x01b165 thru 0x0221b6 (0x7052 bytes) 000793 1783138707.040627 write 0x03303d thru 0x03ddaf (0xad73 bytes) 000798 1783138707.054745 write 0x013f15 thru 0x01d96f (0x9a5b bytes) 000799 1783138707.063261 read 0x00e470 thru 0x01b7b0 (0xd341 bytes) 000800 1783138707.072050 write 0x00055f thru 0x008aea (0x858c bytes) 000801 1783138707.078445 mapwrite 0x00a141 thru 0x0114c7 (0x7387 bytes) 000802 1783138707.227775 read 0x034c8c thru 0x03ffff (0xb374 bytes) 000803 1783138707.228515 mapread 0x0391f3 thru 0x03b7d7 (0x25e5 bytes) 000804 1783138707.229468 read 0x007935 thru 0x014c34 (0xd300 bytes) 000805 1783138707.230134 mapwrite 0x033a5a thru 0x03faf3 (0xc09a bytes) 000806 1783138707.341750 MIRROR_EXTEND to 2 mirrors 000811 1783138707.878966 read 0x002fdc thru 0x003250 (0x275 bytes) 000813 1783138707.886920 mapread 0x013b22 thru 0x023222 (0xf701 bytes) 000815 1783138707.916558 read 0x01edd5 thru 0x020e22 (0x204e bytes) 000816 1783138707.917586 read 0x01bd0c thru 0x024b91 (0x8e86 bytes) 000817 1783138707.918219 write 0x00c9ec thru 0x00d87c (0xe91 bytes) 000820 1783138708.004918 write 0x00a728 thru 0x0100bd (0x5996 bytes) 000823 1783138708.012039 write 0x0110e9 thru 0x01b5bf (0xa4d7 bytes) 000824 1783138708.014593 mapread 0x0194c9 thru 0x01c6ff (0x3237 bytes) 000825 1783138708.016027 trunc from 0x040000 to 0x023a14 (0x1c5eb bytes) 000826 1783138708.076900 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x23a15): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23a15): Mirror 2: 0xe1bee98e (stale) CRC-32 checksum value for chunk [0, 0x23a15): Mirror 1: 0x149569ab Mirror 3: 0x149569ab lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23a15: skipped 000828 1783138708.862656 mapwrite 0x01e519 thru 0x02bfb9 (0xdaa1 bytes) 000830 1783138709.022858 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bfba): Mirror 3: 0xbc63a868 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bfba): Mirror 3: 0xbc63a868 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bfba: skipped 000831 1783138709.349659 mapread 0x007f4e thru 0x00dd62 (0x5e15 bytes) 000832 1783138709.367885 mapread 0x02140f thru 0x02bfb9 (0xabab bytes) 000834 1783138709.382845 mapread 0x02879e thru 0x02bfb9 (0x381c bytes) 000838 1783138709.386141 mapwrite 0x02d14b thru 0x03ce15 (0xfccb bytes) 000839 1783138709.469634 write 0x03ba66 thru 0x03d3a7 (0x1942 bytes) 000840 1783138709.471737 write 0x03595e thru 0x03ca66 (0x7109 bytes) 000842 1783138709.473985 read 0x03795f thru 0x03d3a7 (0x5a49 bytes) 000843 1783138709.475192 write 0x01f3a2 thru 0x02078b (0x13ea bytes) 000844 1783138709.476702 mapwrite 0x0237b6 thru 0x02d156 (0x99a1 bytes) 000846 1783138709.518287 read 0x023871 thru 0x033106 (0xf896 bytes) 000847 1783138709.519174 mapwrite 0x0344b1 thru 0x03ffff (0xbb4f bytes) 000848 1783138709.613238 trunc from 0x040000 to 0x03127d (0xed82 bytes) 000849 1783138709.639970 MIRROR_RESYNC 2 mirrors 000850 1783138710.423848 trunc from 0x03127e to 0x00e897 (0x229e6 bytes) 000851 1783138710.497931 mapread 0x0073da thru 0x00b38a (0x3fb1 bytes) 000853 1783138710.520350 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xe898): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe898): Mirror 3: 0xf7a3335c (stale) CRC-32 checksum value for chunk [0, 0xe898): Mirror 1: 0xf7a3335c Mirror 4: 0xf7a3335c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe898: skipped 000855 1783138711.179648 trunc from 0x00e898 to 0x03dbcb (0x2f334 bytes) 000858 1783138711.320550 trunc from 0x03dbcc to 0x034832 (0x9399 bytes) 000859 1783138711.362903 mapread 0x0232d6 thru 0x03186e (0xe599 bytes) 000860 1783138711.398527 MIRROR_RESYNC 3 mirrors 000862 1783138712.128874 trunc from 0x034833 to 0x012058 (0x227da bytes) 000863 1783138712.257451 trunc from 0x012059 to 0x010d00 (0x1358 bytes) 000865 1783138712.284225 mapread 0x00554b thru 0x00d0a5 (0x7b5b bytes) 000866 1783138712.313494 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x10d01): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10d01): Mirror 3: 0xb8937fa3 (stale) Mirror 4: 0xb8937fa3 (stale) CRC-32 checksum value for chunk [0, 0x10d01): Mirror 1: 0xb8937fa3 Mirror 5: 0xb8937fa3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10d01: skipped 000867 1783138713.194438 mapwrite 0x024c1b thru 0x02828f (0x3675 bytes) 000869 1783138713.419593 read 0x00afc0 thru 0x00e02e (0x306f bytes) 000870 1783138713.462335 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28290): Mirror 4: 0x286a45b0 (stale) Mirror 5: 0x286a45b0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28290): Mirror 4: 0x286a45b0 (stale) Mirror 5: 0x286a45b0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28290: skipped 000871 1783138713.792326 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x28290): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28290): Mirror 4: 0x286a45b0 (stale) Mirror 5: 0x286a45b0 (stale) CRC-32 checksum value for chunk [0, 0x28290): Mirror 1: 0x2558ab02 Mirror 6: 0x2558ab02 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28290: skipped 000872 1783138714.944644 write 0x025430 thru 0x02fcc6 (0xa897 bytes) 000873 1783138715.087171 mapread 0x01b676 thru 0x01e832 (0x31bd bytes) 000875 1783138715.127294 trunc from 0x02fcc7 to 0x01c114 (0x13bb2 bytes) 000876 1783138715.154499 mapread 0x001fef thru 0x0035b5 (0x15c7 bytes) 000877 1783138715.164859 mapwrite 0x03ce6f thru 0x03ffff (0x3191 bytes) 000878 1783138715.277814 MIRROR_RESYNC 4 mirrors 000879 1783138716.390016 mapread 0x036db4 thru 0x03ffff (0x924c bytes) 000881 1783138716.411512 trunc from 0x040000 to 0x0284c3 (0x17b3c bytes) 000882 1783138716.535184 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x284c4): Mirror 4: 0x57b502de (stale) Mirror 5: 0x57b502de (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x284c4): Mirror 4: 0x57b502de (stale) Mirror 5: 0x57b502de (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x284c4: skipped 000883 1783138716.832527 MIRROR_RESYNC 3 mirrors 000884 1783138717.740417 mapread 0x01c59a thru 0x020846 (0x42ad bytes) 000885 1783138717.760824 MIRROR_RESYNC 3 mirrors truncating to largest ever: 0x3fbdb 000886 1783138718.076062 trunc from 0x0284c4 to 0x03fbda (0x17717 bytes) 000887 1783138718.151485 read 0x01d962 thru 0x025bc9 (0x8268 bytes) 000888 1783138718.211514 write 0x00f873 thru 0x01f125 (0xf8b3 bytes) 000889 1783138718.240472 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3fbdb): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fbdb): Mirror 4: 0x53dfb384 (stale) Mirror 5: 0x53dfb384 (stale) CRC-32 checksum value for chunk [0, 0x3fbdb): Mirror 1: 0xc59fcbbd Mirror 6: 0xc59fcbbd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fbdb: skipped 000891 1783138719.123324 write 0x026e4b thru 0x035668 (0xe81e bytes) 000892 1783138719.196976 read 0x03b282 thru 0x03fbda (0x4959 bytes) 000893 1783138719.205443 mapwrite 0x001088 thru 0x00805d (0x6fd6 bytes) 000896 1783138719.261648 mapread 0x038d86 thru 0x03c23b (0x34b6 bytes) 000897 1783138719.276460 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3fbdb): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fbdb): Mirror 4: 0x53dfb384 (stale) Mirror 5: 0x53dfb384 (stale) Mirror 6: 0xc59fcbbd (stale) CRC-32 checksum value for chunk [0, 0x3fbdb): Mirror 1: 0x63c7641c Mirror 7: 0x63c7641c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fbdb: skipped 000899 1783138720.147488 write 0x00ff07 thru 0x0123c4 (0x24be bytes) 000901 1783138720.210020 trunc from 0x03fbdb to 0x02739d (0x1883d bytes) 000903 1783138720.244774 write 0x00b1d2 thru 0x015b1f (0xa94e bytes) 000905 1783138720.277390 mapwrite 0x0075a6 thru 0x0137b9 (0xc214 bytes) 000906 1783138720.396907 write 0x021562 thru 0x02be26 (0xa8c5 bytes) 000907 1783138720.404492 mapread 0x01a025 thru 0x021a9f (0x7a7b bytes) 000909 1783138720.422024 trunc from 0x02be27 to 0x03a1dc (0xe3b6 bytes) 000910 1783138720.465129 trunc from 0x03a1dd to 0x004392 (0x35e4a bytes) 000913 1783138720.492214 mapread 0x001590 thru 0x004392 (0x2e03 bytes) 000917 1783138720.504368 mapwrite 0x020fca thru 0x022719 (0x1750 bytes) 000918 1783138720.704832 mapwrite 0x02ccc8 thru 0x02f6aa (0x29e3 bytes) 000920 1783138720.811613 read 0x02b132 thru 0x02eaa1 (0x3970 bytes) 000923 1783138720.820501 mapread 0x005e15 thru 0x0105a0 (0xa78c bytes) 000924 1783138720.860835 mapread 0x0047f9 thru 0x00ee92 (0xa69a bytes) 000926 1783138720.870340 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2f6ab): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f6ab): Mirror 4: 0x2b7a658c (stale) Mirror 5: 0x2b7a658c (stale) Mirror 6: 0x83329264 (stale) Mirror 7: 0x77f511e1 (stale) CRC-32 checksum value for chunk [0, 0x2f6ab): Mirror 1: 0xa2682744 Mirror 8: 0xa2682744 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f6ab: skipped 000927 1783138722.171779 mapread 0x02164b thru 0x02622e (0x4be4 bytes) 000928 1783138722.222697 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2f6ab): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f6ab): Mirror 4: 0x2b7a658c (stale) Mirror 5: 0x2b7a658c (stale) Mirror 6: 0x83329264 (stale) Mirror 7: 0x77f511e1 (stale) CRC-32 checksum value for chunk [0, 0x2f6ab): Mirror 1: 0xa2682744 Mirror 8: 0xa2682744 Mirror 9: 0xa2682744 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f6ab: skipped 000929 1783138723.049050 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2f6ab): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f6ab): Mirror 4: 0x2b7a658c (stale) Mirror 5: 0x2b7a658c (stale) Mirror 6: 0x83329264 (stale) Mirror 7: 0x77f511e1 (stale) CRC-32 checksum value for chunk [0, 0x2f6ab): Mirror 1: 0xa2682744 Mirror 8: 0xa2682744 Mirror 9: 0xa2682744 Mirror 10: 0xa2682744 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f6ab: skipped 000931 1783138723.767440 trunc from 0x02f6ab to 0x02e7a4 (0xf06 bytes) 000932 1783138723.885927 mapwrite 0x008747 thru 0x01814d (0xfa07 bytes) 000933 1783138724.008922 trunc from 0x02e7a5 to 0x01940b (0x15399 bytes) 000934 1783138724.026429 write 0x00f407 thru 0x01ee20 (0xfa1a bytes) 000937 1783138724.028869 read 0x00efe4 thru 0x011619 (0x2636 bytes) 000938 1783138724.029719 read 0x00743a thru 0x00df6f (0x6b36 bytes) 000939 1783138724.030621 trunc from 0x01ee21 to 0x03f0ca (0x202aa bytes) 000940 1783138724.041489 read 0x021229 thru 0x022c70 (0x1a48 bytes) 000941 1783138724.060355 mapread 0x0056e4 thru 0x00e992 (0x92af bytes) 000943 1783138724.064729 read 0x01d883 thru 0x023533 (0x5cb1 bytes) 000945 1783138724.067263 read 0x00432d thru 0x011353 (0xd027 bytes) 000946 1783138724.067804 mapread 0x021669 thru 0x02a341 (0x8cd9 bytes) 000947 1783138724.070289 write 0x0170da thru 0x01e649 (0x7570 bytes) 000948 1783138724.072778 write 0x02e0a9 thru 0x03cf21 (0xee79 bytes) 000949 1783138724.075343 trunc from 0x03f0cb to 0x011bdb (0x2d4ef bytes) 000951 1783138724.089735 trunc from 0x011bdc to 0x029ce9 (0x1810e bytes) 000953 1783138724.114467 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29cea): Mirror 4: 0x4fa87055 (stale) Mirror 5: 0x4fa87055 (stale) Mirror 6: 0x74b09552 (stale) Mirror 7: 0x8f92beff (stale) Mirror 9: 0x6ffe9d37 (stale) Mirror 10: 0x6ffe9d37 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29cea): Mirror 4: 0x4fa87055 (stale) Mirror 5: 0x4fa87055 (stale) Mirror 6: 0x74b09552 (stale) Mirror 7: 0x8f92beff (stale) Mirror 9: 0x6ffe9d37 (stale) Mirror 10: 0x6ffe9d37 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29cea: skipped 000955 1783138724.749159 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x29cea): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29cea): Mirror 4: 0x4fa87055 (stale) Mirror 5: 0x4fa87055 (stale) Mirror 6: 0x74b09552 (stale) Mirror 7: 0x8f92beff (stale) Mirror 9: 0x6ffe9d37 (stale) Mirror 10: 0x6ffe9d37 (stale) CRC-32 checksum value for chunk [0, 0x29cea): Mirror 1: 0x14c41d3b Mirror 11: 0x14c41d3b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29cea: skipped 000958 1783138725.868867 write 0x008eb8 thru 0x018e5e (0xffa7 bytes) 000959 1783138725.943221 write 0x02b479 thru 0x02c06f (0xbf7 bytes) 000961 1783138725.946443 trunc from 0x02c070 to 0x00f81f (0x1c850 bytes) 000962 1783138725.966522 mapwrite 0x00c313 thru 0x00f911 (0x35ff bytes) 000964 1783138726.028868 trunc from 0x00f912 to 0x00052a (0xf3e7 bytes) 000965 1783138726.045495 mapwrite 0x00f214 thru 0x01e0fc (0xeee9 bytes) 000967 1783138726.143112 read 0x012aa2 thru 0x0180a3 (0x5602 bytes) 000968 1783138726.143797 mapwrite 0x016ec4 thru 0x01a291 (0x33ce bytes) 000969 1783138726.193405 read 0x000fea thru 0x0062c9 (0x52e0 bytes) 000970 1783138726.203482 read 0x0127a6 thru 0x01e0fc (0xb957 bytes) 000971 1783138726.208105 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1e0fd): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e0fd): Mirror 4: 0xb209e23c (stale) Mirror 5: 0xb209e23c (stale) Mirror 6: 0xdf8cb3ed (stale) Mirror 7: 0x3b6902c2 (stale) Mirror 9: 0xa1cc13b4 (stale) Mirror 10: 0xa1cc13b4 (stale) Mirror 11: 0xb1e2ba07 (stale) CRC-32 checksum value for chunk [0, 0x1e0fd): Mirror 1: 0x87e50eed Mirror 12: 0x87e50eed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e0fd: skipped 000972 1783138727.263360 mapread 0x0102c7 thru 0x01e0fc (0xde36 bytes) 000974 1783138727.361762 mapwrite 0x0102aa thru 0x0183f6 (0x814d bytes) 000975 1783138727.649627 MIRROR_RESYNC 9 mirrors 000976 1783138729.392663 mapread 0x011984 thru 0x018ec8 (0x7545 bytes) 000977 1783138729.424117 write 0x00cbc0 thru 0x0121cd (0x560e bytes) 000979 1783138729.524604 mapwrite 0x02ffa1 thru 0x03f00f (0xf06f bytes) 000980 1783138729.637178 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f010): Mirror 4: 0xce7aabbb (stale) Mirror 6: 0xce7aabbb (stale) Mirror 7: 0xce7aabbb (stale) Mirror 9: 0xce7aabbb (stale) Mirror 10: 0xce7aabbb (stale) Mirror 11: 0xce7aabbb (stale) Mirror 12: 0xce7aabbb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f010): Mirror 4: 0xce7aabbb (stale) Mirror 6: 0xce7aabbb (stale) Mirror 7: 0xce7aabbb (stale) Mirror 9: 0xce7aabbb (stale) Mirror 10: 0xce7aabbb (stale) Mirror 11: 0xce7aabbb (stale) Mirror 12: 0xce7aabbb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f010: skipped 000981 1783138730.250161 mapread 0x03254f thru 0x03b47f (0x8f31 bytes) 000982 1783138730.282220 write 0x02a9d3 thru 0x02b42a (0xa58 bytes) 000984 1783138730.304062 mapwrite 0x021b5a thru 0x023554 (0x19fb bytes) 000986 1783138730.405521 trunc from 0x03f010 to 0x01adea (0x24225 bytes) 000987 1783138730.453491 mapread 0x006dfc thru 0x00a04b (0x3250 bytes) 000988 1783138730.464970 trunc from 0x01adeb to 0x0127a4 (0x8646 bytes) 000989 1783138730.494837 MIRROR_RESYNC 8 mirrors 000990 1783138731.769678 write 0x000fc8 thru 0x005570 (0x45a9 bytes) 000991 1783138731.849415 mapread 0x007806 thru 0x0127a4 (0xaf9f bytes) 000994 1783138731.874680 read 0x00bfff thru 0x0127a4 (0x67a6 bytes) 000995 1783138731.875798 mapread 0x00c694 thru 0x0127a4 (0x6111 bytes) 000996 1783138731.876974 read 0x00c727 thru 0x00f1b2 (0x2a8c bytes) 000997 1783138731.878036 mapread 0x007f14 thru 0x0127a4 (0xa891 bytes) 000998 1783138731.880829 read 0x0112b4 thru 0x0127a4 (0x14f1 bytes) 001000 1783138731.882751 read 0x00eaf4 thru 0x0100f3 (0x1600 bytes) All operations completed A-OK! PASS 70a (81s) == sanity-flr test 71: check mirror extend parent fid ==== 00:18:57 (1783138737) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 00:19:01 (1783138741) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2da:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.28768 s, 7.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2da:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2db:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159737 s, 6.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2da:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2db:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bf:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2da:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2db:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bf:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dc:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 00:19:13 (1783138753) Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 2388800 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7280224 .. Extending file size to 3022656 .. Extending file size to 8792352 .. Extending file size to 9211232 .. Extending file size to 7396320 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7914080 .. Extending file size to 8457216 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8714848 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9203936 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 56835 56837 56838 56840 56841 done 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) 192.168.202.160@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 00:20:35 (1783138835) Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3396896 .. Extending file size to 2589088 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6834016 .. Extending file size to 8014272 .. Extending file size to 8500320 .. Extending file size to 7728992 .. Extending file size to 9008224 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9148928 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8646720 .. Extending file size to 8817920 .. Waiting 58269 58271 58273 58275 58276 done 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) 192.168.202.160@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 00:21:55 (1783138915) Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0709057 s, 29.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.458126 s, 4.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xc69700dc (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 00:22:15 (1783138935) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0939497 s, 22.3 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 00:22:22 (1783138942) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1dc:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 00:22:28 (1783138948) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.193786 s, 10.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 00:22:36 (1783138956) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.151762 s, 13.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 00:22:44 (1783138964) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.230382 s, 9.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } PASS 204c (31s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 00:23:15 (1783138995) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg260-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140109 s, 7.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 00:24:11 (1783139051) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg260-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.318194 s, 6.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (59s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 00:25:10 (1783139110) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg260-server mds-ost sync done. Creating new pool test_204f oleg260-server: Pool lustre.test_204f created Adding targets to pool oleg260-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg260-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.182783 s, 11.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg260-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg260-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg260-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 00:26:32 (1783139192) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 00:26:38 (1783139198) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 00:26:46 (1783139206) oleg260-server: Pool lustre.test_206 created Adding targets to pool oleg260-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg260-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg260-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg260-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg260-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 00:27:10 (1783139230) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612236 s, 17.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523749 s, 20.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x3e102c44 Mirror 3: 0x3e102c44 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 00:27:21 (1783139241) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 00:27:24 (1783139244) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 00:27:28 (1783139248) Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.101158 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1370000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1ac0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1ac0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1ac0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xce0000 Read this (4 bytes): BBBB 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) PASS 209a (41s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 00:28:09 (1783139289) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0159169 s, 515 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 00:28:16 (1783139296) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093501 s, 11.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x201 lmm_fid: [0x200000404:0x201:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x201 lmm_fid: [0x200000404:0x201:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e9:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834433 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-04 00:28:18.000000000 -0400 Modify: 2026-07-04 00:28:18.000000000 -0400 Change: 2026-07-04 00:28:19.000000000 -0400 Birth: 2026-07-04 00:28:18.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 00:28:25 (1783139305) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 00:28:28 (1783139308) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.433929 s, 94.4 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0851796 s, 48.1 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 4 00:28 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x203:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 4 00:28 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 00:28:36 (1783139316) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 00:28:44 (1783139324) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1793 sec =========== 00:28:53 (1783139333) === sanity-flr: start cleanup 00:28:54 (1783139334) === === sanity-flr: finish cleanup 00:29:01 (1783139341) ===