-----============= acceptance-small: sanity-flr ============----- Sat Aug 22 06:18:15 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg620-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 06:18:35 (1787393915) === oleg620-client.virtnet: executing check_config_client /mnt/lustre oleg620-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg620-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9725daf5e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9725daf5e800.idle_timeout=debug disable quota as required oleg620-server: oleg620-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 06:19:14 (1787393954) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8222] == sanity-flr test 0a: lfs mirror create with -N option == 06:19:16 (1787393956) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 06:19:33 (1787393973) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 06:19:36 (1787393976) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 06:19:39 (1787393979) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 06:20:03 (1787394003) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 06:20:07 (1787394007) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 06:20:11 (1787394011) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9725daf5e800.stats=clear osc.lustre-OST0001-osc-ffff9725daf5e800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 06:20:19 (1787394019) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 06:20:30 (1787394030) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 06:20:38 (1787394038) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 06:20:41 (1787394041) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 06:20:50 (1787394050) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 06:21:00 (1787394060) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 06:21:07 (1787394067) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 06:21:14 (1787394074) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 06:21:20 (1787394080) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 06:21:29 (1787394089) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 06:21:35 (1787394095) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.500567 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104319 s, 10.1 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 06:21:44 (1787394104) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.211168 s, 39.7 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 06:21:52 (1787394112) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 06:22:04 (1787394124) Stopping /mnt/lustre-ost1 (opts:) on oleg620-server oleg620-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9725daf5e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9725daf5e800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg620-server: oleg620-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg620-client: oleg620-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg620-server: oleg620-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg620-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg620-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg620-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg620-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg620-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg620-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg620-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg620-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg620-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg620-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg620-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg620-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg620-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg620-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg620-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg620-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg620-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg620-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg620-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg620-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg620-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg620-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg620-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg620-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg620-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg620-server: Waiting 1410 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg620-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg620-server oleg620-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9725daf5e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9725daf5e800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg620-server: oleg620-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg620-client: oleg620-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg620-server: oleg620-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg620-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (131s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 06:24:15 (1787394255) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.157304 s, 20.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg620-server oleg620-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9725daf5e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9725daf5e800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg620-server: oleg620-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg620-client: oleg620-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg620-server: oleg620-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg620-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (41s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 06:24:56 (1787394296) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg620-server oleg620-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9725daf5e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9725daf5e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg620-server: oleg620-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg620-client: oleg620-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg620-server: oleg620-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg620-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg620-server oleg620-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9725daf5e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9725daf5e800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg620-server: oleg620-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg620-client: oleg620-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg620-server: oleg620-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg620-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (82s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 06:26:18 (1787394378) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.88834 s, 13.5 MB/s -rw-r--r-- 1 root root 52428800 Aug 22 06:26 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.925s user 0m0.000s sys 0m0.190s Stopping /mnt/lustre-ost1 (opts:) on oleg620-server oleg620-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9725daf5e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9725daf5e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.965s user 0m0.001s sys 0m0.173s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg620-server: oleg620-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg620-client: oleg620-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg620-server: oleg620-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg620-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg620-server oleg620-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9725daf5e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9725daf5e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.320s user 0m0.002s sys 0m0.181s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg620-server: oleg620-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg620-client: oleg620-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg620-server: oleg620-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg620-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg620-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9725daf5e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9725daf5e800.ost_server_uuid in FULL state after 0 sec PASS 33b (89s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 06:27:48 (1787394468) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 06:27:51 (1787394471) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 06:27:55 (1787394475) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 06:27:59 (1787394479) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 06:28:06 (1787394486) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.453446 s, 27.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 06:28:19 (1787394499) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247394 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032106 s, 32.7 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 06:28:27 (1787394507) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 06:28:36 (1787394516) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000830413 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000894476 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00249064 s, 411 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00172019 s, 595 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 06:28:46 (1787394526) create test files with size 15735210 .. d6eb8c3a9ea073ba5a88d663cb4c8a73 - 1055e5bebbd6b973098a4ef1f4dfdb86 - 7d56b08a8f8284dcdc0d913faef5bc2b - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg620-client.virtnet /mnt/lustre (opts:) Starting client: oleg620-client.virtnet: -o user_xattr,flock 192.168.206.120@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 06:29:10 (1787394550) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (18s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 06:29:29 (1787394569) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 06:29:36 (1787394576) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118845 s, 8.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xac:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xad:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419873 s, 25.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xae:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 06:29:46 (1787394586) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0861345 s, 24.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.451449 s, 9.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.167013 s, 12.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 06:30:03 (1787394603) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 06:30:06 (1787394606) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 06:30:09 (1787394609) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294995 s, 35.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 06:30:16 (1787394616) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.293353 s, 10.7 MB/s PASS 44a (9s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 06:30:25 (1787394625) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd8:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 06:30:35 (1787394635) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.350302 s, 29.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg620-client.virtnet /mnt/lustre (opts:) Starting client: oleg620-client.virtnet: -o user_xattr,flock 192.168.206.120@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 06:30:46 (1787394646) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9725e1007000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.293372 s, 35.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg620-client.virtnet /mnt/lustre (opts:) Starting client: oleg620-client.virtnet: -o user_xattr,flock 192.168.206.120@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9725c3f98000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 06:31:01 (1787394661) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (36s) == sanity-flr test 46: Verify setstripe --copy option ==== 06:31:37 (1787394697) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (35s) == sanity-flr test 47: Verify mirror obj alloc =========== 06:32:12 (1787394732) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 06:32:16 (1787394736) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.364633 s, 8.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.198993 s, 15.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x97e47ab9 (nosync,stale) Mirror 2: 0x97e47ab9 (nosync,stale) ** verify mirror 2 doesn't change original checksum: e89f5465b83dbbe6da8f2c7c3f1886a2 - mirror 1 checksum: 5b5a603fbad6936ea6651ed46b905f98 - mirror 2 checksum: e89f5465b83dbbe6da8f2c7c3f1886a2 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-22T06:32:20-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-22T06:32:20-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: e89f5465b83dbbe6da8f2c7c3f1886a2 - mirror 1 checksum: 5b5a603fbad6936ea6651ed46b905f98 - mirror 2 checksum: 5b5a603fbad6936ea6651ed46b905f98 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-22T06:32:20-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-22T06:32:20-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 06:32:26 (1787394746) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383619 s, 27.3 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 06:32:33 (1787394753) Failing mds1 on oleg620-server Stopping /mnt/lustre-mds1 (opts:) on oleg620-server 06:32:44 (1787394764) shut down facet: mds1 facet_host: oleg620-server facet_failover_host: oleg620-server Failover mds1 to oleg620-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg620-server: oleg620-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg620-client: oleg620-server: ssh exited with exit code 1 Started lustre-MDT0000 06:33:17 (1787394797) targets are mounted 06:33:17 (1787394797) facet_failover done oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327649 s, 32.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } PASS 50A (60s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 06:33:33 (1787394813) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.048753 s, 420 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0140122 s, 1.5 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (12s) == sanity-flr test 50b: mirror rsync handles sparseness == 06:33:45 (1787394825) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.104598 s, 196 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0122654 s, 1.7 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (11s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 06:33:56 (1787394836) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0276324 s, 593 kB/s PASS 50c (11s) == sanity-flr test 50d: mirror rsync keep holes ========== 06:34:07 (1787394847) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0558895 s, 366 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 06:34:16 (1787394856) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00936903 s, 437 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0239757 s, 171 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 06:34:24 (1787394864) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) PASS 60b (26s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 06:34:50 (1787394890) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 06:35:21 (1787394921) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 06:35:46 (1787394946) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 06:36:10 (1787394970) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 06:36:18 (1787394978) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 57367 mirror split pid 57368 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 57367 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 57368 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 06:37:26 (1787395046) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 5048 skipping zero length punch hole skipping zero length punch hole 000003 1787395048.602025 write 0x034157 thru 0x036145 (0x1fef bytes) 000005 1787395048.606378 mapread 0x00814c thru 0x00a582 (0x2437 bytes) 000006 1787395048.633038 read 0x00b93d thru 0x00f3ad (0x3a71 bytes) fallocating to largest ever: 0x2da0a 000008 1787395048.635437 fallocate 0x02d256 thru 0x02da09 (0x7b4 bytes) 000009 1787395048.642377 punch from 0x02be33 to 0x034850 (0x8a1e bytes) 000010 1787395048.654347 write 0x033fb6 thru 0x0369c6 (0x2a11 bytes) truncating to largest ever: 0x3a2b4 000011 1787395048.663855 trunc from 0x0369c7 to 0x03a2b3 (0x38ed bytes) fallocating to largest ever: 0x40000 000012 1787395048.687774 fallocate 0x03df7a thru 0x03ffff (0x2086 bytes) 000013 1787395048.694593 trunc from 0x03a2b4 to 0x034873 (0x5a40 bytes) 000014 1787395048.724731 trunc from 0x034594 to 0x034873 (0x2e0 bytes) 000015 1787395048.747399 write 0x02193d thru 0x030629 (0xeced bytes) 000020 1787395048.805108 write 0x0028af thru 0x00645b (0x3bad bytes) 000021 1787395048.844425 MIRROR_EXTEND to 2 mirrors 000022 1787395049.952457 trunc from 0x01b2e3 to 0x024ceb (0x9a09 bytes) 000023 1787395050.026250 trunc from 0x034874 to 0x033fc6 (0x8ad bytes) 000024 1787395050.060598 read 0x01cf7a thru 0x02a71e (0xd7a5 bytes) 000025 1787395050.081287 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x33fc7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33fc7): Mirror 2: 0xc4e434df (stale) CRC-32 checksum value for chunk [0, 0x33fc7): Mirror 1: 0x41799bb4 Mirror 3: 0x41799bb4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33fc7: skipped 000026 1787395051.079365 trunc from 0x0319ed to 0x033fc6 (0x25da bytes) 000027 1787395051.152570 read 0x009796 thru 0x00a84b (0x10b6 bytes) 000028 1787395051.164183 read 0x0194e6 thru 0x0265d9 (0xd0f4 bytes) 000030 1787395051.185451 trunc from 0x02c2b7 to 0x033d8d (0x7ad7 bytes) 000031 1787395051.202534 MIRROR_RESYNC 3 mirrors 000032 1787395052.586851 read 0x01c5a1 thru 0x025216 (0x8c76 bytes) 000033 1787395052.608587 punch from 0x018e23 to 0x01f6bd (0x689b bytes) 000034 1787395052.661988 read 0x02c501 thru 0x033fc6 (0x7ac6 bytes) 000035 1787395052.686111 trunc from 0x033fc7 to 0x01196e (0x22658 bytes) 000036 1787395052.727813 mapwrite 0x0262d5 thru 0x030c8a (0xa9b6 bytes) 000037 1787395052.955447 mapread 0x018f89 thru 0x019c56 (0xcce bytes) 000038 1787395052.980479 trunc from 0x030c8b to 0x03c952 (0xbcc8 bytes) 000039 1787395053.001721 read 0x03302c thru 0x03c952 (0x9927 bytes) 000041 1787395053.015542 trunc from 0x03c953 to 0x02c925 (0x1002d bytes) 000042 1787395053.042256 mapread 0x0082a5 thru 0x015aad (0xd809 bytes) 000043 1787395053.046808 punch from 0x028def to 0x02c925 (0x3b37 bytes) 000044 1787395053.056961 mapread 0x020585 thru 0x0257e1 (0x525d bytes) 000046 1787395053.062204 trunc from 0x02c926 to 0x035b26 (0x9201 bytes) 000047 1787395053.105428 write 0x033267 thru 0x03fb8f (0xc929 bytes) 000048 1787395053.114722 write 0x001496 thru 0x007a3c (0x65a7 bytes) 000049 1787395053.117565 write 0x039bd4 thru 0x03ea62 (0x4e8f bytes) 000050 1787395053.122447 trunc from 0x03fb90 to 0x0266fc (0x19493 bytes) 000051 1787395053.143737 punch from 0x013e9a to 0x0147c2 (0x929 bytes) 000052 1787395053.163734 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x266fd): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x266fd): Mirror 2: 0xb00b4bdc (stale) Mirror 3: 0xb00b4bdc (stale) CRC-32 checksum value for chunk [0, 0x266fd): Mirror 1: 0x174ef868 Mirror 4: 0x174ef868 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x266fd: skipped 000053 1787395054.200321 trunc from 0x0196af to 0x01c6d7 (0x3029 bytes) 000054 1787395054.293536 trunc from 0x0266fd to 0x03f5b6 (0x18eba bytes) 000055 1787395054.389500 fallocate 0x02540a thru 0x026ea1 (0x1a98 bytes) 000056 1787395054.407625 read 0x020fd8 thru 0x022c78 (0x1ca1 bytes) 000057 1787395054.416917 mapwrite 0x0336a3 thru 0x039e05 (0x6763 bytes) 000058 1787395054.477297 read 0x0232b6 thru 0x0303b1 (0xd0fc bytes) 000060 1787395054.541451 punch from 0x021012 to 0x030150 (0xf13f bytes) 000061 1787395054.547882 read 0x013d63 thru 0x01f2ff (0xb59d bytes) 000062 1787395054.549401 read 0x0251cf thru 0x026c95 (0x1ac7 bytes) 000063 1787395054.565150 punch from 0x03901b to 0x03f5b6 (0x659c bytes) 000064 1787395054.575489 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f5b7): Mirror 2: 0x252dfd53 (stale) Mirror 3: 0x252dfd53 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f5b7): Mirror 2: 0x252dfd53 (stale) Mirror 3: 0x252dfd53 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f5b7: skipped 000065 1787395055.032346 MIRROR_RESYNC 3 mirrors 000066 1787395056.331587 fallocate 0x011cb8 thru 0x01e209 (0xc552 bytes) 000067 1787395056.406349 read 0x0192b8 thru 0x01e31c (0x5065 bytes) 000068 1787395056.422652 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f5b7): Mirror 2: 0x49727beb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f5b7): Mirror 2: 0x49727beb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f5b7: skipped 000069 1787395056.773014 fallocate 0x024b7d thru 0x027fee (0x3472 bytes) 000070 1787395056.784529 mapread 0x011531 thru 0x01fc71 (0xe741 bytes) 000071 1787395056.840564 mapread 0x01bee5 thru 0x024952 (0x8a6e bytes) 000072 1787395056.854633 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a2:0x0]: Invalid argument (22) 000075 1787395057.008604 mapread 0x00b5e2 thru 0x010c15 (0x5634 bytes) 000076 1787395057.075975 trunc from 0x03f5b7 to 0x01505a (0x2a55c bytes) 000077 1787395057.105979 mapwrite 0x018a1c thru 0x024549 (0xbb2e bytes) 000078 1787395057.241736 mapread 0x020200 thru 0x024549 (0x434a bytes) 000079 1787395057.249700 write 0x010ead thru 0x01fa7e (0xebd2 bytes) 000080 1787395057.257343 write 0x028897 thru 0x031241 (0x89ab bytes) 000081 1787395057.266373 mapwrite 0x03dbca thru 0x03e1c9 (0x600 bytes) 000083 1787395057.352281 read 0x035684 thru 0x03e1c9 (0x8b46 bytes) 000084 1787395057.357703 trunc from 0x01feaf to 0x02e93c (0xea8e bytes) 000085 1787395057.363688 write 0x00979c thru 0x00ad77 (0x15dc bytes) 000086 1787395057.364891 punch from 0x02b701 to 0x035971 (0xa271 bytes) 000088 1787395057.373515 trunc from 0x01f866 to 0x02adb1 (0xb54c bytes) 000090 1787395057.391982 MIRROR_EXTEND to 2 mirrors 000091 1787395058.638321 read 0x0298b0 thru 0x02a322 (0xa73 bytes) 000093 1787395058.653268 fallocate 0x00173a thru 0x0098ec (0x81b3 bytes) 000094 1787395058.748695 fallocate 0x0262bc thru 0x029889 (0x35ce bytes) 000095 1787395058.767808 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a2:0x0]: Invalid argument (22) 000096 1787395059.093771 fallocate 0x03f002 thru 0x03ffff (0xffe bytes) 000097 1787395059.109481 fallocate 0x018e9c thru 0x028132 (0xf297 bytes) 000099 1787395059.118169 read 0x002842 thru 0x00a009 (0x77c8 bytes) 000100 1787395059.129775 write 0x005f99 thru 0x00f715 (0x977d bytes) 000101 1787395059.144013 read 0x014775 thru 0x0179fb (0x3287 bytes) 000102 1787395059.179523 trunc from 0x02c59a to 0x033bd4 (0x763b bytes) 000105 1787395059.198364 mapread 0x01bbb6 thru 0x026d3e (0xb189 bytes) 000107 1787395059.205655 punch from 0x01ab22 to 0x01ca09 (0x1ee8 bytes) 000108 1787395059.213869 mapread 0x0334c6 thru 0x03e1c9 (0xad04 bytes) 000109 1787395059.221829 mapwrite 0x0107e9 thru 0x01a92b (0xa143 bytes) 000110 1787395059.254783 write 0x02a4c8 thru 0x031d37 (0x7870 bytes) 000111 1787395059.261613 trunc from 0x03e1ca to 0x03342d (0xad9c bytes) 000112 1787395059.277481 punch from 0x025d05 to 0x02607d (0x379 bytes) 000113 1787395059.283922 mapwrite 0x02502a thru 0x02e5a1 (0x9578 bytes) 000115 1787395059.323393 punch from 0x02c068 to 0x03342d (0x73c6 bytes) 000116 1787395059.332610 trunc from 0x012e97 to 0x016e34 (0x3f9e bytes) 000117 1787395059.339851 punch from 0x0074db to 0x015ef0 (0xea16 bytes) 000119 1787395059.355137 fallocate 0x01641c thru 0x0171bb (0xda0 bytes) 000120 1787395059.365962 trunc from 0x0297b6 to 0x030175 (0x69c0 bytes) 000121 1787395059.374787 mapwrite 0x03269c thru 0x03c66f (0x9fd4 bytes) 000122 1787395059.557082 mapread 0x01ca10 thru 0x028d07 (0xc2f8 bytes) 000123 1787395059.570522 mapwrite 0x03e912 thru 0x03ffff (0x16ee bytes) 000124 1787395059.626338 trunc from 0x040000 to 0x03a15d (0x5ea2 bytes) 000125 1787395059.644594 trunc from 0x03a15e to 0x01f924 (0x1a839 bytes) 000126 1787395059.686710 trunc from 0x009a01 to 0x00cbf3 (0x31f3 bytes) 000127 1787395059.694330 write 0x02f712 thru 0x037d95 (0x8684 bytes) 000128 1787395059.702884 fallocate 0x0023fd thru 0x007343 (0x4f47 bytes) 000129 1787395059.710672 fallocate 0x00a0c6 thru 0x00afb7 (0xef2 bytes) 000130 1787395059.716705 punch from 0x01be1d to 0x028df3 (0xcfd7 bytes) 000131 1787395059.729516 mapread 0x0142dc thru 0x020e02 (0xcb27 bytes) 000132 1787395059.831005 trunc from 0x037d96 to 0x008bcb (0x2f1ca bytes) 000133 1787395059.855419 write 0x02c1f9 thru 0x034154 (0x7f5c bytes) 000134 1787395059.861823 write 0x02b663 thru 0x03593b (0xa2d9 bytes) 000135 1787395059.876582 mapwrite 0x024d26 thru 0x032b80 (0xde5b bytes) 000136 1787395059.954269 punch from 0x02ca28 to 0x03593b (0x8f14 bytes) 000137 1787395059.961855 mapwrite 0x01ee69 thru 0x020496 (0x162e bytes) 000138 1787395059.988723 fallocate 0x03808b thru 0x03cb58 (0x4ace bytes) 000139 1787395059.996970 mapwrite 0x0022ae thru 0x00d2d3 (0xb026 bytes) 000140 1787395060.062518 punch from 0x002929 to 0x006726 (0x3dfe bytes) 000141 1787395060.071038 mapwrite 0x01ecf9 thru 0x0247e1 (0x5ae9 bytes) 000142 1787395060.134842 mapread 0x005092 thru 0x013ad9 (0xea48 bytes) 000143 1787395060.185856 trunc from 0x02db80 to 0x03593b (0x7dbc bytes) 000145 1787395060.195217 trunc from 0x03593c to 0x02a294 (0xb6a7 bytes) 000146 1787395060.240344 punch from 0x024b92 to 0x028ab5 (0x3f24 bytes) 000147 1787395060.255382 write 0x008bb6 thru 0x00dede (0x5329 bytes) 000148 1787395060.265764 write 0x001ac0 thru 0x00736d (0x58ae bytes) 000149 1787395060.271745 mapread 0x02740e thru 0x0296fc (0x22ef bytes) 000150 1787395060.309499 trunc from 0x02a295 to 0x011a55 (0x1883f bytes) 000152 1787395060.339774 MIRROR_EXTEND to 2 mirrors 000154 1787395061.130781 mapread 0x003fa3 thru 0x006711 (0x276f bytes) 000155 1787395061.171516 mapread 0x006719 thru 0x00ccf8 (0x65e0 bytes) 000156 1787395061.172621 trunc from 0x0031e4 to 0x0084cd (0x52ea bytes) 000157 1787395061.248779 trunc from 0x00f674 to 0x010019 (0x9a6 bytes) 000158 1787395061.256809 write 0x0049a8 thru 0x0120ab (0xd704 bytes) 000159 1787395061.282483 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a2:0x0]: Invalid argument (22) 000161 1787395061.473664 write 0x002bfe thru 0x00c897 (0x9c9a bytes) 000165 1787395061.494297 fallocate 0x03676b thru 0x03ffff (0x9895 bytes) 000166 1787395061.500496 mapread 0x00774a thru 0x0120ab (0xa962 bytes) 000167 1787395061.521699 mapwrite 0x002b14 thru 0x0068f7 (0x3de4 bytes) 000168 1787395061.542767 read 0x00b5fa thru 0x01096e (0x5375 bytes) 000169 1787395061.544244 punch from 0x009cc7 to 0x0120ab (0x83e5 bytes) 000170 1787395061.550956 punch from 0x007515 to 0x008b51 (0x163d bytes) 000171 1787395061.557360 trunc from 0x006538 to 0x009b69 (0x3632 bytes) 000174 1787395061.568826 trunc from 0x0120ac to 0x00406f (0xe03c bytes) 000175 1787395061.587492 trunc from 0x004070 to 0x00e8a7 (0xa838 bytes) 000176 1787395061.605271 read 0x00c750 thru 0x00e8a7 (0x2158 bytes) 000177 1787395061.619712 read 0x00310e thru 0x004523 (0x1416 bytes) 000178 1787395061.620502 read 0x003942 thru 0x00c46e (0x8b2d bytes) 000180 1787395061.624710 trunc from 0x009299 to 0x00e8a7 (0x560f bytes) 000181 1787395061.631665 mapread 0x000d84 thru 0x00a11f (0x939c bytes) 000182 1787395061.643721 mapwrite 0x03ddf9 thru 0x03ffff (0x2207 bytes) 000183 1787395061.694292 mapread 0x02d0ec thru 0x0311cc (0x40e1 bytes) 000185 1787395061.700201 mapread 0x01959d thru 0x02070d (0x7171 bytes) 000187 1787395061.709526 punch from 0x03cdf2 to 0x03f0ba (0x22c9 bytes) 000189 1787395061.721825 trunc from 0x01fb33 to 0x02a32e (0xa7fc bytes) 000190 1787395061.730587 mapread 0x025980 thru 0x02fcab (0xa32c bytes) 000191 1787395061.794760 mapwrite 0x027d3b thru 0x034e20 (0xd0e6 bytes) 000192 1787395061.849423 MIRROR_EXTEND to 2 mirrors 000193 1787395062.881666 write 0x034689 thru 0x03ffa0 (0xb918 bytes) 000194 1787395062.929272 trunc from 0x035a5f to 0x03ffff (0xa5a1 bytes) 000195 1787395062.948319 trunc from 0x032ca3 to 0x03647d (0x37db bytes) 000196 1787395062.960568 fallocate 0x01485e thru 0x016a2c (0x21cf bytes) 000197 1787395062.972367 fallocate 0x0293ab thru 0x03793c (0xe592 bytes) 000198 1787395062.998495 MIRROR_RESYNC 2 mirrors 000200 1787395063.899505 trunc from 0x0261d1 to 0x028f78 (0x2da8 bytes) 000201 1787395063.942390 mapwrite 0x0155ad thru 0x01f44a (0x9e9e bytes) 000202 1787395063.988482 mapread 0x028bce thru 0x034ac4 (0xbef7 bytes) 000203 1787395064.016762 trunc from 0x017e2d to 0x018ebb (0x108f bytes) 000204 1787395064.022326 mapread 0x012b84 thru 0x01c676 (0x9af3 bytes) 000205 1787395064.029658 mapread 0x00eef0 thru 0x01ec69 (0xfd7a bytes) 000206 1787395064.043669 trunc from 0x00c1c6 to 0x015ac1 (0x98fc bytes) 000207 1787395064.050102 trunc from 0x040000 to 0x005366 (0x3ac99 bytes) 000208 1787395064.073717 read 0x001c85 thru 0x005366 (0x36e2 bytes) 000209 1787395064.083482 read 0x003d98 thru 0x005366 (0x15cf bytes) 000210 1787395064.084354 read 0x001c62 thru 0x002fef (0x138e bytes) 000211 1787395064.085816 trunc from 0x002f18 to 0x005366 (0x244f bytes) 000212 1787395064.092205 mapread 0x002106 thru 0x003d9a (0x1c95 bytes) 000213 1787395064.104839 fallocate 0x029b4e thru 0x039600 (0xfab3 bytes) 000214 1787395064.112112 mapwrite 0x0268f9 thru 0x03292a (0xc032 bytes) 000215 1787395064.155847 punch from 0x037a0f to 0x039600 (0x1bf2 bytes) 000216 1787395064.163556 write 0x0154bc thru 0x018350 (0x2e95 bytes) 000218 1787395064.167505 mapread 0x001239 thru 0x005f94 (0x4d5c bytes) 000219 1787395064.171450 trunc from 0x01b669 to 0x027958 (0xc2f0 bytes) 000220 1787395064.185563 punch from 0x031f15 to 0x039600 (0x76ec bytes) 000221 1787395064.199538 MIRROR_RESYNC 2 mirrors 000222 1787395064.974853 mapread 0x014e8f thru 0x01c72e (0x78a0 bytes) 000223 1787395065.008806 write 0x0071ff thru 0x00743f (0x241 bytes) 000224 1787395065.064805 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x39601): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39601): Mirror 2: 0x3bb15bb0 (stale) CRC-32 checksum value for chunk [0, 0x39601): Mirror 1: 0xdf81d017 Mirror 3: 0xdf81d017 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39601: skipped 000225 1787395066.475831 trunc from 0x039601 to 0x039d63 (0x763 bytes) 000226 1787395066.578493 punch from 0x0044a0 to 0x0101ad (0xbd0e bytes) 000227 1787395066.589454 mapwrite 0x024232 thru 0x030d61 (0xcb30 bytes) 000228 1787395066.687843 read 0x01ccc8 thru 0x022690 (0x59c9 bytes) 000229 1787395066.703554 trunc from 0x039d64 to 0x02c131 (0xdc32 bytes) 000230 1787395066.742905 trunc from 0x01dd6b to 0x01ede3 (0x1079 bytes) 000231 1787395066.748505 trunc from 0x01106e to 0x01ffb1 (0xef44 bytes) 000232 1787395066.761346 punch from 0x0250ac to 0x02afea (0x5f3f bytes) 000233 1787395066.776010 trunc from 0x0226d1 to 0x02c131 (0x9a61 bytes) 000234 1787395066.791500 trunc from 0x02c132 to 0x00312c (0x29005 bytes) 000235 1787395066.826165 trunc from 0x002763 to 0x00312c (0x9ca bytes) 000236 1787395066.846725 trunc from 0x00312d to 0x031833 (0x2e707 bytes) 000237 1787395066.877186 trunc from 0x031834 to 0x03c9e9 (0xb1b6 bytes) 000238 1787395066.914779 fallocate 0x00e246 thru 0x01c76c (0xe527 bytes) 000239 1787395066.926613 read 0x003084 thru 0x007a57 (0x49d4 bytes) 000240 1787395066.974886 mapwrite 0x0057c0 thru 0x011942 (0xc183 bytes) 000241 1787395067.035102 read 0x03832f thru 0x03c9e9 (0x46bb bytes) 000242 1787395067.036229 fallocate 0x03d9a7 thru 0x03ffff (0x2659 bytes) 000243 1787395067.046692 trunc from 0x02613f to 0x02f7e1 (0x96a3 bytes) 000244 1787395067.057314 fallocate 0x0372c8 thru 0x03fa78 (0x87b1 bytes) 000245 1787395067.066244 read 0x0142b6 thru 0x020a8d (0xc7d8 bytes) 000246 1787395067.083800 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fa79): Mirror 2: 0x496c5392 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fa79): Mirror 2: 0x496c5392 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fa79: skipped 000247 1787395067.465745 MIRROR_RESYNC 2 mirrors 000248 1787395068.243589 mapwrite 0x037d51 thru 0x03ffff (0x82af bytes) 000249 1787395068.354301 fallocate 0x034a82 thru 0x03ffff (0xb57e bytes) 000250 1787395068.363272 read 0x00cf95 thru 0x01aa5e (0xdaca bytes) 000251 1787395068.384025 trunc from 0x00785c to 0x008cff (0x14a4 bytes) 000252 1787395068.395728 read 0x01d4d2 thru 0x02b7c2 (0xe2f1 bytes) 000253 1787395068.423523 mapread 0x034bea thru 0x039414 (0x482b bytes) 000254 1787395068.426660 write 0x034c07 thru 0x036fd7 (0x23d1 bytes) 000257 1787395068.429224 read 0x0166e6 thru 0x0230af (0xc9ca bytes) 000258 1787395068.432468 mapwrite 0x00697e thru 0x0119c3 (0xb046 bytes) 000259 1787395068.488542 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x106e4b0a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb743f520 Mirror 3: 0xb743f520 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000260 1787395069.506550 mapwrite 0x021970 thru 0x0293f3 (0x7a84 bytes) 000261 1787395069.637690 fallocate 0x03d7c8 thru 0x03ffff (0x2838 bytes) 000263 1787395069.647775 mapread 0x002f74 thru 0x00b284 (0x8311 bytes) 000264 1787395069.678154 mapread 0x01d7a7 thru 0x01e1fc (0xa56 bytes) 000265 1787395069.682160 mapwrite 0x00beab thru 0x01232b (0x6481 bytes) 000266 1787395069.763447 read 0x01923b thru 0x021e74 (0x8c3a bytes) 000267 1787395069.778225 mapwrite 0x01d668 thru 0x01e443 (0xddc bytes) 000268 1787395069.823204 read 0x02e9a9 thru 0x0313e2 (0x2a3a bytes) 000269 1787395069.855277 write 0x01e9cb thru 0x02e3f2 (0xfa28 bytes) 000270 1787395069.865689 trunc from 0x018afe to 0x0209b0 (0x7eb3 bytes) 000271 1787395069.891387 trunc from 0x040000 to 0x013d58 (0x2c2a7 bytes) 000272 1787395069.938550 trunc from 0x00b58b to 0x01275d (0x71d3 bytes) 000273 1787395069.949323 mapread 0x00f3b7 thru 0x012f85 (0x3bcf bytes) 000274 1787395069.995291 write 0x031483 thru 0x03cc75 (0xb7f3 bytes) 000275 1787395070.017484 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3cc76): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cc76): Mirror 2: 0x236eb134 (stale) Mirror 3: 0x1d2f54a (stale) CRC-32 checksum value for chunk [0, 0x3cc76): Mirror 1: 0xb2df04ea Mirror 4: 0xb2df04ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cc76: skipped 000276 1787395071.314879 trunc from 0x03cc76 to 0x02bd43 (0x10f32 bytes) 000277 1787395071.403652 punch from 0x018c74 to 0x01942f (0x7bc bytes) 000278 1787395071.413372 read 0x01e54e thru 0x024d86 (0x6839 bytes) 000279 1787395071.423686 punch from 0x02198e to 0x02bd43 (0xa3b6 bytes) 000280 1787395071.433656 trunc from 0x025b77 to 0x026e2c (0x12b6 bytes) 000281 1787395071.443003 trunc from 0x02ad08 to 0x02bd43 (0x103c bytes) 000282 1787395071.449176 write 0x01e15d thru 0x0297ef (0xb693 bytes) 000283 1787395071.456437 trunc from 0x02a508 to 0x02bd43 (0x183c bytes) 000284 1787395071.464304 trunc from 0x02bd44 to 0x01fd48 (0xbffb bytes) 000285 1787395071.484901 punch from 0x00d0be to 0x01ccc7 (0xfc0a bytes) 000286 1787395071.489713 mapread 0x00a2f6 thru 0x01a265 (0xff70 bytes) 000287 1787395071.520572 mapwrite 0x000c30 thru 0x004d47 (0x4118 bytes) 000288 1787395071.554561 mapread 0x00227a thru 0x0121a1 (0xff28 bytes) 000289 1787395071.556007 read 0x01a6a9 thru 0x01fd48 (0x56a0 bytes) 000290 1787395071.557116 trunc from 0x013d60 to 0x0191be (0x545f bytes) 000291 1787395071.562469 trunc from 0x01fd49 to 0x03a758 (0x1aa10 bytes) 000292 1787395071.584810 trunc from 0x023651 to 0x0260b6 (0x2a66 bytes) 000293 1787395071.590252 fallocate 0x0345fc thru 0x039cf4 (0x56f9 bytes) 000294 1787395071.595599 punch from 0x0336e1 to 0x03a758 (0x7078 bytes) 000296 1787395071.601300 trunc from 0x02b27a to 0x0352c7 (0xa04e bytes) 000297 1787395071.617183 mapwrite 0x002a40 thru 0x00e8b1 (0xbe72 bytes) 000298 1787395071.662570 trunc from 0x03a759 to 0x018c8e (0x21aca bytes) 000299 1787395071.688726 trunc from 0x018c8f to 0x02b4fc (0x1286e bytes) 000300 1787395071.708679 trunc from 0x02b4fd to 0x0056c6 (0x25e36 bytes) 000301 1787395071.726395 trunc from 0x0056c7 to 0x01dbc6 (0x18500 bytes) 000302 1787395071.751898 mapwrite 0x003a07 thru 0x012b8d (0xf187 bytes) 000303 1787395071.803836 trunc from 0x01dbc7 to 0x008347 (0x1587f bytes) 000304 1787395071.825395 trunc from 0x000c30 to 0x008347 (0x7718 bytes) 000305 1787395071.832240 mapwrite 0x011b5d thru 0x018e39 (0x72dd bytes) 000306 1787395071.927775 punch from 0x00e71a to 0x018e39 (0xa720 bytes) 000307 1787395071.938598 punch from 0x014a75 to 0x018e39 (0x43c5 bytes) 000308 1787395071.947471 punch from 0x005c96 to 0x0143cc (0xe737 bytes) 000309 1787395071.954352 mapread 0x016507 thru 0x018e39 (0x2933 bytes) 000311 1787395071.973778 punch from 0x0181f3 to 0x018e39 (0xc47 bytes) 000312 1787395071.979643 trunc from 0x0044c5 to 0x0117fa (0xd336 bytes) 000313 1787395071.985451 mapwrite 0x03402b thru 0x03e846 (0xa81c bytes) 000314 1787395072.035910 fallocate 0x01325e thru 0x01a1f3 (0x6f96 bytes) 000315 1787395072.045734 mapwrite 0x012a67 thru 0x013db0 (0x134a bytes) 000316 1787395072.072833 fallocate 0x00b7e3 thru 0x0152dc (0x9afa bytes) 000317 1787395072.080101 punch from 0x01e09e to 0x02dc9a (0xfbfd bytes) 000318 1787395072.088279 mapwrite 0x01b526 thru 0x023060 (0x7b3b bytes) 000319 1787395072.208729 mapwrite 0x0066b3 thru 0x00ac54 (0x45a2 bytes) 000320 1787395072.248181 mapread 0x00c2db thru 0x017e38 (0xbb5e bytes) 000321 1787395072.288593 read 0x034a04 thru 0x03779a (0x2d97 bytes) 000323 1787395072.292250 write 0x017ea7 thru 0x021049 (0x91a3 bytes) 000325 1787395072.294719 trunc from 0x00d083 to 0x019d6e (0xccec bytes) 000326 1787395072.309484 mapwrite 0x00580d thru 0x009b0a (0x42fe bytes) 000328 1787395072.335093 read 0x03e0f1 thru 0x03e846 (0x756 bytes) 000329 1787395072.335703 write 0x00c210 thru 0x00d9ad (0x179e bytes) 000331 1787395072.337375 mapwrite 0x01351d thru 0x01bbd0 (0x86b4 bytes) 000332 1787395072.361590 trunc from 0x020405 to 0x02b68d (0xb289 bytes) 000333 1787395072.367614 punch from 0x0167d4 to 0x01e5d4 (0x7e01 bytes) 000334 1787395072.372273 punch from 0x033d4b to 0x03e718 (0xa9ce bytes) 000335 1787395072.382126 write 0x006850 thru 0x00bc29 (0x53da bytes) 000336 1787395072.383422 punch from 0x0089bd to 0x010cf9 (0x833d bytes) 000337 1787395072.397550 trunc from 0x03e847 to 0x02d433 (0x11413 bytes) 000338 1787395072.411172 fallocate 0x038984 thru 0x03d192 (0x480f bytes) 000339 1787395072.417240 mapwrite 0x011faf thru 0x0127a3 (0x7f5 bytes) 000341 1787395072.440318 mapread 0x023fb3 thru 0x025ee7 (0x1f35 bytes) 000342 1787395072.454615 punch from 0x030743 to 0x033089 (0x2947 bytes) 000343 1787395072.463243 trunc from 0x028795 to 0x036533 (0xdd9f bytes) 000344 1787395072.480149 mapread 0x0167e7 thru 0x01d60c (0x6e26 bytes) 000345 1787395072.481898 mapwrite 0x037db4 thru 0x03ffff (0x824c bytes) 000346 1787395072.554387 fallocate 0x03f63d thru 0x03f8d6 (0x29a bytes) 000347 1787395072.562350 fallocate 0x03c5bd thru 0x03fe20 (0x3864 bytes) 000348 1787395072.570483 write 0x02d90d thru 0x038aad (0xb1a1 bytes) 000349 1787395072.578463 trunc from 0x040000 to 0x002b92 (0x3d46d bytes) 000350 1787395072.597386 read 0x000841 thru 0x002b92 (0x2352 bytes) 000351 1787395072.598651 fallocate 0x03eb32 thru 0x03ffff (0x14ce bytes) 000352 1787395072.605601 trunc from 0x007dc2 to 0x009888 (0x1ac7 bytes) 000353 1787395072.616095 MIRROR_RESYNC 4 mirrors 000354 1787395073.678604 read 0x00735a thru 0x00b98b (0x4632 bytes) 000355 1787395073.693905 MIRROR_EXTEND to 5 mirrors 000356 1787395074.424328 read 0x01093c thru 0x01293d (0x2002 bytes) 000357 1787395074.449418 trunc from 0x03b264 to 0x03ffff (0x4d9c bytes) 000358 1787395074.489148 trunc from 0x040000 to 0x03bd82 (0x427d bytes) 000359 1787395074.534596 fallocate 0x01e9ed thru 0x02cadc (0xe0f0 bytes) 000361 1787395074.556365 mapread 0x0028ac thru 0x006179 (0x38ce bytes) 000362 1787395074.599184 punch from 0x0063d4 to 0x00d9ef (0x761c bytes) 000363 1787395074.613777 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3bd83): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd83): Mirror 2: 0x15ae2bad (stale) Mirror 3: 0x15ae2bad (stale) Mirror 4: 0x15ae2bad (stale) Mirror 5: 0x15ae2bad (stale) CRC-32 checksum value for chunk [0, 0x3bd83): Mirror 1: 0x15ae2bad Mirror 6: 0x15ae2bad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd83: skipped 000364 1787395075.692767 mapwrite 0x03bc12 thru 0x03ffff (0x43ee bytes) 000365 1787395075.775544 mapwrite 0x027228 thru 0x02a626 (0x33ff bytes) 000367 1787395075.807451 fallocate 0x00eb08 thru 0x013555 (0x4a4e bytes) 000368 1787395075.812660 punch from 0x00729b to 0x01714b (0xfeb1 bytes) 000369 1787395075.817338 mapread 0x010a9c thru 0x013905 (0x2e6a bytes) 000370 1787395075.833583 punch from 0x0111d9 to 0x013dee (0x2c16 bytes) 000371 1787395075.843701 punch from 0x02f0ae to 0x03cabc (0xda0f bytes) 000372 1787395075.849498 read 0x03adec thru 0x03ffff (0x5214 bytes) 000374 1787395075.855806 mapread 0x02404f thru 0x02f6ba (0xb66c bytes) 000375 1787395075.865134 write 0x004289 thru 0x006115 (0x1e8d bytes) 000376 1787395075.871797 fallocate 0x03f0a3 thru 0x03ffff (0xf5d bytes) 000377 1787395075.876403 write 0x0059a4 thru 0x0122a2 (0xc8ff bytes) 000378 1787395075.886540 mapwrite 0x01483f thru 0x0195b3 (0x4d75 bytes) 000379 1787395075.907139 read 0x02c2b6 thru 0x03b7a5 (0xf4f0 bytes) 000380 1787395075.917236 mapwrite 0x01cfef thru 0x02878e (0xb7a0 bytes) 000381 1787395075.936641 mapread 0x00f58b thru 0x01240e (0x2e84 bytes) 000382 1787395075.938648 punch from 0x01e718 to 0x0213a5 (0x2c8e bytes) 000383 1787395075.943704 fallocate 0x0153b2 thru 0x01b8fc (0x654b bytes) 000384 1787395075.952334 mapwrite 0x022cd7 thru 0x02de55 (0xb17f bytes) 000385 1787395075.978832 read 0x0250cc thru 0x02dd80 (0x8cb5 bytes) 000386 1787395075.979732 trunc from 0x040000 to 0x00c2e5 (0x33d1a bytes) 000387 1787395076.017699 punch from 0x00655a to 0x00c2e5 (0x5d8c bytes) 000388 1787395076.029512 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xc2e6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc2e6): Mirror 2: 0x450e7fe7 (stale) Mirror 3: 0x450e7fe7 (stale) Mirror 4: 0x450e7fe7 (stale) Mirror 5: 0x450e7fe7 (stale) Mirror 6: 0x450e7fe7 (stale) CRC-32 checksum value for chunk [0, 0xc2e6): Mirror 1: 0xc1232341 Mirror 7: 0xc1232341 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc2e6: skipped 000389 1787395077.135538 read 0x009336 thru 0x00c2e5 (0x2fb0 bytes) 000390 1787395077.153809 fallocate 0x02a521 thru 0x032ecc (0x89ac bytes) 000391 1787395077.230795 punch from 0x005d0c to 0x009321 (0x3616 bytes) 000392 1787395077.240868 trunc from 0x003c2b to 0x009e10 (0x61e6 bytes) 000395 1787395077.268833 MIRROR_RESYNC 7 mirrors 000396 1787395078.623395 punch from 0x00ba8a to 0x00c2e5 (0x85c bytes) 000397 1787395078.660206 punch from 0x007335 to 0x00c2e5 (0x4fb1 bytes) 000398 1787395078.665603 punch from 0x0060e2 to 0x0075b0 (0x14cf bytes) 000399 1787395078.674170 fallocate 0x03efa6 thru 0x03ffff (0x105a bytes) 000400 1787395078.679797 trunc from 0x00c2e6 to 0x003a16 (0x88cf bytes) 000401 1787395078.716153 punch from 0x0033f9 to 0x003a16 (0x61e bytes) 000402 1787395078.720580 mapwrite 0x03f939 thru 0x03ffff (0x6c7 bytes) 000403 1787395078.764498 write 0x01b9b4 thru 0x02290a (0x6f57 bytes) 000404 1787395078.778243 read 0x0199fb thru 0x01fbdf (0x61e5 bytes) 000405 1787395078.793801 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 3: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) Mirror 5: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x927b749a Mirror 8: 0x927b749a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000406 1787395080.178623 mapwrite 0x015b7b thru 0x022250 (0xc6d6 bytes) 000407 1787395080.388319 trunc from 0x040000 to 0x01e5ec (0x21a13 bytes) 000408 1787395080.499768 write 0x019c25 thru 0x021cfc (0x80d8 bytes) 000409 1787395080.504638 fallocate 0x02f8b0 thru 0x034b3b (0x528c bytes) 000410 1787395080.524572 read 0x02e447 thru 0x034b3b (0x66f5 bytes) 000411 1787395080.540030 trunc from 0x030300 to 0x034b3b (0x483c bytes) 000412 1787395080.555452 MIRROR_RESYNC 8 mirrors 000413 1787395082.450496 MIRROR_RESYNC 8 mirrors 000414 1787395082.985711 fallocate 0x00dd3d thru 0x01b92f (0xdbf3 bytes) 000415 1787395083.039743 trunc from 0x0105af to 0x01086a (0x2bc bytes) 000416 1787395083.061859 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34b3c): Mirror 2: 0xf3bb13cd (stale) Mirror 3: 0xf3bb13cd (stale) Mirror 4: 0xf3bb13cd (stale) Mirror 6: 0xf3bb13cd (stale) Mirror 7: 0xf3bb13cd (stale) Mirror 8: 0xf3bb13cd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34b3c): Mirror 2: 0xf3bb13cd (stale) Mirror 3: 0xf3bb13cd (stale) Mirror 4: 0xf3bb13cd (stale) Mirror 6: 0xf3bb13cd (stale) Mirror 7: 0xf3bb13cd (stale) Mirror 8: 0xf3bb13cd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34b3c: skipped 000417 1787395083.564625 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x34b3c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34b3c): Mirror 2: 0xf3bb13cd (stale) Mirror 3: 0xf3bb13cd (stale) Mirror 4: 0xf3bb13cd (stale) Mirror 6: 0xf3bb13cd (stale) Mirror 7: 0xf3bb13cd (stale) Mirror 8: 0xf3bb13cd (stale) CRC-32 checksum value for chunk [0, 0x34b3c): Mirror 1: 0xf3bb13cd Mirror 9: 0xf3bb13cd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34b3c: skipped 000418 1787395084.713285 trunc from 0x034b3c to 0x02a887 (0xa2b4 bytes) 000419 1787395084.815472 read 0x002bf9 thru 0x0105fb (0xda03 bytes) 000420 1787395084.837458 punch from 0x029a79 to 0x02a887 (0xe0f bytes) 000421 1787395084.843857 mapread 0x0205d1 thru 0x02293b (0x236b bytes) 000422 1787395084.880998 read 0x0150aa thru 0x01af22 (0x5e79 bytes) 000423 1787395084.897327 write 0x0210fa thru 0x0261c4 (0x50cb bytes) 000424 1787395084.902670 read 0x027a61 thru 0x02a887 (0x2e27 bytes) 000425 1787395084.904163 mapwrite 0x003578 thru 0x008d33 (0x57bc bytes) 000426 1787395084.936276 mapread 0x004fc1 thru 0x005619 (0x659 bytes) 000427 1787395084.938414 trunc from 0x027368 to 0x02a887 (0x3520 bytes) 000429 1787395084.957454 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2a888): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a888): Mirror 2: 0x53a86e34 (stale) Mirror 3: 0x53a86e34 (stale) Mirror 4: 0x53a86e34 (stale) Mirror 6: 0x53a86e34 (stale) Mirror 7: 0x53a86e34 (stale) Mirror 8: 0x53a86e34 (stale) Mirror 9: 0x53a86e34 (stale) CRC-32 checksum value for chunk [0, 0x2a888): Mirror 1: 0x6ccd92fe Mirror 10: 0x6ccd92fe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a888: skipped 000430 1787395086.183177 trunc from 0x02a888 to 0x02b949 (0x10c2 bytes) 000431 1787395086.234194 punch from 0x015c0f to 0x01f6cf (0x9ac1 bytes) 000433 1787395086.240308 read 0x01ee3a thru 0x021224 (0x23eb bytes) 000435 1787395086.251526 mapread 0x01f006 thru 0x02b289 (0xc284 bytes) 000436 1787395086.272805 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2b94a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b94a): Mirror 2: 0x27a2386f (stale) Mirror 3: 0x27a2386f (stale) Mirror 4: 0x27a2386f (stale) Mirror 6: 0x27a2386f (stale) Mirror 7: 0x27a2386f (stale) Mirror 8: 0x27a2386f (stale) Mirror 9: 0x27a2386f (stale) Mirror 10: 0x45a00fad (stale) CRC-32 checksum value for chunk [0, 0x2b94a): Mirror 1: 0x5a1a1656 Mirror 11: 0x5a1a1656 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b94a: skipped 000437 1787395087.508080 read 0x000ad2 thru 0x006a52 (0x5f81 bytes) 000439 1787395087.549706 mapwrite 0x02f540 thru 0x0364d0 (0x6f91 bytes) 000440 1787395087.742642 punch from 0x009db8 to 0x00d427 (0x3670 bytes) 000441 1787395087.748422 mapread 0x0062b1 thru 0x00aaf7 (0x4847 bytes) 000442 1787395087.764355 mapwrite 0x039e96 thru 0x03ffff (0x616a bytes) 000444 1787395087.848545 mapread 0x00d4b0 thru 0x01adc9 (0xd91a bytes) 000445 1787395087.872093 trunc from 0x040000 to 0x02ad7c (0x15283 bytes) 000446 1787395087.911740 mapread 0x029d1e thru 0x02ad7c (0x105f bytes) 000447 1787395087.925201 read 0x01353a thru 0x021565 (0xe02c bytes) 000448 1787395087.926209 punch from 0x021406 to 0x021843 (0x43e bytes) 000449 1787395087.936676 write 0x03b969 thru 0x03e8d7 (0x2f6f bytes) 000450 1787395087.938807 mapwrite 0x03d389 thru 0x03ffff (0x2c77 bytes) 000451 1787395087.990504 read 0x022e4e thru 0x032362 (0xf515 bytes) 000452 1787395087.994699 mapread 0x006d88 thru 0x014ae4 (0xdd5d bytes) 000453 1787395087.999446 fallocate 0x0069f6 thru 0x0144ff (0xdb0a bytes) 000454 1787395088.009347 trunc from 0x01762a to 0x0230d5 (0xbaac bytes) 000455 1787395088.017615 trunc from 0x040000 to 0x0138cf (0x2c730 bytes) 000456 1787395088.033671 mapwrite 0x036bca thru 0x03fe43 (0x927a bytes) 000457 1787395088.079050 punch from 0x031252 to 0x0379bd (0x676c bytes) 000458 1787395088.083447 mapread 0x02ddb7 thru 0x0391da (0xb424 bytes) 000459 1787395088.113786 mapwrite 0x0071ea thru 0x007b2d (0x944 bytes) 000460 1787395088.132409 trunc from 0x024091 to 0x02c1e5 (0x8155 bytes) 000461 1787395088.139721 write 0x01a6a1 thru 0x01ceaa (0x280a bytes) 000462 1787395088.141740 mapwrite 0x02add2 thru 0x03119c (0x63cb bytes) 000463 1787395088.221002 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fe44): Mirror 2: 0x7f677ab7 (stale) Mirror 3: 0x7f677ab7 (stale) Mirror 4: 0x7f677ab7 (stale) Mirror 6: 0x7f677ab7 (stale) Mirror 7: 0x7f677ab7 (stale) Mirror 8: 0x7f677ab7 (stale) Mirror 9: 0x7f677ab7 (stale) Mirror 11: 0x50d2292e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe44): Mirror 2: 0x7f677ab7 (stale) Mirror 3: 0x7f677ab7 (stale) Mirror 4: 0x7f677ab7 (stale) Mirror 6: 0x7f677ab7 (stale) Mirror 7: 0x7f677ab7 (stale) Mirror 8: 0x7f677ab7 (stale) Mirror 9: 0x7f677ab7 (stale) Mirror 11: 0x50d2292e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe44: skipped 000464 1787395088.972625 trunc from 0x03fe44 to 0x02588f (0x1a5b4 bytes) 000465 1787395088.997310 fallocate 0x01659f thru 0x02351d (0xcf7f bytes) 000466 1787395089.003331 fallocate 0x01baf2 thru 0x02aed0 (0xf3df bytes) 000467 1787395089.009614 mapread 0x020c08 thru 0x02486f (0x3c68 bytes) 000468 1787395089.032399 trunc from 0x025890 to 0x02e20c (0x897d bytes) 000470 1787395089.049706 mapread 0x004110 thru 0x010956 (0xc847 bytes) 000471 1787395089.063350 mapread 0x007a5f thru 0x00804b (0x5ed bytes) 000473 1787395089.065419 punch from 0x02d2d5 to 0x02ddd8 (0xb04 bytes) 000474 1787395089.070169 punch from 0x02a0fc to 0x02a241 (0x146 bytes) 000475 1787395089.076718 read 0x000ff9 thru 0x00448e (0x3496 bytes) 000476 1787395089.082233 punch from 0x009e42 to 0x012687 (0x8846 bytes) 000477 1787395089.087708 read 0x01f274 thru 0x02e20c (0xef99 bytes) 000478 1787395089.108093 write 0x022365 thru 0x02e29b (0xbf37 bytes) 000479 1787395089.110237 mapread 0x02db55 thru 0x02e29b (0x747 bytes) 000480 1787395089.111746 fallocate 0x00a0b0 thru 0x016b9e (0xcaef bytes) 000481 1787395089.117233 mapwrite 0x015bd4 thru 0x0206e0 (0xab0d bytes) 000482 1787395089.140299 fallocate 0x0099ef thru 0x0161f7 (0xc809 bytes) 000483 1787395089.146052 mapread 0x024051 thru 0x029734 (0x56e4 bytes) 000484 1787395089.147613 punch from 0x0149a3 to 0x023747 (0xeda5 bytes) 000485 1787395089.154661 fallocate 0x00ffbb thru 0x01b091 (0xb0d7 bytes) 000486 1787395089.162717 mapread 0x0265f7 thru 0x02e29b (0x7ca5 bytes) 000487 1787395089.165266 trunc from 0x0022d8 to 0x004968 (0x2691 bytes) 000488 1787395089.174299 write 0x00492b thru 0x0124ed (0xdbc3 bytes) 000489 1787395089.179373 read 0x008594 thru 0x0154d2 (0xcf3f bytes) 000490 1787395089.189264 mapwrite 0x035c5f thru 0x03d7fe (0x7ba0 bytes) 000491 1787395089.249532 write 0x035cc0 thru 0x03ffff (0xa340 bytes) 000492 1787395089.251120 read 0x016e96 thru 0x0204f6 (0x9661 bytes) 000493 1787395089.252607 trunc from 0x040000 to 0x011fdf (0x2e020 bytes) 000495 1787395089.270187 read 0x00931f thru 0x009dcd (0xaaf bytes) 000496 1787395089.273252 read 0x005704 thru 0x011fdf (0xc8dc bytes) 000498 1787395089.275443 punch from 0x00f420 to 0x011fdf (0x2bc0 bytes) 000499 1787395089.281462 read 0x00dcff thru 0x011fdf (0x42e1 bytes) 000500 1787395089.292294 trunc from 0x011fe0 to 0x00c17c (0x5e63 bytes) 000501 1787395089.337348 mapread 0x003acb thru 0x0070bf (0x35f5 bytes) 000502 1787395089.351662 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0xc17d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc17d): Mirror 2: 0x454139e3 (stale) Mirror 3: 0x454139e3 (stale) Mirror 4: 0x454139e3 (stale) Mirror 6: 0x454139e3 (stale) Mirror 7: 0x454139e3 (stale) Mirror 8: 0x454139e3 (stale) Mirror 9: 0x454139e3 (stale) Mirror 11: 0x1cc27521 (stale) CRC-32 checksum value for chunk [0, 0xc17d): Mirror 1: 0x656b6ad Mirror 12: 0x656b6ad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc17d: skipped 000503 1787395090.447491 trunc from 0x00369a to 0x00c17c (0x8ae3 bytes) 000504 1787395090.507689 fallocate 0x017d1a thru 0x01e6e0 (0x69c7 bytes) 000505 1787395090.515303 mapread 0x01adbf thru 0x01e6e0 (0x3922 bytes) 000507 1787395090.545143 mapwrite 0x01808f thru 0x023e47 (0xbdb9 bytes) 000508 1787395090.637785 mapwrite 0x008b1b thru 0x00b92f (0x2e15 bytes) 000509 1787395090.662577 mapwrite 0x007148 thru 0x00c5b4 (0x546d bytes) 000510 1787395090.690524 write 0x005759 thru 0x00b282 (0x5b2a bytes) 000511 1787395090.691911 read 0x0208a8 thru 0x023e47 (0x35a0 bytes) 000512 1787395090.693571 read 0x01e1ee thru 0x023e47 (0x5c5a bytes) 000513 1787395090.694797 mapread 0x007782 thru 0x00bdb0 (0x462f bytes) 000514 1787395090.697714 read 0x00bf8a thru 0x010ba6 (0x4c1d bytes) 000515 1787395090.698586 read 0x007fb9 thru 0x011c56 (0x9c9e bytes) 000517 1787395090.710881 read 0x01c207 thru 0x023e47 (0x7c41 bytes) 000518 1787395090.711580 mapread 0x011f4c thru 0x01f1a1 (0xd256 bytes) 000519 1787395090.714745 punch from 0x0233ac to 0x023e47 (0xa9c bytes) 000520 1787395090.721613 fallocate 0x011be4 thru 0x01c063 (0xa480 bytes) 000521 1787395090.727658 mapwrite 0x039dd4 thru 0x03ffff (0x622c bytes) 000523 1787395090.783369 MIRROR_RESYNC 10 mirrors 000524 1787395094.025867 read 0x004382 thru 0x00b723 (0x73a2 bytes) 000525 1787395094.033747 punch from 0x0237d5 to 0x0238ba (0xe6 bytes) 000526 1787395094.114585 trunc from 0x01c6aa to 0x01e838 (0x218f bytes) 000527 1787395094.127536 mapread 0x023fb3 thru 0x02d554 (0x95a2 bytes) 000528 1787395094.169630 read 0x03b7c9 thru 0x03ce9e (0x16d6 bytes) 000530 1787395094.182752 read 0x006325 thru 0x014e50 (0xeb2c bytes) 000531 1787395094.189401 trunc from 0x00c3ad to 0x00c792 (0x3e6 bytes) 000532 1787395094.208110 trunc from 0x040000 to 0x02fb27 (0x104d8 bytes) 000533 1787395094.248149 write 0x024eef thru 0x028a8f (0x3ba1 bytes) 000534 1787395094.250042 trunc from 0x01657f to 0x0175dd (0x105f bytes) 000535 1787395094.255312 fallocate 0x03c818 thru 0x03ffff (0x37e8 bytes) 000536 1787395094.260392 mapread 0x008866 thru 0x011e1f (0x95ba bytes) 000537 1787395094.277317 MIRROR_RESYNC 10 mirrors 000538 1787395097.794357 write 0x01d1f5 thru 0x024b0f (0x791b bytes) 000539 1787395097.897191 trunc from 0x02314d to 0x032283 (0xf137 bytes) 000540 1787395097.925266 mapread 0x039378 thru 0x03992b (0x5b4 bytes) 000541 1787395097.936610 write 0x01325c thru 0x01e17e (0xaf23 bytes) 000542 1787395097.949340 punch from 0x003d63 to 0x00c643 (0x88e1 bytes) 000544 1787395097.957983 fallocate 0x00e45c thru 0x0105e2 (0x2187 bytes) 000545 1787395097.964202 trunc from 0x040000 to 0x005ecf (0x3a130 bytes) 000546 1787395097.991970 mapwrite 0x0394eb thru 0x03ffff (0x6b15 bytes) 000547 1787395098.048149 punch from 0x0309ee to 0x03a695 (0x9ca8 bytes) 000548 1787395098.054364 trunc from 0x040000 to 0x00f258 (0x30da7 bytes) 000549 1787395098.076362 punch from 0x006acf to 0x00ee4a (0x837c bytes) 000550 1787395098.088412 MIRROR_RESYNC 10 mirrors 000551 1787395099.878673 MIRROR_EXTEND to 11 mirrors 000552 1787395100.632572 fallocate 0x02a52c thru 0x02be8b (0x1960 bytes) 000553 1787395100.699710 read 0x005bf5 thru 0x0092fc (0x3708 bytes) 000554 1787395100.718099 mapwrite 0x02d511 thru 0x02fcf8 (0x27e8 bytes) 000555 1787395100.826476 trunc from 0x02fcf9 to 0x027143 (0x8bb5 bytes) 000556 1787395100.854532 write 0x02d363 thru 0x034b66 (0x7804 bytes) 000557 1787395100.868419 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x34b67): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34b67): Mirror 2: 0x834f3b7d (stale) Mirror 3: 0x834f3b7d (stale) Mirror 4: 0x834f3b7d (stale) Mirror 6: 0x834f3b7d (stale) Mirror 7: 0x834f3b7d (stale) Mirror 8: 0x834f3b7d (stale) Mirror 9: 0x834f3b7d (stale) Mirror 11: 0x834f3b7d (stale) Mirror 12: 0x834f3b7d (stale) Mirror 13: 0x834f3b7d (stale) CRC-32 checksum value for chunk [0, 0x34b67): Mirror 1: 0x6f036dd9 Mirror 14: 0x6f036dd9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34b67: skipped 000559 1787395102.358976 trunc from 0x0202c4 to 0x026115 (0x5e52 bytes) 000560 1787395102.424029 punch from 0x00d46d to 0x017bd0 (0xa764 bytes) 000561 1787395102.430509 fallocate 0x036db1 thru 0x03ffff (0x924f bytes) 000562 1787395102.436866 mapwrite 0x022355 thru 0x0313eb (0xf097 bytes) 000563 1787395102.492383 MIRROR_RESYNC 12 mirrors 000564 1787395105.150104 punch from 0x0320a3 to 0x034b66 (0x2ac4 bytes) 000565 1787395105.192404 write 0x03ef96 thru 0x03ffff (0x106a bytes) 000566 1787395105.193550 read 0x00a7eb thru 0x0119c2 (0x71d8 bytes) 000567 1787395105.205319 mapwrite 0x03fdc5 thru 0x03ffff (0x23b bytes) 000568 1787395105.220772 read 0x00771f thru 0x00e5bb (0x6e9d bytes) 000569 1787395105.241642 fallocate 0x023fe2 thru 0x033925 (0xf944 bytes) 000570 1787395105.260980 trunc from 0x040000 to 0x010b9c (0x2f463 bytes) 000571 1787395105.306053 read 0x004954 thru 0x00ecc4 (0xa371 bytes) 000572 1787395105.316200 mapwrite 0x010968 thru 0x010fce (0x667 bytes) 000573 1787395105.371122 fallocate 0x019eeb thru 0x0263fa (0xc510 bytes) 000574 1787395105.388294 trunc from 0x0263fb to 0x02269e (0x3d5c bytes) 000575 1787395105.417552 mapwrite 0x025f28 thru 0x02bf4e (0x6027 bytes) 000576 1787395105.482093 read 0x0185f3 thru 0x02645d (0xde6b bytes) 000577 1787395105.496138 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x2bf4f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bf4f): Mirror 2: 0x7baefa48 (stale) Mirror 3: 0x7baefa48 (stale) Mirror 4: 0x7baefa48 (stale) Mirror 6: 0x7baefa48 (stale) Mirror 7: 0x7baefa48 (stale) Mirror 8: 0x7baefa48 (stale) Mirror 9: 0x7baefa48 (stale) Mirror 11: 0x7baefa48 (stale) Mirror 12: 0x7baefa48 (stale) Mirror 13: 0x7baefa48 (stale) Mirror 14: 0x7baefa48 (stale) CRC-32 checksum value for chunk [0, 0x2bf4f): Mirror 1: 0x1104180e Mirror 15: 0x1104180e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bf4f: skipped 000578 1787395106.843278 fallocate 0x02f70b thru 0x035d3e (0x6634 bytes) 000579 1787395106.902361 write 0x03db74 thru 0x03ffff (0x248c bytes) 000580 1787395106.913211 punch from 0x00257f to 0x008cda (0x675c bytes) 000581 1787395106.936295 read 0x005628 thru 0x00b0d3 (0x5aac bytes) 000582 1787395106.946411 fallocate 0x025c1c thru 0x029b53 (0x3f38 bytes) 000584 1787395106.952704 fallocate 0x024fdd thru 0x028ad1 (0x3af5 bytes) 000585 1787395106.970329 fallocate 0x032022 thru 0x03ffff (0xdfde bytes) 000586 1787395106.981563 mapwrite 0x00d4b2 thru 0x00e888 (0x13d7 bytes) 000587 1787395107.013623 trunc from 0x040000 to 0x002d96 (0x3d269 bytes) 000588 1787395107.063126 trunc from 0x002d97 to 0x0130c2 (0x1032c bytes) 000589 1787395107.080893 mapwrite 0x005000 thru 0x0105e0 (0xb5e1 bytes) 000590 1787395107.112571 mapwrite 0x01c931 thru 0x01c9a6 (0x76 bytes) 000591 1787395107.147203 fallocate 0x0115f2 thru 0x01c26a (0xac79 bytes) 000592 1787395107.161664 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x1c9a7): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c9a7): Mirror 2: 0xd71ec4e5 (stale) Mirror 3: 0xd71ec4e5 (stale) Mirror 4: 0xd71ec4e5 (stale) Mirror 6: 0xd71ec4e5 (stale) Mirror 7: 0xd71ec4e5 (stale) Mirror 8: 0xd71ec4e5 (stale) Mirror 9: 0xd71ec4e5 (stale) Mirror 11: 0xd71ec4e5 (stale) Mirror 12: 0xd71ec4e5 (stale) Mirror 13: 0xd71ec4e5 (stale) Mirror 14: 0xd71ec4e5 (stale) Mirror 15: 0xdec9217c (stale) CRC-32 checksum value for chunk [0, 0x1c9a7): Mirror 1: 0x62f58331 Mirror 16: 0x62f58331 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c9a7: skipped 000593 1787395108.407226 read 0x005f51 thru 0x00c119 (0x61c9 bytes) 000594 1787395108.425827 read 0x013bcb thru 0x01c9a6 (0x8ddc bytes) 000595 1787395108.449236 trunc from 0x016f33 to 0x01c9a6 (0x5a74 bytes) 000596 1787395108.525269 punch from 0x006a66 to 0x0142c4 (0xd85f bytes) 000597 1787395108.538904 trunc from 0x01c9a7 to 0x004796 (0x18210 bytes) 000598 1787395108.631839 mapwrite 0x01bc4f thru 0x0238fb (0x7cad bytes) 000600 1787395108.770315 mapwrite 0x01855e thru 0x01f3c6 (0x6e69 bytes) 000601 1787395108.837321 fallocate 0x02e883 thru 0x035fcb (0x7749 bytes) 000602 1787395108.848758 punch from 0x0025e3 to 0x00507a (0x2a98 bytes) 000603 1787395108.854794 punch from 0x00ec80 to 0x012f84 (0x4305 bytes) 000604 1787395108.864972 punch from 0x01f6f0 to 0x0238fb (0x420c bytes) 000606 1787395108.878052 mapread 0x007833 thru 0x012f68 (0xb736 bytes) 000607 1787395109.049906 MIRROR_RESYNC 14 mirrors 000608 1787395112.390078 punch from 0x00945b to 0x00e464 (0x500a bytes) 000609 1787395112.447991 write 0x00f3f3 thru 0x01baf7 (0xc705 bytes) 000610 1787395112.463249 mapwrite 0x02b67f thru 0x03715f (0xbae1 bytes) 000611 1787395112.606382 trunc from 0x028691 to 0x0301da (0x7b4a bytes) 000612 1787395112.616406 trunc from 0x037160 to 0x02a2f7 (0xce68 bytes) 000614 1787395112.662744 mapwrite 0x00263a thru 0x00a0c2 (0x7a89 bytes) 000615 1787395112.703451 punch from 0x01587c to 0x023800 (0xdf85 bytes) 000616 1787395112.708564 punch from 0x016358 to 0x020180 (0x9e29 bytes) 000617 1787395112.715377 trunc from 0x02a2f8 to 0x016053 (0x142a4 bytes) 000618 1787395112.731881 punch from 0x0093d4 to 0x00f088 (0x5cb5 bytes) 000619 1787395112.740230 fallocate 0x011536 thru 0x015483 (0x3f4e bytes) 000620 1787395112.745639 fallocate 0x03b5da thru 0x03ffff (0x4a26 bytes) 000621 1787395112.754994 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16054): Mirror 3: 0xb96ccd04 (stale) Mirror 4: 0xb96ccd04 (stale) Mirror 6: 0xb96ccd04 (stale) Mirror 7: 0xb96ccd04 (stale) Mirror 8: 0xb96ccd04 (stale) Mirror 9: 0xb96ccd04 (stale) Mirror 11: 0xb96ccd04 (stale) Mirror 12: 0xb96ccd04 (stale) Mirror 13: 0xb96ccd04 (stale) Mirror 14: 0xb96ccd04 (stale) Mirror 15: 0xb96ccd04 (stale) Mirror 16: 0xb96ccd04 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16054): Mirror 3: 0xb96ccd04 (stale) Mirror 4: 0xb96ccd04 (stale) Mirror 6: 0xb96ccd04 (stale) Mirror 7: 0xb96ccd04 (stale) Mirror 8: 0xb96ccd04 (stale) Mirror 9: 0xb96ccd04 (stale) Mirror 11: 0xb96ccd04 (stale) Mirror 12: 0xb96ccd04 (stale) Mirror 13: 0xb96ccd04 (stale) Mirror 14: 0xb96ccd04 (stale) Mirror 15: 0xb96ccd04 (stale) Mirror 16: 0xb96ccd04 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16054: skipped 000622 1787395113.629019 mapread 0x0049bd thru 0x008e25 (0x4469 bytes) 000624 1787395113.668556 trunc from 0x00bc45 to 0x0105d4 (0x4990 bytes) 000625 1787395113.692089 fallocate 0x03fd0a thru 0x03ffff (0x2f6 bytes) 000626 1787395113.700041 mapwrite 0x021216 thru 0x0213a4 (0x18f bytes) 000627 1787395113.772809 trunc from 0x0213a5 to 0x0065d6 (0x1adce bytes) 000628 1787395113.798169 fallocate 0x02fa5f thru 0x031f5c (0x24fe bytes) 000629 1787395113.806830 write 0x002a6e thru 0x003770 (0xd03 bytes) 000630 1787395113.809181 mapread 0x02a7eb thru 0x02c2e8 (0x1afe bytes) 000631 1787395113.847278 mapread 0x004c3f thru 0x0054fa (0x8bc bytes) 000632 1787395113.849550 read 0x0200f3 thru 0x02360b (0x3519 bytes) 000633 1787395113.859693 trunc from 0x031f5d to 0x028b15 (0x9447 bytes) 000634 1787395113.878375 punch from 0x00acdd to 0x01223e (0x7562 bytes) 000635 1787395113.884803 trunc from 0x028b16 to 0x02ca67 (0x3f52 bytes) 000636 1787395113.903490 write 0x03f9dc thru 0x03ffff (0x624 bytes) 000637 1787395113.909435 MIRROR_RESYNC 13 mirrors 000638 1787395117.549405 mapwrite 0x03d4e9 thru 0x03ffff (0x2b17 bytes) 000639 1787395117.663591 trunc from 0x040000 to 0x03fef9 (0x106 bytes) 000640 1787395117.688219 read 0x005437 thru 0x014e9d (0xfa67 bytes) 000641 1787395117.697586 write 0x03a4cf thru 0x03ffff (0x5b31 bytes) 000642 1787395117.699282 mapwrite 0x03ca06 thru 0x03ffff (0x35fa bytes) 000643 1787395117.718419 fallocate 0x005c65 thru 0x010fde (0xb37a bytes) 000645 1787395117.725943 write 0x03c110 thru 0x03df39 (0x1e2a bytes) 000647 1787395117.735478 fallocate 0x01467c thru 0x01596c (0x12f1 bytes) 000648 1787395117.755534 trunc from 0x040000 to 0x00fcbb (0x30344 bytes) 000649 1787395117.791207 trunc from 0x00d21d to 0x00fcbb (0x2a9f bytes) 000650 1787395117.810583 mapread 0x00b4ac thru 0x00fcbb (0x4810 bytes) 000651 1787395117.830640 read 0x007bbb thru 0x00fb7f (0x7fc5 bytes) 000652 1787395117.832237 mapread 0x00ab67 thru 0x00fcbb (0x5155 bytes) 000653 1787395117.838874 fallocate 0x03a9ff thru 0x03ffff (0x5601 bytes) 000654 1787395117.850501 read 0x03789f thru 0x03fa2a (0x818c bytes) 000655 1787395117.883654 trunc from 0x040000 to 0x035007 (0xaff8 bytes) 000656 1787395117.908194 mapread 0x00892f thru 0x00b03a (0x270c bytes) 000657 1787395117.909893 read 0x02d963 thru 0x02eb0a (0x11a8 bytes) 000658 1787395117.912188 trunc from 0x035008 to 0x02261e (0x129e9 bytes) 000660 1787395117.947353 trunc from 0x01dc57 to 0x02261e (0x49c8 bytes) 000661 1787395117.958422 fallocate 0x008ec6 thru 0x0103a8 (0x74e3 bytes) 000662 1787395117.970334 trunc from 0x008c56 to 0x0136dd (0xaa88 bytes) 000663 1787395117.980582 trunc from 0x02261f to 0x03a905 (0x182e7 bytes) 000664 1787395118.006412 read 0x0327e6 thru 0x03a081 (0x789c bytes) 000665 1787395118.035043 mapread 0x00bec5 thru 0x00c749 (0x885 bytes) 000666 1787395118.038252 fallocate 0x00cdd4 thru 0x01aafe (0xdd2b bytes) 000668 1787395118.071646 read 0x0345bc thru 0x03a905 (0x634a bytes) 000669 1787395118.072601 write 0x000af9 thru 0x00a9a1 (0x9ea9 bytes) 000670 1787395118.077520 mapread 0x022983 thru 0x0275c2 (0x4c40 bytes) 000671 1787395118.081590 mapwrite 0x015db4 thru 0x015fb1 (0x1fe bytes) 000672 1787395118.156043 read 0x0350d1 thru 0x03a905 (0x5835 bytes) 000673 1787395118.157163 mapread 0x01a1d8 thru 0x027426 (0xd24f bytes) 000674 1787395118.170015 mapwrite 0x02b582 thru 0x0352e6 (0x9d65 bytes) 000675 1787395118.214848 fallocate 0x02688a thru 0x02f491 (0x8c08 bytes) 000676 1787395118.225206 write 0x03ca82 thru 0x03f5a0 (0x2b1f bytes) 000677 1787395118.238741 read 0x03328d thru 0x03b48c (0x8200 bytes) 000678 1787395118.247801 mapwrite 0x0210b8 thru 0x02d9f6 (0xc93f bytes) 000679 1787395118.292459 trunc from 0x03f5a1 to 0x029dd8 (0x157c8 bytes) 000680 1787395118.330844 read 0x00c9f2 thru 0x01616d (0x977c bytes) 000681 1787395118.336216 read 0x01998d thru 0x01ad05 (0x1379 bytes) 000682 1787395118.337203 write 0x03b57d thru 0x03d12d (0x1bb1 bytes) 000683 1787395118.338457 read 0x006187 thru 0x009f04 (0x3d7e bytes) 000684 1787395118.340264 read 0x01adc4 thru 0x01fffc (0x5239 bytes) 000685 1787395118.341459 mapread 0x034303 thru 0x03d12d (0x8e2b bytes) 000687 1787395118.390542 MIRROR_RESYNC 13 mirrors 000688 1787395122.702793 trunc from 0x000bcf to 0x00b1a9 (0xa5db bytes) 000689 1787395122.749314 read 0x00001b thru 0x00a8d9 (0xa8bf bytes) 000690 1787395122.761577 read 0x0314b4 thru 0x036bd9 (0x5726 bytes) 000691 1787395122.772179 read 0x01c3a0 thru 0x02b9ce (0xf62f bytes) 000692 1787395122.773169 mapwrite 0x01b6e7 thru 0x01cc9f (0x15b9 bytes) 000693 1787395122.787780 read 0x0300f0 thru 0x03caff (0xca10 bytes) 000694 1787395122.788768 mapread 0x0298ec thru 0x0375b5 (0xdcca bytes) 000695 1787395122.790813 mapread 0x038449 thru 0x03d12d (0x4ce5 bytes) 000696 1787395122.793414 trunc from 0x03d12e to 0x02bf86 (0x111a7 bytes) 000697 1787395122.822346 trunc from 0x02bf87 to 0x03908e (0xd108 bytes) 000698 1787395122.841884 mapread 0x01f21d thru 0x020bc3 (0x19a7 bytes) 000699 1787395122.843148 read 0x01eb5e thru 0x0292e0 (0xa783 bytes) 000700 1787395122.847880 MIRROR_SPLIT mirror 4 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3908f): Mirror 3: 0xc2735255 (stale) Mirror 4: 0xc2735255 (stale) Mirror 6: 0xc2735255 (stale) Mirror 8: 0xc2735255 (stale) Mirror 9: 0xc2735255 (stale) Mirror 11: 0xc2735255 (stale) Mirror 12: 0xc2735255 (stale) Mirror 13: 0xc2735255 (stale) Mirror 14: 0xc2735255 (stale) Mirror 15: 0xc2735255 (stale) Mirror 16: 0xc2735255 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3908f): Mirror 3: 0xc2735255 (stale) Mirror 4: 0xc2735255 (stale) Mirror 6: 0xc2735255 (stale) Mirror 8: 0xc2735255 (stale) Mirror 9: 0xc2735255 (stale) Mirror 11: 0xc2735255 (stale) Mirror 12: 0xc2735255 (stale) Mirror 13: 0xc2735255 (stale) Mirror 14: 0xc2735255 (stale) Mirror 15: 0xc2735255 (stale) Mirror 16: 0xc2735255 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3908f: skipped 000701 1787395123.634037 read 0x0087fe thru 0x008ac8 (0x2cb bytes) 000702 1787395123.640029 fallocate 0x02e8f7 thru 0x033180 (0x488a bytes) 000703 1787395123.648768 mapwrite 0x005977 thru 0x0089a8 (0x3032 bytes) 000705 1787395123.694194 mapwrite 0x0004a9 thru 0x008c6c (0x87c4 bytes) 000706 1787395123.714428 mapread 0x038444 thru 0x03908e (0xc4b bytes) 000707 1787395123.738149 MIRROR_RESYNC 12 mirrors 000709 1787395127.671425 fallocate 0x032a87 thru 0x038f68 (0x64e2 bytes) 000710 1787395127.718695 punch from 0x012af3 to 0x012c5f (0x16d bytes) 000711 1787395127.738235 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3908f): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3908f): Mirror 3: 0x920cbd7a (stale) Mirror 4: 0x920cbd7a (stale) Mirror 6: 0x920cbd7a (stale) Mirror 8: 0x920cbd7a (stale) Mirror 9: 0x920cbd7a (stale) Mirror 11: 0x920cbd7a (stale) Mirror 12: 0x920cbd7a (stale) Mirror 13: 0x920cbd7a (stale) Mirror 14: 0x920cbd7a (stale) Mirror 15: 0x920cbd7a (stale) Mirror 16: 0x920cbd7a (stale) CRC-32 checksum value for chunk [0, 0x3908f): Mirror 1: 0x920cbd7a Mirror 17: 0x920cbd7a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3908f: skipped 000712 1787395129.426281 trunc from 0x00d8b9 to 0x00f30b (0x1a53 bytes) 000713 1787395129.502558 fallocate 0x00f77c thru 0x01703d (0x78c2 bytes) 000714 1787395129.513285 write 0x0187cb thru 0x01c558 (0x3d8e bytes) 000715 1787395129.544179 fallocate 0x024e3c thru 0x02bc8f (0x6e54 bytes) 000716 1787395129.559733 write 0x00eeee thru 0x01d582 (0xe695 bytes) 000717 1787395129.582199 write 0x010db4 thru 0x017f52 (0x719f bytes) 000718 1787395129.591518 trunc from 0x02fd31 to 0x03908e (0x935e bytes) 000719 1787395129.601767 mapread 0x02f9f6 thru 0x0334ec (0x3af7 bytes) 000720 1787395129.651702 trunc from 0x03908f to 0x0055b0 (0x33ade bytes) 000721 1787395129.715976 trunc from 0x0044ab to 0x0055b0 (0x1106 bytes) 000722 1787395129.734421 mapwrite 0x0202cc thru 0x0297f5 (0x952a bytes) 000724 1787395129.851395 MIRROR_SPLIT mirror 9 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x297f6): Mirror 3: 0x5a897a20 (stale) Mirror 4: 0x5a897a20 (stale) Mirror 6: 0x5a897a20 (stale) Mirror 8: 0x5a897a20 (stale) Mirror 9: 0x5a897a20 (stale) Mirror 11: 0x5a897a20 (stale) Mirror 12: 0x5a897a20 (stale) Mirror 13: 0x5a897a20 (stale) Mirror 15: 0x5a897a20 (stale) Mirror 16: 0x5a897a20 (stale) Mirror 17: 0x5a897a20 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x297f6): Mirror 3: 0x5a897a20 (stale) Mirror 4: 0x5a897a20 (stale) Mirror 6: 0x5a897a20 (stale) Mirror 8: 0x5a897a20 (stale) Mirror 9: 0x5a897a20 (stale) Mirror 11: 0x5a897a20 (stale) Mirror 12: 0x5a897a20 (stale) Mirror 13: 0x5a897a20 (stale) Mirror 15: 0x5a897a20 (stale) Mirror 16: 0x5a897a20 (stale) Mirror 17: 0x5a897a20 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x297f6: skipped 000725 1787395130.748846 punch from 0x013f32 to 0x021364 (0xd433 bytes) 000726 1787395130.774393 mapwrite 0x029c4b thru 0x033983 (0x9d39 bytes) 000727 1787395130.963601 punch from 0x000911 to 0x00fed2 (0xf5c2 bytes) 000728 1787395130.987681 trunc from 0x0143d8 to 0x020318 (0xbf41 bytes) 000729 1787395131.002906 write 0x02fc94 thru 0x036f3f (0x72ac bytes) 000730 1787395131.010112 read 0x007ab1 thru 0x00fd7c (0x82cc bytes) 000731 1787395131.032867 write 0x03b416 thru 0x03ed90 (0x397b bytes) 000732 1787395131.034403 write 0x03e659 thru 0x03ffff (0x19a7 bytes) 000733 1787395131.037274 mapread 0x01fe04 thru 0x0221aa (0x23a7 bytes) 000734 1787395131.062182 mapread 0x029717 thru 0x030f37 (0x7821 bytes) 000735 1787395131.065738 trunc from 0x010a9f to 0x020361 (0xf8c3 bytes) 000736 1787395131.098180 MIRROR_RESYNC 12 mirrors 000737 1787395134.355036 fallocate 0x023c53 thru 0x02ded9 (0xa287 bytes) 000738 1787395134.403549 punch from 0x03b658 to 0x03e7ef (0x3198 bytes) 000739 1787395134.411298 punch from 0x008adb to 0x0101b0 (0x76d6 bytes) 000740 1787395134.419488 mapwrite 0x01865f thru 0x0230c5 (0xaa67 bytes) 000741 1787395134.516317 mapread 0x0241d8 thru 0x034087 (0xfeb0 bytes) 000742 1787395134.533549 write 0x033ded thru 0x035199 (0x13ad bytes) 000744 1787395134.551132 MIRROR_SPLIT mirror 9 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xcbaf7104 (stale) Mirror 4: 0xcbaf7104 (stale) Mirror 6: 0xcbaf7104 (stale) Mirror 8: 0xcbaf7104 (stale) Mirror 9: 0xcbaf7104 (stale) Mirror 11: 0xcbaf7104 (stale) Mirror 12: 0xcbaf7104 (stale) Mirror 13: 0xcbaf7104 (stale) Mirror 16: 0xcbaf7104 (stale) Mirror 17: 0xcbaf7104 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xcbaf7104 (stale) Mirror 4: 0xcbaf7104 (stale) Mirror 6: 0xcbaf7104 (stale) Mirror 8: 0xcbaf7104 (stale) Mirror 9: 0xcbaf7104 (stale) Mirror 11: 0xcbaf7104 (stale) Mirror 12: 0xcbaf7104 (stale) Mirror 13: 0xcbaf7104 (stale) Mirror 16: 0xcbaf7104 (stale) Mirror 17: 0xcbaf7104 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000745 1787395135.644690 trunc from 0x040000 to 0x009f8b (0x36074 bytes) 000746 1787395135.698297 punch from 0x0013a3 to 0x009f8b (0x8be9 bytes) 000747 1787395135.707698 mapwrite 0x022d5d thru 0x02e2ca (0xb56e bytes) 000748 1787395135.842011 MIRROR_RESYNC 11 mirrors 000749 1787395137.722120 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a2:0x0]: Invalid argument (22) 000750 1787395138.101310 mapread 0x002890 thru 0x005335 (0x2aa6 bytes) 000751 1787395138.121018 read 0x02a834 thru 0x02e2ca (0x3a97 bytes) 000752 1787395138.129681 mapread 0x01224f thru 0x021998 (0xf74a bytes) 000753 1787395138.145236 mapread 0x023ad9 thru 0x028253 (0x477b bytes) 000754 1787395138.157522 write 0x010010 thru 0x0176fb (0x76ec bytes) 000756 1787395138.225697 fallocate 0x00b4e2 thru 0x017e43 (0xc962 bytes) 000757 1787395138.232406 mapwrite 0x00f199 thru 0x010c27 (0x1a8f bytes) 000758 1787395138.266638 trunc from 0x0209d2 to 0x02ae13 (0xa442 bytes) 000759 1787395138.273753 trunc from 0x02e2cb to 0x03d4ca (0xf200 bytes) 000760 1787395138.298992 mapread 0x03c396 thru 0x03d4ca (0x1135 bytes) 000761 1787395138.319376 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3d4cb): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d4cb): Mirror 3: 0x337f2cc (stale) Mirror 6: 0x337f2cc (stale) Mirror 8: 0x337f2cc (stale) Mirror 9: 0x337f2cc (stale) Mirror 11: 0x337f2cc (stale) Mirror 12: 0x337f2cc (stale) Mirror 13: 0x337f2cc (stale) Mirror 16: 0x337f2cc (stale) Mirror 17: 0x337f2cc (stale) CRC-32 checksum value for chunk [0, 0x3d4cb): Mirror 1: 0x3cb76528 Mirror 18: 0x3cb76528 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d4cb: skipped 000762 1787395139.811791 trunc from 0x03d4cb to 0x001547 (0x3bf83 bytes) 000763 1787395139.926842 fallocate 0x014d5d thru 0x02110b (0xc3af bytes) 000764 1787395139.937324 mapread 0x01edb5 thru 0x02110b (0x2357 bytes) 000765 1787395139.965108 write 0x0192bc thru 0x0209b7 (0x76fc bytes) 000767 1787395139.968271 trunc from 0x02110c to 0x031ce2 (0x10bd7 bytes) 000768 1787395140.000061 write 0x00aa6a thru 0x00b374 (0x90b bytes) 000769 1787395140.015780 mapread 0x005df3 thru 0x012cc5 (0xced3 bytes) 000770 1787395140.083428 mapread 0x009dc1 thru 0x00e7f6 (0x4a36 bytes) 000771 1787395140.093623 trunc from 0x031ce3 to 0x001492 (0x30850 bytes) 000772 1787395140.129692 read 0x00077b thru 0x001492 (0xd18 bytes) 000774 1787395140.137016 mapread 0x000d6c thru 0x001492 (0x727 bytes) 000775 1787395140.142903 read 0x000a11 thru 0x001492 (0xa82 bytes) 000776 1787395140.145437 trunc from 0x001493 to 0x003b79 (0x26e7 bytes) 000778 1787395140.173320 mapread 0x002514 thru 0x003b79 (0x1666 bytes) 000779 1787395140.186901 mapwrite 0x03e3df thru 0x03ffff (0x1c21 bytes) 000780 1787395140.267593 mapwrite 0x03320c thru 0x03d24b (0xa040 bytes) 000781 1787395140.309404 fallocate 0x01635f thru 0x0225f0 (0xc292 bytes) 000782 1787395140.318885 read 0x0246cf thru 0x02b7cb (0x70fd bytes) 000783 1787395140.321445 trunc from 0x022da7 to 0x02f9d9 (0xcc33 bytes) 000784 1787395140.356636 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7a007b0d (stale) Mirror 6: 0x7a007b0d (stale) Mirror 8: 0x7a007b0d (stale) Mirror 9: 0x7a007b0d (stale) Mirror 11: 0x7a007b0d (stale) Mirror 13: 0x7a007b0d (stale) Mirror 16: 0x7a007b0d (stale) Mirror 17: 0x7a007b0d (stale) Mirror 18: 0xe45e9b5e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7a007b0d (stale) Mirror 6: 0x7a007b0d (stale) Mirror 8: 0x7a007b0d (stale) Mirror 9: 0x7a007b0d (stale) Mirror 11: 0x7a007b0d (stale) Mirror 13: 0x7a007b0d (stale) Mirror 16: 0x7a007b0d (stale) Mirror 17: 0x7a007b0d (stale) Mirror 18: 0xe45e9b5e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000785 1787395141.174137 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x7a007b0d (stale) Mirror 8: 0x7a007b0d (stale) Mirror 9: 0x7a007b0d (stale) Mirror 11: 0x7a007b0d (stale) Mirror 13: 0x7a007b0d (stale) Mirror 16: 0x7a007b0d (stale) Mirror 17: 0x7a007b0d (stale) Mirror 18: 0xe45e9b5e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x7a007b0d (stale) Mirror 8: 0x7a007b0d (stale) Mirror 9: 0x7a007b0d (stale) Mirror 11: 0x7a007b0d (stale) Mirror 13: 0x7a007b0d (stale) Mirror 16: 0x7a007b0d (stale) Mirror 17: 0x7a007b0d (stale) Mirror 18: 0xe45e9b5e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000786 1787395142.013771 mapwrite 0x02262f thru 0x02a7b1 (0x8183 bytes) 000787 1787395142.099695 mapread 0x015baa thru 0x01a052 (0x44a9 bytes) 000789 1787395142.114481 write 0x0060a3 thru 0x006d0a (0xc68 bytes) 000790 1787395142.132407 mapwrite 0x021670 thru 0x027f5c (0x68ed bytes) 000791 1787395142.165014 trunc from 0x016643 to 0x01daff (0x74bd bytes) 000792 1787395142.177475 read 0x017320 thru 0x021a7b (0xa75c bytes) 000794 1787395142.205664 mapread 0x004026 thru 0x005774 (0x174f bytes) 000795 1787395142.252207 MIRROR_RESYNC 9 mirrors 000797 1787395145.273831 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a2:0x0]: Invalid argument (22) 000798 1787395145.821683 trunc from 0x0226bd to 0x024737 (0x207b bytes) 000799 1787395145.919193 trunc from 0x040000 to 0x0191c1 (0x26e3e bytes) 000800 1787395145.954127 punch from 0x005266 to 0x0105cf (0xb36a bytes) 000801 1787395145.960127 write 0x01b8b2 thru 0x02416f (0x88be bytes) 000802 1787395145.961760 punch from 0x00dca2 to 0x012a2d (0x4d8c bytes) 000803 1787395145.967867 punch from 0x01abb7 to 0x023a12 (0x8e5c bytes) 000804 1787395145.987477 mapwrite 0x034295 thru 0x03824f (0x3fbb bytes) 000805 1787395146.049040 punch from 0x00c0a3 to 0x019dae (0xdd0c bytes) 000806 1787395146.055379 fallocate 0x0393c1 thru 0x03ffff (0x6c3f bytes) 000807 1787395146.061004 mapwrite 0x03674e thru 0x03ffff (0x98b2 bytes) 000808 1787395146.154783 MIRROR_RESYNC 8 mirrors 000809 1787395148.359541 mapwrite 0x00eb90 thru 0x00ffd4 (0x1445 bytes) 000810 1787395148.490701 write 0x011d04 thru 0x01ca18 (0xad15 bytes) 000811 1787395148.505301 read 0x033101 thru 0x03a725 (0x7625 bytes) 000812 1787395148.518988 trunc from 0x040000 to 0x000505 (0x3fafa bytes) 000813 1787395148.548021 trunc from 0x000506 to 0x03c0d9 (0x3bbd4 bytes) 000814 1787395148.575363 mapwrite 0x02297f thru 0x02e826 (0xbea8 bytes) 000815 1787395148.635648 trunc from 0x03c0da to 0x00dba9 (0x2e530 bytes) 000816 1787395148.655283 mapread 0x009442 thru 0x00dba9 (0x4768 bytes) 000817 1787395148.675181 fallocate 0x0119c9 thru 0x01803a (0x6672 bytes) 000818 1787395148.683402 trunc from 0x00dbaa to 0x01ab00 (0xcf57 bytes) 000819 1787395148.711686 trunc from 0x00de45 to 0x01ab00 (0xccbc bytes) 000820 1787395148.719462 write 0x0312a1 thru 0x03417a (0x2eda bytes) 000821 1787395148.721988 write 0x002a37 thru 0x00d76b (0xad35 bytes) 000822 1787395148.730707 read 0x009671 thru 0x012287 (0x8c17 bytes) 000823 1787395148.748358 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3417b): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3417b): Mirror 6: 0x9acbe520 (stale) Mirror 8: 0x9acbe520 (stale) Mirror 11: 0x9acbe520 (stale) Mirror 13: 0x9acbe520 (stale) Mirror 16: 0x9acbe520 (stale) Mirror 17: 0x9acbe520 (stale) Mirror 18: 0x9acbe520 (stale) CRC-32 checksum value for chunk [0, 0x3417b): Mirror 1: 0x3be21149 Mirror 19: 0x3be21149 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3417b: skipped 000824 1787395150.139810 write 0x01b35e thru 0x01e14c (0x2def bytes) 000826 1787395150.213423 punch from 0x01e8f8 to 0x02752a (0x8c33 bytes) 000827 1787395150.229430 mapwrite 0x032081 thru 0x0395e6 (0x7566 bytes) 000828 1787395150.294600 punch from 0x036d79 to 0x0395e6 (0x286e bytes) 000829 1787395150.305342 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x395e7): Mirror 6: 0xfd389c77 (stale) Mirror 8: 0xfd389c77 (stale) Mirror 11: 0xfd389c77 (stale) Mirror 16: 0xfd389c77 (stale) Mirror 17: 0xfd389c77 (stale) Mirror 18: 0xfd389c77 (stale) Mirror 19: 0xbc6e7cd3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x395e7): Mirror 6: 0xfd389c77 (stale) Mirror 8: 0xfd389c77 (stale) Mirror 11: 0xfd389c77 (stale) Mirror 16: 0xfd389c77 (stale) Mirror 17: 0xfd389c77 (stale) Mirror 18: 0xfd389c77 (stale) Mirror 19: 0xbc6e7cd3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x395e7: skipped 000831 1787395150.966200 read 0x037add thru 0x0395e6 (0x1b0a bytes) 000832 1787395150.974380 fallocate 0x03681f thru 0x03ffff (0x97e1 bytes) 000833 1787395150.984499 fallocate 0x033825 thru 0x03c27b (0x8a57 bytes) 000834 1787395151.008086 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x395e7): Mirror 8: 0xfd389c77 (stale) Mirror 11: 0xfd389c77 (stale) Mirror 16: 0xfd389c77 (stale) Mirror 17: 0xfd389c77 (stale) Mirror 18: 0xfd389c77 (stale) Mirror 19: 0xbc6e7cd3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x395e7): Mirror 8: 0xfd389c77 (stale) Mirror 11: 0xfd389c77 (stale) Mirror 16: 0xfd389c77 (stale) Mirror 17: 0xfd389c77 (stale) Mirror 18: 0xfd389c77 (stale) Mirror 19: 0xbc6e7cd3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x395e7: skipped 000835 1787395151.796908 mapread 0x02c399 thru 0x0395e6 (0xd24e bytes) 000836 1787395151.827052 mapread 0x0211d9 thru 0x02ff6e (0xed96 bytes) 000837 1787395151.841003 mapread 0x027286 thru 0x02e8d6 (0x7651 bytes) 000838 1787395151.842175 mapwrite 0x00bab0 thru 0x01ad97 (0xf2e8 bytes) 000839 1787395151.938372 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x395e7): Mirror 8: 0xfd389c77 (stale) Mirror 11: 0xfd389c77 (stale) Mirror 17: 0xfd389c77 (stale) Mirror 18: 0xfd389c77 (stale) Mirror 19: 0xbc6e7cd3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x395e7): Mirror 8: 0xfd389c77 (stale) Mirror 11: 0xfd389c77 (stale) Mirror 17: 0xfd389c77 (stale) Mirror 18: 0xfd389c77 (stale) Mirror 19: 0xbc6e7cd3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x395e7: skipped 000840 1787395152.711694 read 0x014a81 thru 0x01a786 (0x5d06 bytes) 000841 1787395152.721105 punch from 0x02741e to 0x03232b (0xaf0e bytes) 000842 1787395152.730423 mapwrite 0x01f848 thru 0x029a26 (0xa1df bytes) 000843 1787395152.773274 fallocate 0x0345b3 thru 0x039582 (0x4fd0 bytes) 000845 1787395152.779316 write 0x03f391 thru 0x03ffff (0xc6f bytes) 000846 1787395152.788436 read 0x039b06 thru 0x03eba0 (0x509b bytes) 000848 1787395152.819285 fallocate 0x03ab53 thru 0x03ffff (0x54ad bytes) 000849 1787395152.838429 read 0x03baed thru 0x03ef30 (0x3444 bytes) 000850 1787395152.840516 mapwrite 0x028e30 thru 0x0338b2 (0xaa83 bytes) 000851 1787395152.901076 punch from 0x003ef8 to 0x007956 (0x3a5f bytes) 000852 1787395152.922496 trunc from 0x040000 to 0x006606 (0x399f9 bytes) 000853 1787395152.981022 mapwrite 0x032ff1 thru 0x034670 (0x1680 bytes) 000854 1787395153.049588 write 0x0234d5 thru 0x030af9 (0xd625 bytes) 000855 1787395153.051104 read 0x02e6f7 thru 0x033f7b (0x5885 bytes) 000856 1787395153.057182 mapread 0x009153 thru 0x00ccf3 (0x3ba1 bytes) 000857 1787395153.059038 trunc from 0x034671 to 0x032cc8 (0x19a8 bytes) 000859 1787395153.104705 mapread 0x00413e thru 0x004782 (0x645 bytes) 000860 1787395153.105712 write 0x032f46 thru 0x038c57 (0x5d12 bytes) 000861 1787395153.131898 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38c58): Mirror 8: 0x934f58b7 (stale) Mirror 11: 0x934f58b7 (stale) Mirror 18: 0x934f58b7 (stale) Mirror 19: 0x4af56c9f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38c58): Mirror 8: 0x934f58b7 (stale) Mirror 11: 0x934f58b7 (stale) Mirror 18: 0x934f58b7 (stale) Mirror 19: 0x4af56c9f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38c58: skipped 000862 1787395153.683891 trunc from 0x038c58 to 0x00b4c3 (0x2d794 bytes) 000863 1787395153.709057 punch from 0x009034 to 0x00b4c3 (0x2490 bytes) 000864 1787395153.715187 trunc from 0x00b4c4 to 0x01dfbd (0x12afa bytes) 000865 1787395153.733921 punch from 0x014007 to 0x01be0e (0x7e08 bytes) 000866 1787395153.739770 trunc from 0x01dfbe to 0x023f50 (0x5f93 bytes) 000867 1787395153.755146 write 0x000c85 thru 0x0077b0 (0x6b2c bytes) 000868 1787395153.766219 trunc from 0x023f51 to 0x0021ab (0x21da5 bytes) 000869 1787395153.786591 punch from 0x001e18 to 0x0021ab (0x394 bytes) 000870 1787395153.799376 trunc from 0x001c87 to 0x0021ab (0x525 bytes) 000871 1787395153.806761 punch from 0x001041 to 0x0021ab (0x116b bytes) 000872 1787395153.814171 write 0x03f4a0 thru 0x03ffff (0xb60 bytes) 000873 1787395153.824085 mapwrite 0x01f3b9 thru 0x0202e5 (0xf2d bytes) 000874 1787395153.850268 read 0x01ab04 thru 0x02019f (0x569c bytes) 000875 1787395153.859304 mapwrite 0x021b10 thru 0x02ea9d (0xcf8e bytes) 000876 1787395153.913600 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xd3cb19c1 (stale) Mirror 11: 0xd3cb19c1 (stale) Mirror 18: 0xd3cb19c1 (stale) Mirror 19: 0xbfc58599 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb3005668 Mirror 20: 0xb3005668 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000877 1787395155.156459 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xd3cb19c1 (stale) Mirror 11: 0xd3cb19c1 (stale) Mirror 18: 0xd3cb19c1 (stale) Mirror 19: 0xbfc58599 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb3005668 Mirror 20: 0xb3005668 Mirror 21: 0xb3005668 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000878 1787395156.435424 mapwrite 0x01c040 thru 0x029f88 (0xdf49 bytes) 000880 1787395156.610881 punch from 0x00cd65 to 0x00da3d (0xcd9 bytes) 000881 1787395156.618734 punch from 0x024a67 to 0x03307f (0xe619 bytes) 000883 1787395156.625627 fallocate 0x014368 thru 0x0185c6 (0x425f bytes) 000884 1787395156.631767 write 0x00a2d9 thru 0x01665d (0xc385 bytes) 000885 1787395156.642856 mapwrite 0x0200d1 thru 0x025907 (0x5837 bytes) 000886 1787395156.680261 write 0x03b8f0 thru 0x03ffff (0x4710 bytes) 000887 1787395156.693814 punch from 0x027709 to 0x0351f1 (0xdae9 bytes) 000888 1787395156.711735 punch from 0x02d8c0 to 0x034ca7 (0x73e8 bytes) 000889 1787395156.718752 write 0x016b75 thru 0x0265aa (0xfa36 bytes) 000891 1787395156.723708 mapread 0x02c572 thru 0x0308cd (0x435c bytes) 000892 1787395156.752180 mapwrite 0x0074d1 thru 0x00ca3c (0x556c bytes) 000893 1787395156.831311 mapwrite 0x00face thru 0x011590 (0x1ac3 bytes) 000894 1787395156.859696 mapread 0x01f105 thru 0x01f7df (0x6db bytes) 000895 1787395156.863376 write 0x019897 thru 0x024569 (0xacd3 bytes) 000896 1787395156.867432 trunc from 0x040000 to 0x01cb28 (0x234d7 bytes) 000897 1787395156.893552 punch from 0x019262 to 0x01cb28 (0x38c7 bytes) 000898 1787395156.911453 read 0x007b61 thru 0x016a5d (0xeefd bytes) 000899 1787395156.912398 punch from 0x007502 to 0x0142a0 (0xcd9f bytes) 000900 1787395156.926821 write 0x03ad25 thru 0x03ae1e (0xfa bytes) 000901 1787395156.936433 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ae1f): Mirror 8: 0x4f75a23 (stale) Mirror 11: 0x4f75a23 (stale) Mirror 19: 0x7a113023 (stale) Mirror 20: 0xe888870b (stale) Mirror 21: 0xe888870b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ae1f): Mirror 8: 0x4f75a23 (stale) Mirror 11: 0x4f75a23 (stale) Mirror 19: 0x7a113023 (stale) Mirror 20: 0xe888870b (stale) Mirror 21: 0xe888870b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ae1f: skipped 000902 1787395157.437234 trunc from 0x018719 to 0x0223d1 (0x9cb9 bytes) 000903 1787395157.447267 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3ae1f): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ae1f): Mirror 8: 0x4f75a23 (stale) Mirror 11: 0x4f75a23 (stale) Mirror 19: 0x7a113023 (stale) Mirror 20: 0xe888870b (stale) Mirror 21: 0xe888870b (stale) CRC-32 checksum value for chunk [0, 0x3ae1f): Mirror 1: 0x11c9b8fd Mirror 22: 0x11c9b8fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ae1f: skipped 000904 1787395159.166842 punch from 0x006fea to 0x00ae5e (0x3e75 bytes) 000905 1787395159.234815 mapread 0x01485b thru 0x01c32d (0x7ad3 bytes) 000906 1787395159.297335 punch from 0x00637a to 0x014755 (0xe3dc bytes) 000907 1787395159.310812 trunc from 0x03ae1f to 0x03e8a9 (0x3a8b bytes) 000908 1787395159.379301 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3e8aa): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e8aa): Mirror 8: 0x36dfbf6 (stale) Mirror 11: 0x36dfbf6 (stale) Mirror 19: 0x3674a424 (stale) Mirror 20: 0xea9177a7 (stale) Mirror 21: 0xea9177a7 (stale) Mirror 22: 0xb12b697c (stale) CRC-32 checksum value for chunk [0, 0x3e8aa): Mirror 1: 0x2d309e15 Mirror 23: 0x2d309e15 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e8aa: skipped 000909 1787395161.146496 punch from 0x0096b6 to 0x00ef3f (0x588a bytes) 000910 1787395161.261137 write 0x02bde7 thru 0x02f704 (0x391e bytes) 000911 1787395161.298411 trunc from 0x03e8aa to 0x025314 (0x19595 bytes) 000912 1787395161.323638 write 0x020673 thru 0x02ef03 (0xe891 bytes) 000913 1787395161.330052 read 0x003dff thru 0x00a3c7 (0x65c9 bytes) 000914 1787395161.339658 read 0x025d36 thru 0x02dfb2 (0x827d bytes) 000916 1787395161.341304 trunc from 0x02ef04 to 0x00c6c6 (0x2283d bytes) 000917 1787395161.357410 read 0x0083bb thru 0x00c6c6 (0x430c bytes) 000918 1787395161.372160 punch from 0x00b41d to 0x00c6c6 (0x12aa bytes) 000919 1787395161.396636 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0xc6c7): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc6c7): Mirror 8: 0x7a7387c6 (stale) Mirror 11: 0x7a7387c6 (stale) Mirror 19: 0x72538b02 (stale) Mirror 20: 0xa419b158 (stale) Mirror 21: 0xa419b158 (stale) Mirror 22: 0xdc50d6fb (stale) Mirror 23: 0xa419b158 (stale) CRC-32 checksum value for chunk [0, 0xc6c7): Mirror 1: 0xa419b158 Mirror 24: 0xa419b158 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc6c7: skipped 000921 1787395162.811623 fallocate 0x00a542 thru 0x015c9a (0xb759 bytes) 000922 1787395162.887830 write 0x006e09 thru 0x015be7 (0xeddf bytes) 000923 1787395162.907742 read 0x009752 thru 0x015bcf (0xc47e bytes) 000925 1787395162.911727 read 0x007b91 thru 0x009611 (0x1a81 bytes) 000926 1787395162.913805 trunc from 0x009b7e to 0x015be7 (0xc06a bytes) 000928 1787395162.942219 trunc from 0x015be8 to 0x02a9fd (0x14e16 bytes) 000929 1787395163.030999 mapwrite 0x03c7d9 thru 0x03ffff (0x3827 bytes) 000930 1787395163.141911 read 0x010a0d thru 0x02084b (0xfe3f bytes) 000931 1787395163.142857 fallocate 0x0275dc thru 0x027d62 (0x787 bytes) 000932 1787395163.159131 trunc from 0x040000 to 0x03ca5c (0x35a3 bytes) 000933 1787395163.230089 read 0x016eea thru 0x0223f9 (0xb510 bytes) 000934 1787395163.240844 trunc from 0x03ca5d to 0x00b4f0 (0x3156c bytes) 000935 1787395163.273049 fallocate 0x00bd0c thru 0x012961 (0x6c56 bytes) 000936 1787395163.305151 MIRROR_RESYNC 9 mirrors 000937 1787395166.181738 write 0x00a81e thru 0x018ce7 (0xe4ca bytes) 000938 1787395166.240741 punch from 0x000d52 to 0x00beb4 (0xb163 bytes) 000939 1787395166.278246 trunc from 0x018ce8 to 0x034730 (0x1ba49 bytes) 000940 1787395166.353061 punch from 0x01afb0 to 0x029866 (0xe8b7 bytes) 000941 1787395166.362121 trunc from 0x034731 to 0x012d02 (0x21a2e bytes) 000942 1787395166.397701 fallocate 0x025341 thru 0x032f05 (0xdbc5 bytes) 000943 1787395166.421525 MIRROR_RESYNC 9 mirrors 000944 1787395168.712149 mapread 0x018f31 thru 0x026609 (0xd6d9 bytes) 000945 1787395168.750671 fallocate 0x010101 thru 0x01b8ec (0xb7ec bytes) 000946 1787395168.808226 trunc from 0x032f06 to 0x0184f8 (0x1aa0d bytes) 000947 1787395168.844850 write 0x0343c8 thru 0x0385cc (0x4205 bytes) 000948 1787395168.846492 mapwrite 0x036592 thru 0x03ffff (0x9a6e bytes) 000950 1787395168.947859 mapwrite 0x032742 thru 0x03bd72 (0x9631 bytes) 000951 1787395168.981112 mapwrite 0x02cd81 thru 0x03545e (0x86de bytes) 000952 1787395169.037275 mapwrite 0x00cddf thru 0x00f974 (0x2b96 bytes) 000953 1787395169.079077 trunc from 0x040000 to 0x0367bf (0x9840 bytes) 000954 1787395169.101643 write 0x01bfb4 thru 0x01d8f4 (0x1941 bytes) 000955 1787395169.118284 read 0x020451 thru 0x02e5ea (0xe19a bytes) 000956 1787395169.127734 punch from 0x02ec37 to 0x0367bf (0x7b89 bytes) 000957 1787395169.134264 trunc from 0x010477 to 0x014bcf (0x4759 bytes) 000958 1787395169.141379 mapwrite 0x03ed70 thru 0x03ffff (0x1290 bytes) 000959 1787395169.183127 write 0x03ca04 thru 0x03f028 (0x2625 bytes) 000960 1787395169.185670 mapwrite 0x02faaf thru 0x03b195 (0xb6e7 bytes) 000961 1787395169.223184 punch from 0x00b3f1 to 0x014fb5 (0x9bc5 bytes) 000962 1787395169.229508 punch from 0x026a1f to 0x032d74 (0xc356 bytes) 000963 1787395169.250084 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x2944ba73 (stale) Mirror 11: 0x2944ba73 (stale) Mirror 19: 0x2944ba73 (stale) Mirror 20: 0x2944ba73 (stale) Mirror 21: 0x2944ba73 (stale) Mirror 23: 0x2944ba73 (stale) Mirror 24: 0x2944ba73 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x2944ba73 (stale) Mirror 11: 0x2944ba73 (stale) Mirror 19: 0x2944ba73 (stale) Mirror 20: 0x2944ba73 (stale) Mirror 21: 0x2944ba73 (stale) Mirror 23: 0x2944ba73 (stale) Mirror 24: 0x2944ba73 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000964 1787395170.037212 read 0x004de8 thru 0x014d4b (0xff64 bytes) 000965 1787395170.051752 fallocate 0x0078a1 thru 0x015e80 (0xe5e0 bytes) 000966 1787395170.057467 punch from 0x00f8d6 to 0x01d058 (0xd783 bytes) 000967 1787395170.076194 write 0x033a9d thru 0x036e01 (0x3365 bytes) 000968 1787395170.114034 punch from 0x02d35c to 0x030058 (0x2cfd bytes) 000969 1787395170.119846 trunc from 0x040000 to 0x001c7d (0x3e382 bytes) 000970 1787395170.148605 mapread 0x000067 thru 0x001c7d (0x1c17 bytes) 000971 1787395170.160216 read 0x001aa9 thru 0x001c7d (0x1d5 bytes) 000972 1787395170.161180 trunc from 0x0001c2 to 0x001c7d (0x1abc bytes) 000973 1787395170.167743 fallocate 0x03353a thru 0x03cc71 (0x9738 bytes) 000974 1787395170.173205 write 0x035b65 thru 0x03ffff (0xa49b bytes) 000975 1787395170.176503 write 0x0042f8 thru 0x013130 (0xee39 bytes) 000976 1787395170.186887 punch from 0x000f78 to 0x00394c (0x29d5 bytes) 000977 1787395170.191767 trunc from 0x040000 to 0x03147d (0xeb82 bytes) 000978 1787395170.245232 mapwrite 0x024f4b thru 0x02e1a8 (0x925e bytes) 000979 1787395170.306997 trunc from 0x03147e to 0x0325cc (0x114f bytes) 000980 1787395170.331347 read 0x012f49 thru 0x022b3b (0xfbf3 bytes) 000981 1787395170.332240 trunc from 0x005b46 to 0x013a09 (0xdec4 bytes) 000984 1787395170.343573 trunc from 0x0325cd to 0x0038a5 (0x2ed27 bytes) 000985 1787395170.378580 punch from 0x000628 to 0x0038a5 (0x327e bytes) 000986 1787395170.385147 write 0x03d35a thru 0x03ffff (0x2ca6 bytes) 000987 1787395170.388305 mapread 0x00c136 thru 0x01896e (0xc839 bytes) 000988 1787395170.404191 mapwrite 0x014b16 thru 0x0153c1 (0x8ac bytes) 000989 1787395170.432456 mapwrite 0x01a9aa thru 0x0257ad (0xae04 bytes) 000990 1787395170.487347 trunc from 0x040000 to 0x009c06 (0x363f9 bytes) 000991 1787395170.538193 MIRROR_RESYNC 8 mirrors 000992 1787395171.957785 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a2:0x0]: Invalid argument (22) 000996 1787395172.490250 trunc from 0x009c07 to 0x03a70e (0x30b08 bytes) 000997 1787395172.555750 write 0x007f53 thru 0x00d12a (0x51d8 bytes) 000998 1787395172.571279 trunc from 0x03a70f to 0x00d0b7 (0x2d657 bytes) 000999 1787395172.604424 punch from 0x007b9d to 0x007c6e (0xd2 bytes) 001000 1787395172.644493 mapread 0x002b79 thru 0x003d88 (0x1210 bytes) All operations completed A-OK! PASS 70a (133s) == sanity-flr test 71: check mirror extend parent fid ==== 06:39:39 (1787395179) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1db:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (12s) == sanity-flr test 72: check append on FLR file ========== 06:39:51 (1787395191) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.342102 s, 6.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xed:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156724 s, 6.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xed:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xed:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xee:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 06:40:01 (1787395201) Starting client: oleg620-client.virtnet: -o user_xattr,flock 192.168.206.120@tcp:/lustre /mnt/lustre2 Starting client: oleg620-client.virtnet: -o user_xattr,flock 192.168.206.120@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7110336 .. Extending file size to 1244704 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1881216 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5874880 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8580288 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9146528 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8922304 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 61105 61106 61108 61110 61111 192.168.206.120@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg620-client.virtnet /mnt/lustre2 (opts:) 192.168.206.120@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg620-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (86s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 06:41:27 (1787395287) Starting client: oleg620-client.virtnet: -o user_xattr,flock 192.168.206.120@tcp:/lustre /mnt/lustre2 Starting client: oleg620-client.virtnet: -o user_xattr,flock 192.168.206.120@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3576288 .. Extending file size to 2253024 .. failed Extending file size to 8718144 .. Extending file size to 2461568 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 6198784 .. Extending file size to 9112000 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8865984 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9151264 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62659 62660 62661 62663 62665 done 192.168.206.120@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg620-client.virtnet /mnt/lustre2 (opts:) 192.168.206.120@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg620-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (83s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 06:42:50 (1787395370) Starting client: oleg620-client.virtnet: -o user_xattr,flock 192.168.206.120@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2272 1269032 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17732 3589288 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25636 3581384 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43368 7170672 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2272 1269032 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17732 3589288 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25636 3581384 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43368 7170672 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.148489 s, 14.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.232343 s, 9.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x9ecf0d19 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.120@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg620-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 06:43:13 (1787395393) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.155335 s, 13.5 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 06:43:22 (1787395402) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3a:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x121:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x121:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x122:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x103:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 06:43:31 (1787395411) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.097814 s, 21.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x228:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x228:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x229:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 06:43:40 (1787395420) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.101007 s, 20.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x123:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x124:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x104:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x123:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x124:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x104:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x105:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 06:43:49 (1787395429) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.164314 s, 12.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x229:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x229:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22b:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 06:44:24 (1787395464) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg620-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111474 s, 9.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x22a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x22a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x22b:0x0] } PASS 204d (54s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 06:45:18 (1787395518) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg620-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x125:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x126:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.150566 s, 13.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x125:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x126:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x106:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x125:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x126:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x106:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (56s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 06:46:14 (1787395574) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg620-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg620-server: Pool lustre.test_204f created Adding targets to pool oleg620-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg620-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.150226 s, 14.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x127:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x107:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x127:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x107:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg620-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg620-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg620-server: Pool lustre.test_204f destroyed PASS 204f (75s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 06:47:30 (1787395650) PASS 205a (11s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 06:47:40 (1787395660) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 06:47:49 (1787395669) oleg620-server: Pool lustre.test_206 created Adding targets to pool oleg620-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg620-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg620-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg620-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg620-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 06:48:18 (1787395698) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128252 s, 8.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706209 s, 14.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xac73576a Mirror 3: 0xac73576a PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 06:48:29 (1787395709) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 06:48:33 (1787395713) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 06:48:37 (1787395717) Starting client: oleg620-client.virtnet: -o user_xattr,flock 192.168.206.120@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0768197 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1e20000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x22a0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x22a0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x22a0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1cd0000 Read this (4 bytes): BBBB 192.168.206.120@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg620-client.virtnet /mnt/lustre2 (opts:) PASS 209a (40s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 06:49:18 (1787395758) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0269282 s, 304 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 06:49:25 (1787395765) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901004 s, 11.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x59 lmm_fid: [0x200000405:0x59:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x59 lmm_fid: [0x200000405:0x59:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611225 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-22 06:49:27.000000000 -0400 Modify: 2026-08-22 06:49:27.000000000 -0400 Change: 2026-08-22 06:49:29.000000000 -0400 Birth: 2026-08-22 06:49:28.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 06:49:34 (1787395774) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681524 s, 15.4 MB/s oleg620-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg620-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg620-server: O/280000401/d18/132: File not found by ext2_lookup PASS 210b (24s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 06:49:58 (1787395798) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.249591 s, 164 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0500643 s, 81.8 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 22 06:50 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5e:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 22 06:50 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 06:50:08 (1787395808) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (10s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 06:50:17 (1787395817) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1929 sec =========== 06:50:26 (1787395826) === sanity-flr: start cleanup 06:50:28 (1787395828) === === sanity-flr: finish cleanup 06:50:36 (1787395836) ===