-----============= acceptance-small: sanity-flr ============----- Thu Jul 30 02:27:39 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg430-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 02:28:12 (1785392892) === oleg430-client.virtnet: executing check_config_client /mnt/lustre oleg430-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg430-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff978e120ef000.idle_timeout=debug osc.lustre-OST0001-osc-ffff978e120ef000.idle_timeout=debug disable quota as required oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 02:29:01 (1785392941) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8255] == sanity-flr test 0a: lfs mirror create with -N option == 02:29:05 (1785392945) PASS 0a (27s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 02:29:32 (1785392972) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (5s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 02:29:37 (1785392977) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 02:29:41 (1785392981) PASS 0d (36s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 02:30:18 (1785393018) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (5s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 02:30:24 (1785393024) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (5s) == sanity-flr test 0g: lfs mirror create flags support === 02:30:29 (1785393029) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff978e120ef000.stats=clear osc.lustre-OST0001-osc-ffff978e120ef000.stats=clear PASS 0g (11s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 02:30:40 (1785393040) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 02:30:53 (1785393053) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 02:31:02 (1785393062) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 02:31:06 (1785393066) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 02:31:16 (1785393076) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 02:31:27 (1785393087) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 02:31:36 (1785393096) PASS 3 (11s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 02:31:47 (1785393107) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (10s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 02:31:57 (1785393117) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (9s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 02:32:12 (1785393132) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 02:32:20 (1785393140) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.305183 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883516 s, 11.9 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 02:32:29 (1785393149) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.391267 s, 21.4 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 02:32:38 (1785393158) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 02:32:50 (1785393170) Stopping /mnt/lustre-ost1 (opts:) on oleg430-server oleg430-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff978e120ef000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff978e120ef000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg430-server: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg430-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg430-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg430-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg430-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg430-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg430-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg430-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg430-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg430-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg430-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg430-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg430-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg430-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg430-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg430-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg430-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg430-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg430-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg430-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg430-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg430-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg430-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg430-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg430-server oleg430-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff978e120ef000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff978e120ef000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg430-server: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg430-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (159s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 02:35:30 (1785393330) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.861557 s, 13.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg430-server oleg430-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff978e120ef000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff978e120ef000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg430-server: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg430-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (61s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 02:36:31 (1785393391) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg430-server oleg430-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff978e120ef000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff978e120ef000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg430-server: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg430-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg430-server oleg430-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff978e120ef000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff978e120ef000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg430-server: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg430-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (111s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 02:38:23 (1785393503) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.88276 s, 13.5 MB/s -rw-r--r-- 1 root root 52428800 Jul 30 02:38 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.552s user 0m0.000s sys 0m0.189s Stopping /mnt/lustre-ost1 (opts:) on oleg430-server oleg430-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff978e120ef000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff978e120ef000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m2.416s user 0m0.001s sys 0m0.176s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg430-server: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg430-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg430-server oleg430-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff978e120ef000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff978e120ef000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.541s user 0m0.001s sys 0m0.147s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg430-server: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg430-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg430-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff978e120ef000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff978e120ef000.ost_server_uuid in FULL state after 0 sec PASS 33b (133s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 02:40:35 (1785393635) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 02:40:40 (1785393640) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 02:40:44 (1785393644) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 02:40:48 (1785393648) PASS 35 (10s) == sanity-flr test 36a: write to mirrored files ========== 02:40:58 (1785393658) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.500131 s, 25.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 02:41:11 (1785393671) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456117 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631762 s, 16.6 MB/s PASS 36b (10s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 02:41:21 (1785393681) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 02:41:30 (1785393690) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00229203 s, 447 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000832531 s, 1.2 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000733549 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00197737 s, 518 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 02:41:42 (1785393702) create test files with size 15736157 .. 63ae47755e85b6104c317c92ab3556c4 - b2e19e069de5704945408918a3057410 - 5475c7d9f81e2eec6b5290d572e380d7 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg430-client.virtnet /mnt/lustre (opts:) Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre PASS 37 (30s) == sanity-flr test 38: resync ============================ 02:42:12 (1785393732) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 02:42:29 (1785393749) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 02:42:37 (1785393757) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889105 s, 11.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xca:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367737 s, 28.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaa:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (12s) == sanity-flr test 41: lfs mirror resync check =========== 02:42:49 (1785393769) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.128814 s, 16.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.524879 s, 8.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.206826 s, 10.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (23s) == sanity-flr test 42: lfs mirror verify ================= 02:43:13 (1785393793) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (5s) == sanity-flr test 43a: mirror pick on write ============= 02:43:17 (1785393797) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 02:43:22 (1785393802) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055683 s, 18.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (11s) == sanity-flr test 44a: lfs mirror split check =========== 02:43:32 (1785393812) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.391082 s, 8.0 MB/s PASS 44a (12s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 02:43:44 (1785393824) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd5:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb4:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb4:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 02:43:55 (1785393835) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.541994 s, 19.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg430-client.virtnet /mnt/lustre (opts:) Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (17s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 02:44:12 (1785393852) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff978e12379000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.363416 s, 28.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg430-client.virtnet /mnt/lustre (opts:) Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff978e08065000.xattr_cache=1 PASS 44e (21s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 02:44:33 (1785393873) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (61s) == sanity-flr test 46: Verify setstripe --copy option ==== 02:45:34 (1785393934) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 02:46:15 (1785393975) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (5s) == sanity-flr test 48: Verify snapshot mirror ============ 02:46:20 (1785393980) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.312376 s, 10.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.24791 s, 12.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x73494a2b (nosync,stale) Mirror 2: 0x73494a2b (nosync,stale) ** verify mirror 2 doesn't change original checksum: 2125ca19f6e162c597f736c5edd1c4fa - mirror 1 checksum: 15ea52dd886d83dbe42a38f253ffe096 - mirror 2 checksum: 2125ca19f6e162c597f736c5edd1c4fa - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-30T02:46:23-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-30T02:46:23-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc0:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 2125ca19f6e162c597f736c5edd1c4fa - mirror 1 checksum: 15ea52dd886d83dbe42a38f253ffe096 - mirror 2 checksum: 15ea52dd886d83dbe42a38f253ffe096 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-30T02:46:23-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-30T02:46:23-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc0:0x0] } PASS 48 (13s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 02:46:33 (1785393993) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546894 s, 19.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (9s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 02:46:42 (1785394002) Failing mds1 on oleg430-server Stopping /mnt/lustre-mds1 (opts:) on oleg430-server 02:46:56 (1785394016) shut down facet: mds1 facet_host: oleg430-server facet_failover_host: oleg430-server Failover mds1 to oleg430-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-MDT0000 02:47:36 (1785394056) targets are mounted 02:47:36 (1785394056) facet_failover done oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643439 s, 16.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } PASS 50A (74s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 02:47:57 (1785394077) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0667977 s, 307 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0349623 s, 586 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (12s) == sanity-flr test 50b: mirror rsync handles sparseness == 02:48:09 (1785394089) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0922841 s, 222 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0214989 s, 953 kB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (16s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 02:48:25 (1785394105) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00529585 s, 3.1 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 02:48:35 (1785394115) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0750522 s, 273 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (10s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 02:48:45 (1785394125) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00985591 s, 416 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0387034 s, 106 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 02:48:55 (1785394135) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (26s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 02:49:20 (1785394160) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (34s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 02:49:54 (1785394194) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (29s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 02:50:23 (1785394223) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 02:50:49 (1785394249) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 02:50:58 (1785394258) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57650 mirror split pid 57651 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5070: 57650 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5070: 57651 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (70s) == sanity-flr test 70a: flr mode fsx test ================ 02:52:08 (1785394328) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 4331 truncating to largest ever: 0x120cb 000001 1785394331.407581 trunc from 00000000 to 0x0120ca (0x120cb bytes) 000003 1785394331.456601 mapwrite 0x01bf36 thru 0x024896 (0x8961 bytes) 000004 1785394331.614622 trunc from 0x017e5b to 0x0211bf (0x9365 bytes) truncating to largest ever: 0x3e1ac 000005 1785394331.623074 trunc from 0x024897 to 0x03e1ab (0x19915 bytes) 000006 1785394331.659861 read 0x01f8fc thru 0x02e1a6 (0xe8ab bytes) 000007 1785394331.669044 read 0x02ceb6 thru 0x0325c5 (0x5710 bytes) 000008 1785394331.686101 trunc from 0x021ba7 to 0x024ec9 (0x3323 bytes) 000010 1785394331.706338 punch from 0x031c4f to 0x03e1ab (0xc55d bytes) 000012 1785394331.712351 trunc from 0x00bf4f to 0x00d626 (0x16d8 bytes) 000013 1785394331.726364 trunc from 0x000960 to 0x0063a7 (0x5a48 bytes) 000014 1785394331.744271 trunc from 0x03e1ac to 0x00c78a (0x31a21 bytes) 000015 1785394331.766067 write 0x00ec4e thru 0x01621e (0x75d1 bytes) 000016 1785394331.767494 fallocate 0x0061bc thru 0x00d793 (0x75d8 bytes) 000017 1785394331.771910 trunc from 0x00611d to 0x00f14b (0x902f bytes) 000018 1785394331.784276 write 0x015c17 thru 0x025a93 (0xfe7d bytes) 000021 1785394331.792081 trunc from 0x025a94 to 0x00d423 (0x18670 bytes) 000022 1785394331.814303 trunc from 0x002c0f to 0x007b70 (0x4f62 bytes) 000023 1785394331.823004 mapwrite 0x01046d thru 0x01b39d (0xaf31 bytes) 000024 1785394331.933705 write 0x03b1be thru 0x03ffff (0x4e42 bytes) 000025 1785394331.936318 punch from 0x01da88 to 0x022641 (0x4bba bytes) 000026 1785394331.943322 write 0x024320 thru 0x028ed8 (0x4bb9 bytes) 000027 1785394331.953694 read 0x0134ef thru 0x01e868 (0xb37a bytes) 000028 1785394331.963693 trunc from 0x040000 to 0x0340a2 (0xbf5d bytes) 000029 1785394331.978607 trunc from 0x0340a3 to 0x02b624 (0x8a7e bytes) 000030 1785394332.012591 trunc from 0x02b625 to 0x03b528 (0xff04 bytes) 000031 1785394332.055256 punch from 0x007e4c to 0x015586 (0xd73b bytes) 000032 1785394332.059533 punch from 0x039c94 to 0x03b528 (0x1895 bytes) 000033 1785394332.064248 mapread 0x003f7e thru 0x007008 (0x308b bytes) 000034 1785394332.075441 mapread 0x027622 thru 0x02d42b (0x5e0a bytes) 000035 1785394332.096466 MIRROR_EXTEND to 2 mirrors 000036 1785394332.768423 MIRROR_RESYNC 2 mirrors 000037 1785394333.067418 trunc from 0x03b529 to 0x01ca23 (0x1eb05 bytes) 000038 1785394333.150296 fallocate 0x025461 thru 0x0343f5 (0xef95 bytes) 000039 1785394333.157074 trunc from 0x001c1a to 0x01170c (0xfaf3 bytes) 000040 1785394333.163620 trunc from 0x011048 to 0x011f37 (0xef0 bytes) 000041 1785394333.170882 write 0x00ab49 thru 0x00d0b1 (0x2569 bytes) 000043 1785394333.190917 mapread 0x00cbee thru 0x01b02d (0xe440 bytes) 000044 1785394333.237419 punch from 0x018373 to 0x01ca23 (0x46b1 bytes) 000045 1785394333.244626 mapread 0x01c1f2 thru 0x01ca23 (0x832 bytes) 000046 1785394333.255520 read 0x0116ae thru 0x01ca23 (0xb376 bytes) 000047 1785394333.257503 fallocate 0x012ea3 thru 0x0139b8 (0xb16 bytes) 000048 1785394333.262593 punch from 0x006332 to 0x0071a1 (0xe70 bytes) 000049 1785394333.279252 trunc from 0x013d80 to 0x01ca23 (0x8ca4 bytes) 000050 1785394333.303189 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1ca24): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ca24): Mirror 2: 0x652a9879 (stale) CRC-32 checksum value for chunk [0, 0x1ca24): Mirror 1: 0xd9ba7182 Mirror 3: 0xd9ba7182 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ca24: skipped 000051 1785394334.281428 read 0x015494 thru 0x01ca23 (0x7590 bytes) 000054 1785394334.291215 mapwrite 0x03b108 thru 0x03ca19 (0x1912 bytes) 000055 1785394334.354652 read 0x005e81 thru 0x00628b (0x40b bytes) 000056 1785394334.363949 mapread 0x017f1d thru 0x0183be (0x4a2 bytes) 000058 1785394334.366046 write 0x00150f thru 0x008fb1 (0x7aa3 bytes) 000059 1785394334.367981 read 0x01cd34 thru 0x027597 (0xa864 bytes) 000061 1785394334.369389 punch from 0x00213f to 0x007c78 (0x5b3a bytes) 000062 1785394334.381981 punch from 0x00bd78 to 0x012a70 (0x6cf9 bytes) 000063 1785394334.386544 trunc from 0x03ca1a to 0x009ae3 (0x32f36 bytes) 000064 1785394334.402462 mapwrite 0x004497 thru 0x00bcaa (0x7814 bytes) 000065 1785394334.441285 read 0x00b2e4 thru 0x00bcaa (0x9c7 bytes) 000066 1785394334.442235 write 0x00e549 thru 0x01085e (0x2316 bytes) 000067 1785394334.444202 write 0x01fbeb thru 0x02f87d (0xfc93 bytes) 000068 1785394334.446437 trunc from 0x02f87e to 0x0134c1 (0x1c3bc bytes) 000069 1785394334.459270 read 0x00d741 thru 0x00dacb (0x38b bytes) 000070 1785394334.467464 trunc from 0x0134c2 to 0x01854d (0x508c bytes) 000071 1785394334.482448 punch from 0x014f43 to 0x01854d (0x360b bytes) 000072 1785394334.486409 write 0x03c0c8 thru 0x03ffff (0x3f38 bytes) 000073 1785394334.491590 fallocate 0x007243 thru 0x01487a (0xd638 bytes) 000074 1785394334.500861 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4c7dde6d (stale) Mirror 3: 0xebcdbbec (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x39ee18b2 Mirror 4: 0x39ee18b2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000075 1785394335.363501 mapwrite 0x00d1af thru 0x012804 (0x5656 bytes) 000076 1785394335.549749 trunc from 0x040000 to 0x008bc1 (0x3743e bytes) 000077 1785394335.675973 MIRROR_RESYNC 4 mirrors 000078 1785394337.159519 mapread 0x0068de thru 0x008bc1 (0x22e4 bytes) 000079 1785394337.240972 trunc from 0x0029bc to 0x0037de (0xe23 bytes) 000080 1785394337.288456 punch from 0x000f9a to 0x008bc1 (0x7c28 bytes) 000081 1785394337.307390 read 0x00423c thru 0x00648e (0x2253 bytes) 000082 1785394337.322446 punch from 0x005614 to 0x008bc1 (0x35ae bytes) 000083 1785394337.344084 punch from 0x003e8e to 0x008bc1 (0x4d34 bytes) 000084 1785394337.350463 mapwrite 0x027700 thru 0x0346c8 (0xcfc9 bytes) 000086 1785394337.475559 trunc from 0x00c49a to 0x01612b (0x9c92 bytes) 000087 1785394337.490444 trunc from 0x0346c9 to 0x03328b (0x143d bytes) 000088 1785394337.534143 read 0x002834 thru 0x010da6 (0xe573 bytes) 000089 1785394337.554841 mapread 0x019a32 thru 0x028894 (0xee63 bytes) 000090 1785394337.556552 read 0x006edb thru 0x00864a (0x1770 bytes) 000091 1785394337.557134 trunc from 0x03328c to 0x03bd47 (0x8abc bytes) 000092 1785394337.588136 punch from 0x035446 to 0x0385ab (0x3166 bytes) 000093 1785394337.602469 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bd48): Mirror 2: 0xd4420477 (stale) Mirror 3: 0xd4420477 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd48): Mirror 2: 0xd4420477 (stale) Mirror 3: 0xd4420477 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd48: skipped 000094 1785394337.845203 fallocate 0x002f1b thru 0x0108d0 (0xd9b6 bytes) 000095 1785394337.849433 punch from 0x02764d to 0x030143 (0x8af7 bytes) 000096 1785394337.856429 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bd48): Mirror 3: 0xd4420477 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd48): Mirror 3: 0xd4420477 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd48: skipped 000097 1785394338.164442 MIRROR_RESYNC 2 mirrors 000098 1785394338.994500 write 0x007399 thru 0x011e36 (0xaa9e bytes) 000099 1785394339.034679 trunc from 0x03645b to 0x039270 (0x2e16 bytes) 000100 1785394339.039774 mapread 0x015c69 thru 0x01fd96 (0xa12e bytes) 000101 1785394339.055880 punch from 0x038930 to 0x038b70 (0x241 bytes) 000102 1785394339.060338 write 0x03b9c2 thru 0x03ffff (0x463e bytes) 000103 1785394339.064744 trunc from 0x040000 to 0x00e879 (0x31786 bytes) 000104 1785394339.089479 write 0x0176f8 thru 0x01eda0 (0x76a9 bytes) 000105 1785394339.092683 fallocate 0x0132f8 thru 0x015500 (0x2209 bytes) 000106 1785394339.099546 mapwrite 0x019a82 thru 0x019c5f (0x1de bytes) fallocating to largest ever: 0x40000 000107 1785394339.116445 fallocate 0x03c9ce thru 0x03ffff (0x3632 bytes) 000108 1785394339.126826 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1eda1): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1eda1): Mirror 3: 0x33681332 (stale) CRC-32 checksum value for chunk [0, 0x1eda1): Mirror 1: 0xa4045e59 Mirror 4: 0xa4045e59 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1eda1: skipped 000109 1785394339.833551 mapwrite 0x02c028 thru 0x037f9a (0xbf73 bytes) 000110 1785394340.094459 mapwrite 0x0168a7 thru 0x02642b (0xfb85 bytes) 000111 1785394340.198402 trunc from 0x02bdb7 to 0x02d02c (0x1276 bytes) 000112 1785394340.208561 punch from 0x00e9d8 to 0x010ace (0x20f7 bytes) 000114 1785394340.237871 fallocate 0x00c4f5 thru 0x011909 (0x5415 bytes) 000115 1785394340.265404 write 0x0126ae thru 0x01a7a1 (0x80f4 bytes) 000116 1785394340.274436 read 0x01f2f1 thru 0x02ecdc (0xf9ec bytes) 000117 1785394340.291426 fallocate 0x0294e0 thru 0x037b9d (0xe6be bytes) 000118 1785394340.301372 mapwrite 0x0207b9 thru 0x028093 (0x78db bytes) 000119 1785394340.359225 trunc from 0x037f9b to 0x027bef (0x103ab bytes) 000120 1785394340.448972 punch from 0x025737 to 0x027bef (0x24b9 bytes) 000121 1785394340.455995 trunc from 0x027bf0 to 0x01496f (0x13280 bytes) 000122 1785394340.476630 mapread 0x0090dd thru 0x00cca5 (0x3bc9 bytes) 000124 1785394340.519575 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14970): Mirror 4: 0x77c0740d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14970): Mirror 4: 0x77c0740d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14970: skipped 000125 1785394340.796423 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x14970): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14970): Mirror 4: 0x77c0740d (stale) CRC-32 checksum value for chunk [0, 0x14970): Mirror 1: 0x49bc4bfe Mirror 5: 0x49bc4bfe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14970: skipped 000126 1785394341.728994 write 0x0078f7 thru 0x008a21 (0x112b bytes) 000127 1785394341.777546 fallocate 0x020cfb thru 0x02d38d (0xc693 bytes) 000129 1785394341.785520 fallocate 0x006c8e thru 0x00ff49 (0x92bc bytes) 000130 1785394341.790883 mapwrite 0x0109bb thru 0x014978 (0x3fbe bytes) 000131 1785394341.842010 read 0x00058a thru 0x00402f (0x3aa6 bytes) 000132 1785394341.851453 read 0x001d66 thru 0x00414c (0x23e7 bytes) 000133 1785394341.856485 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14979): Mirror 4: 0xe566ce61 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14979): Mirror 4: 0xe566ce61 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14979: skipped 000134 1785394342.193626 fallocate 0x014883 thru 0x024001 (0xf77f bytes) 000135 1785394342.211287 read 0x00e638 thru 0x01ab7c (0xc545 bytes) 000136 1785394342.268322 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000137 1785394342.564745 trunc from 0x01aa5f to 0x024001 (0x95a3 bytes) 000138 1785394342.577966 trunc from 0x00b692 to 0x018b39 (0xd4a8 bytes) 000140 1785394342.584973 trunc from 0x024002 to 0x0296d8 (0x56d7 bytes) 000141 1785394342.656232 trunc from 0x005815 to 0x012635 (0xce21 bytes) 000143 1785394342.663902 read 0x00a063 thru 0x00fd9a (0x5d38 bytes) 000144 1785394342.693291 trunc from 0x0296d9 to 0x01448a (0x1524e bytes) 000145 1785394342.721694 trunc from 0x01448b to 0x006612 (0xde78 bytes) 000146 1785394342.737230 fallocate 0x00b2ed thru 0x00f895 (0x45a9 bytes) 000147 1785394342.755404 punch from 0x0044b0 to 0x00d84e (0x939f bytes) 000148 1785394342.761070 write 0x00ab27 thru 0x01419b (0x9675 bytes) 000151 1785394342.779909 write 0x0089a4 thru 0x01036d (0x79ca bytes) 000152 1785394342.791569 trunc from 0x00fbb5 to 0x0117d2 (0x1c1e bytes) 000153 1785394342.818949 trunc from 0x01419c to 0x02f9a7 (0x1b80c bytes) 000154 1785394342.855847 write 0x035537 thru 0x0388ad (0x3377 bytes) 000155 1785394342.862360 mapwrite 0x01145a thru 0x016c68 (0x580f bytes) 000156 1785394342.953469 trunc from 0x0388ae to 0x00112e (0x3777f bytes) 000157 1785394342.987167 write 0x026e19 thru 0x02b571 (0x4759 bytes) 000158 1785394342.993616 punch from 0x00b2c5 to 0x00fa3b (0x4777 bytes) 000159 1785394343.005856 mapwrite 0x0054ca thru 0x00d867 (0x839e bytes) 000160 1785394343.046311 mapread 0x025e95 thru 0x02b571 (0x56dd bytes) 000161 1785394343.061389 trunc from 0x02b572 to 0x028c5c (0x2915 bytes) 000162 1785394343.082327 mapread 0x01065f thru 0x01683f (0x61e1 bytes) 000163 1785394343.095477 trunc from 0x00bd5a to 0x014481 (0x8728 bytes) 000165 1785394343.105097 read 0x016b64 thru 0x020d56 (0xa1f3 bytes) 000167 1785394343.124356 write 0x036c03 thru 0x03ffff (0x93fd bytes) 000168 1785394343.130391 mapwrite 0x01a9ad thru 0x02155d (0x6bb1 bytes) 000169 1785394343.171574 mapread 0x00e72c thru 0x0133e9 (0x4cbe bytes) 000171 1785394343.178023 mapwrite 0x004fd4 thru 0x006965 (0x1992 bytes) 000172 1785394343.206924 trunc from 0x02997a to 0x02d5c8 (0x3c4f bytes) 000173 1785394343.220701 trunc from 0x040000 to 0x0238c4 (0x1c73b bytes) 000174 1785394343.242027 write 0x037925 thru 0x03d223 (0x58ff bytes) 000175 1785394343.251540 MIRROR_EXTEND to 2 mirrors 000177 1785394344.231899 mapwrite 0x001670 thru 0x01039c (0xed2d bytes) 000178 1785394344.405829 trunc from 0x00409c to 0x01008d (0xbff2 bytes) 000179 1785394344.418118 mapread 0x029de5 thru 0x02a607 (0x823 bytes) 000180 1785394344.431523 fallocate 0x025702 thru 0x02b853 (0x6152 bytes) 000182 1785394344.447879 write 0x00f325 thru 0x014bb1 (0x588d bytes) 000183 1785394344.479306 punch from 0x01c9b0 to 0x02ba21 (0xf072 bytes) 000184 1785394344.495549 mapread 0x0057d4 thru 0x00be3e (0x666b bytes) 000185 1785394344.510873 read 0x027188 thru 0x02c555 (0x53ce bytes) 000187 1785394344.522374 write 0x03914c thru 0x039dc4 (0xc79 bytes) 000188 1785394344.526697 write 0x03b179 thru 0x03ea09 (0x3891 bytes) 000190 1785394344.532939 trunc from 0x0293b2 to 0x03089f (0x74ee bytes) 000191 1785394344.537794 trunc from 0x03ea0a to 0x033bf8 (0xae11 bytes) 000192 1785394344.567688 write 0x02367d thru 0x0316ff (0xe083 bytes) 000193 1785394344.575816 trunc from 0x033bf9 to 0x013183 (0x20a75 bytes) 000195 1785394344.591799 mapwrite 0x03ae6c thru 0x03f8e3 (0x4a78 bytes) 000196 1785394344.631471 fallocate 0x00fb7d thru 0x010084 (0x508 bytes) 000197 1785394344.636405 mapread 0x029b7a thru 0x0390a2 (0xf529 bytes) 000199 1785394344.667027 trunc from 0x0224bb to 0x025d2f (0x3875 bytes) 000200 1785394344.673828 trunc from 0x006b1e to 0x00a3db (0x38be bytes) 000201 1785394344.681237 read 0x03e62e thru 0x03f8e3 (0x12b6 bytes) 000204 1785394344.685311 read 0x00f5da thru 0x01aed5 (0xb8fc bytes) 000205 1785394344.695186 trunc from 0x020c39 to 0x02d567 (0xc92f bytes) 000206 1785394344.699722 trunc from 0x03f8e4 to 0x0007c7 (0x3f11c bytes) 000207 1785394344.729094 punch from 0x0003b6 to 0x0007c7 (0x412 bytes) 000208 1785394344.737103 mapwrite 0x01804e thru 0x01ed68 (0x6d1b bytes) 000209 1785394344.868331 punch from 0x01a38f to 0x01d846 (0x34b8 bytes) 000210 1785394344.873815 punch from 0x00a1b9 to 0x00ff13 (0x5d5b bytes) 000211 1785394344.878410 fallocate 0x00d95e thru 0x0168e6 (0x8f89 bytes) 000212 1785394344.882102 mapread 0x00c5f2 thru 0x01a916 (0xe325 bytes) 000213 1785394344.978939 punch from 0x014a7e to 0x01ed68 (0xa2eb bytes) 000214 1785394345.000031 trunc from 0x00dfc9 to 0x0163a9 (0x83e1 bytes) 000215 1785394345.004809 mapread 0x002c5b thru 0x004e1c (0x21c2 bytes) 000216 1785394345.034751 trunc from 0x0065a9 to 0x0158a1 (0xf2f9 bytes) 000217 1785394345.041446 trunc from 0x01ed69 to 0x039d7b (0x1b013 bytes) 000218 1785394345.053418 write 0x02e99f thru 0x03d31e (0xe980 bytes) 000219 1785394345.068856 mapread 0x000166 thru 0x007700 (0x759b bytes) 000221 1785394345.091254 punch from 0x00c8b1 to 0x01bc6f (0xf3bf bytes) 000222 1785394345.099407 trunc from 0x03242e to 0x03d31e (0xaef1 bytes) 000223 1785394345.114272 trunc from 0x03b787 to 0x03d31e (0x1b98 bytes) 000224 1785394345.118072 mapread 0x001a8e thru 0x00801d (0x6590 bytes) 000225 1785394345.122887 mapwrite 0x02055e thru 0x029ccf (0x9772 bytes) 000226 1785394345.218143 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d31f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d31f): Mirror 2: 0xb65dc576 (stale) CRC-32 checksum value for chunk [0, 0x3d31f): Mirror 1: 0x4ec3beac Mirror 3: 0x4ec3beac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d31f: skipped 000227 1785394346.160239 punch from 0x02acf4 to 0x031df1 (0x70fe bytes) 000228 1785394346.192654 mapread 0x02d7ad thru 0x037de4 (0xa638 bytes) 000229 1785394346.209990 fallocate 0x02a239 thru 0x039086 (0xee4e bytes) 000230 1785394346.214212 mapwrite 0x01581d thru 0x01fff1 (0xa7d5 bytes) 000231 1785394346.245404 write 0x0324fb thru 0x03a747 (0x824d bytes) 000232 1785394346.257793 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d31f): Mirror 2: 0xb65dc576 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d31f): Mirror 2: 0xb65dc576 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d31f: skipped 000233 1785394346.684366 mapwrite 0x023405 thru 0x031c2a (0xe826 bytes) 000234 1785394346.854379 MIRROR_RESYNC 2 mirrors 000235 1785394347.436106 fallocate 0x038e51 thru 0x03ffff (0x71af bytes) 000236 1785394347.464417 write 0x002263 thru 0x010522 (0xe2c0 bytes) 000237 1785394347.473402 punch from 0x01ecf4 to 0x022fb3 (0x42c0 bytes) 000239 1785394347.478893 mapread 0x030f25 thru 0x031c3d (0xd19 bytes) 000240 1785394347.486265 mapread 0x0200bb thru 0x025f98 (0x5ede bytes) 000241 1785394347.495503 punch from 0x035b34 to 0x03656e (0xa3b bytes) 000242 1785394347.506144 punch from 0x02f54c to 0x0355f1 (0x60a6 bytes) 000243 1785394347.510403 mapwrite 0x032b37 thru 0x037b9e (0x5068 bytes) 000244 1785394347.533817 mapread 0x014d0e thru 0x019c10 (0x4f03 bytes) 000245 1785394347.542237 trunc from 0x03d31f to 0x0378b4 (0x5a6a bytes) 000246 1785394347.559044 mapread 0x0065e9 thru 0x0084ae (0x1ec6 bytes) 000247 1785394347.561381 mapwrite 0x03dd43 thru 0x03ffff (0x22bd bytes) 000248 1785394347.592382 mapwrite 0x00cc7a thru 0x017069 (0xa3f0 bytes) 000249 1785394347.612943 punch from 0x0041f1 to 0x00cead (0x8cbd bytes) 000250 1785394347.617350 punch from 0x00419e to 0x013c54 (0xfab7 bytes) 000251 1785394347.625151 MIRROR_RESYNC 2 mirrors 000252 1785394348.688448 fallocate 0x01e03b thru 0x027439 (0x93ff bytes) 000253 1785394348.716091 write 0x0267af thru 0x02f8cf (0x9121 bytes) 000254 1785394348.728545 fallocate 0x007af7 thru 0x010ea2 (0x93ac bytes) 000255 1785394348.734100 write 0x01eeec thru 0x02c824 (0xd939 bytes) 000256 1785394348.740363 mapwrite 0x0155bf thru 0x01a636 (0x5078 bytes) 000257 1785394348.779443 trunc from 0x040000 to 0x032ed3 (0xd12c bytes) 000258 1785394348.798752 mapwrite 0x00b191 thru 0x00b6ac (0x51c bytes) 000259 1785394348.815444 mapread 0x001478 thru 0x00e7f6 (0xd37f bytes) 000260 1785394348.821346 trunc from 0x014b3d to 0x0233ea (0xe8ae bytes) 000261 1785394348.825862 mapwrite 0x00bf0d thru 0x00f1f6 (0x32ea bytes) 000262 1785394348.840493 mapread 0x0224b7 thru 0x0297a5 (0x72ef bytes) 000263 1785394348.849099 write 0x018d80 thru 0x0248f1 (0xbb72 bytes) 000264 1785394348.854871 punch from 0x006783 to 0x00eb7c (0x83fa bytes) 000265 1785394348.860477 punch from 0x007c7e to 0x014a07 (0xcd8a bytes) 000266 1785394348.865571 fallocate 0x00ac5e thru 0x012f7e (0x8321 bytes) 000267 1785394348.870366 punch from 0x02ae22 to 0x02d214 (0x23f3 bytes) 000268 1785394348.875883 write 0x00f272 thru 0x015280 (0x600f bytes) 000269 1785394348.884779 mapwrite 0x003165 thru 0x0066cc (0x3568 bytes) 000270 1785394348.910192 trunc from 0x032ed4 to 0x03dada (0xac07 bytes) 000271 1785394348.926244 fallocate 0x00a0f0 thru 0x0110fe (0x700f bytes) 000272 1785394348.932056 mapread 0x0000f7 thru 0x004215 (0x411f bytes) 000273 1785394348.938602 trunc from 0x03dadb to 0x01a0ce (0x23a0c bytes) 000274 1785394348.954569 fallocate 0x00e3b7 thru 0x01b839 (0xd483 bytes) 000275 1785394348.960380 fallocate 0x023525 thru 0x02bb6b (0x8647 bytes) 000276 1785394348.966790 mapwrite 0x0275f7 thru 0x02a250 (0x2c5a bytes) 000277 1785394349.007832 mapread 0x013047 thru 0x021902 (0xe8bc bytes) 000278 1785394349.027108 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2a251): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a251): Mirror 2: 0x7bdfe6a0 (stale) CRC-32 checksum value for chunk [0, 0x2a251): Mirror 1: 0x5039273a Mirror 3: 0x5039273a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a251: skipped 000279 1785394349.969596 mapread 0x025cc8 thru 0x02a250 (0x4589 bytes) 000280 1785394350.015133 mapread 0x0154c3 thru 0x023049 (0xdb87 bytes) 000281 1785394350.027427 fallocate 0x013785 thru 0x015d5c (0x25d8 bytes) 000282 1785394350.059466 mapread 0x004a59 thru 0x00a2e2 (0x588a bytes) 000283 1785394350.073713 punch from 0x01db9f to 0x0258d3 (0x7d35 bytes) 000284 1785394350.078162 read 0x00883e thru 0x012be4 (0xa3a7 bytes) 000286 1785394350.087445 write 0x024f0f thru 0x030944 (0xba36 bytes) 000287 1785394350.099315 mapwrite 0x027e84 thru 0x0280b8 (0x235 bytes) 000288 1785394350.114319 write 0x0237b0 thru 0x02dc68 (0xa4b9 bytes) 000289 1785394350.119459 read 0x003b02 thru 0x007af9 (0x3ff8 bytes) 000290 1785394350.124137 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30945): Mirror 3: 0x66f80105 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30945): Mirror 3: 0x66f80105 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30945: skipped 000291 1785394350.339489 punch from 0x007573 to 0x011957 (0xa3e5 bytes) 000294 1785394350.348259 trunc from 0x030945 to 0x01fdf7 (0x10b4d bytes) 000295 1785394350.361614 mapwrite 0x004620 thru 0x00ae84 (0x6865 bytes) 000296 1785394350.395765 mapread 0x00bd62 thru 0x017a13 (0xbcb2 bytes) 000297 1785394350.404771 read 0x0041bb thru 0x00b234 (0x707a bytes) 000298 1785394350.408444 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000299 1785394350.510446 punch from 0x01e000 to 0x01fdf7 (0x1df8 bytes) 000301 1785394350.515875 trunc from 0x01fdf8 to 0x021d21 (0x1f2a bytes) 000302 1785394350.528209 read 0x00fee6 thru 0x01f655 (0xf770 bytes) 000303 1785394350.538147 mapread 0x017e0a thru 0x021aea (0x9ce1 bytes) 000304 1785394350.546401 write 0x00c8c0 thru 0x01a419 (0xdb5a bytes) 000305 1785394350.551904 trunc from 0x021d22 to 0x0337ba (0x11a99 bytes) 000307 1785394350.566770 write 0x026da6 thru 0x0330f6 (0xc351 bytes) 000308 1785394350.577588 trunc from 0x004eea to 0x005a76 (0xb8d bytes) 000309 1785394350.581330 fallocate 0x035d2f thru 0x03ba42 (0x5d14 bytes) 000311 1785394350.586403 trunc from 0x0337bb to 0x003423 (0x30397 bytes) 000312 1785394350.604460 trunc from 0x003424 to 0x00615b (0x2d38 bytes) 000313 1785394350.620190 mapwrite 0x01a679 thru 0x01f1a5 (0x4b2d bytes) 000314 1785394350.657853 mapwrite 0x012a1b thru 0x012ce1 (0x2c7 bytes) 000315 1785394350.674552 MIRROR_EXTEND to 2 mirrors 000316 1785394351.500063 fallocate 0x01fee3 thru 0x02753a (0x7658 bytes) 000319 1785394351.594042 trunc from 0x02753b to 0x03d348 (0x15e0e bytes) 000320 1785394351.633092 fallocate 0x0036ab thru 0x00a618 (0x6f6e bytes) 000321 1785394351.644055 punch from 0x016117 to 0x016c72 (0xb5c bytes) 000322 1785394351.656807 punch from 0x015938 to 0x0176a1 (0x1d6a bytes) 000323 1785394351.664709 write 0x02cf89 thru 0x03931b (0xc393 bytes) 000324 1785394351.680946 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d349): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d349): Mirror 2: 0x3b19ff42 (stale) CRC-32 checksum value for chunk [0, 0x3d349): Mirror 1: 0x6a50e95d Mirror 3: 0x6a50e95d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d349: skipped 000325 1785394352.809607 punch from 0x014045 to 0x01fc1c (0xbbd8 bytes) 000326 1785394352.847304 trunc from 0x03d349 to 0x02c319 (0x1102f bytes) 000327 1785394352.864286 mapread 0x00726b thru 0x00cacf (0x5865 bytes) 000328 1785394352.880849 write 0x0101b8 thru 0x01284f (0x2698 bytes) 000330 1785394352.887496 trunc from 0x02c31a to 0x00f5ae (0x1cd6b bytes) 000331 1785394352.902530 read 0x001e6e thru 0x00beb7 (0xa04a bytes) 000332 1785394352.907476 trunc from 0x0031b3 to 0x005fdb (0x2e29 bytes) 000333 1785394352.911753 write 0x0212eb thru 0x027b8b (0x68a1 bytes) 000334 1785394352.913584 write 0x0336de thru 0x0399f8 (0x631b bytes) 000335 1785394352.915459 trunc from 0x0399f9 to 0x013907 (0x260f1 bytes) 000336 1785394352.926343 write 0x014dee thru 0x01751f (0x2732 bytes) 000337 1785394352.928041 punch from 0x009f97 to 0x013abc (0x9b26 bytes) 000338 1785394352.932746 punch from 0x01102d to 0x016608 (0x55dc bytes) 000339 1785394352.943496 trunc from 0x00275e to 0x00e26a (0xbb0d bytes) 000340 1785394352.947868 read 0x01465f thru 0x01751f (0x2ec1 bytes) 000341 1785394352.956403 mapwrite 0x010078 thru 0x01cf1a (0xcea3 bytes) 000342 1785394352.992805 fallocate 0x029d7c thru 0x02c8ef (0x2b74 bytes) 000343 1785394352.997871 punch from 0x016a1a to 0x01bd51 (0x5338 bytes) 000344 1785394353.001803 punch from 0x01012e to 0x0186cf (0x85a2 bytes) 000345 1785394353.005824 write 0x00c97f thru 0x014acf (0x8151 bytes) 000346 1785394353.011188 read 0x013924 thru 0x01660e (0x2ceb bytes) 000347 1785394353.017413 write 0x01c765 thru 0x023ca3 (0x753f bytes) 000348 1785394353.019597 trunc from 0x023ca4 to 0x01a007 (0x9c9c bytes) 000349 1785394353.030341 trunc from 0x01a008 to 0x028641 (0xe63a bytes) 000350 1785394353.043525 mapread 0x0095f7 thru 0x00dea9 (0x48b3 bytes) 000352 1785394353.046296 trunc from 0x01f262 to 0x024508 (0x52a7 bytes) 000353 1785394353.049916 write 0x02be3b thru 0x02cd9d (0xf63 bytes) 000354 1785394353.051797 trunc from 0x01834d to 0x01ece2 (0x6996 bytes) 000355 1785394353.055405 write 0x038be6 thru 0x03ee82 (0x629d bytes) 000356 1785394353.057533 read 0x011cf0 thru 0x014f42 (0x3253 bytes) 000357 1785394353.058558 trunc from 0x0389c7 to 0x03ee82 (0x64bc bytes) 000358 1785394353.077097 fallocate 0x025512 thru 0x02ed34 (0x9823 bytes) 000359 1785394353.085845 mapwrite 0x0058ce thru 0x00f254 (0x9987 bytes) 000361 1785394353.106534 fallocate 0x00617f thru 0x00e164 (0x7fe6 bytes) 000363 1785394353.114057 mapread 0x001ad4 thru 0x010e5f (0xf38c bytes) 000365 1785394353.121525 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ee83): Mirror 2: 0xc16e4528 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ee83): Mirror 2: 0xc16e4528 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ee83: skipped 000366 1785394353.336023 trunc from 0x03ee83 to 0x0182f1 (0x26b91 bytes) 000367 1785394353.356313 mapwrite 0x021444 thru 0x0291df (0x7d9c bytes) 000368 1785394353.403692 read 0x011a30 thru 0x013f3d (0x250e bytes) 000370 1785394353.409550 read 0x0044d3 thru 0x00d056 (0x8b84 bytes) 000371 1785394353.415813 fallocate 0x016f83 thru 0x01e6be (0x773c bytes) 000372 1785394353.419630 write 0x02ddb4 thru 0x036450 (0x869d bytes) 000373 1785394353.421187 punch from 0x0269fc to 0x028ac3 (0x20c8 bytes) 000374 1785394353.424966 trunc from 0x036451 to 0x009d5c (0x2c6f4 bytes) 000375 1785394353.437548 punch from 0x009606 to 0x009d5c (0x757 bytes) 000376 1785394353.441672 write 0x0230a2 thru 0x02b87a (0x87d9 bytes) 000378 1785394353.444712 trunc from 0x02b87b to 0x017f22 (0x13958 bytes) 000379 1785394353.506680 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000380 1785394353.720427 fallocate 0x01cc78 thru 0x02beba (0xf243 bytes) 000381 1785394353.724154 write 0x007281 thru 0x010fa0 (0x9d20 bytes) 000382 1785394353.774436 trunc from 0x017f23 to 0x02cd2d (0x14e0b bytes) 000383 1785394353.840354 MIRROR_EXTEND to 2 mirrors 000384 1785394354.994222 trunc from 0x02cd2e to 0x00261c (0x2a711 bytes) 000386 1785394355.078476 mapread 0x0022d1 thru 0x00261c (0x34c bytes) 000387 1785394355.089825 punch from 0x001380 to 0x00261c (0x129d bytes) 000388 1785394355.100801 punch from 0x000a3e to 0x00261c (0x1bdf bytes) 000389 1785394355.112348 trunc from 0x00223a to 0x00261c (0x3e3 bytes) 000390 1785394355.123084 trunc from 0x00261d to 0x0042b2 (0x1c96 bytes) 000391 1785394355.156614 mapwrite 0x02eb3f thru 0x035f35 (0x73f7 bytes) 000393 1785394355.292763 mapread 0x015b34 thru 0x023861 (0xdd2e bytes) 000394 1785394355.317653 mapread 0x02f2f4 thru 0x035f35 (0x6c42 bytes) 000395 1785394355.324435 fallocate 0x00cc04 thru 0x01b4e6 (0xe8e3 bytes) 000396 1785394355.337565 trunc from 0x035f36 to 0x0042b2 (0x31c83 bytes) 000398 1785394355.369616 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000399 1785394355.636212 fallocate 0x003048 thru 0x00bccf (0x8c88 bytes) 000400 1785394355.647675 punch from 0x00251d to 0x00bccf (0x97b3 bytes) 000401 1785394355.652426 trunc from 0x00bcd0 to 0x03e1b6 (0x324e7 bytes) 000402 1785394355.687281 write 0x036793 thru 0x03ffff (0x986d bytes) 000403 1785394355.700827 fallocate 0x0083a7 thru 0x016881 (0xe4db bytes) 000404 1785394355.712917 write 0x016f14 thru 0x01a228 (0x3315 bytes) 000405 1785394355.736845 mapread 0x03a040 thru 0x03ffff (0x5fc0 bytes) 000406 1785394355.741430 write 0x021d3d thru 0x02cf92 (0xb256 bytes) 000407 1785394355.762054 fallocate 0x02b353 thru 0x0390a1 (0xdd4f bytes) 000408 1785394355.772167 write 0x00f880 thru 0x01caa4 (0xd225 bytes) 000410 1785394355.809903 punch from 0x0022e4 to 0x009360 (0x707d bytes) 000411 1785394355.817217 trunc from 0x02b168 to 0x036984 (0xb81d bytes) 000412 1785394355.850407 trunc from 0x00c1ae to 0x017ceb (0xbb3e bytes) 000413 1785394355.874272 read 0x00d831 thru 0x017f38 (0xa708 bytes) 000414 1785394355.896472 write 0x03685f thru 0x03ffff (0x97a1 bytes) 000415 1785394355.906298 mapread 0x014fbc thru 0x016fa6 (0x1feb bytes) 000416 1785394355.908475 trunc from 0x01da02 to 0x01e005 (0x604 bytes) 000417 1785394355.921127 mapread 0x000884 thru 0x009ff3 (0x9770 bytes) 000418 1785394355.991755 mapread 0x00d428 thru 0x0121fb (0x4dd4 bytes) 000419 1785394355.992933 write 0x02b57e thru 0x02f98a (0x440d bytes) 000420 1785394356.034788 trunc from 0x040000 to 0x0011c9 (0x3ee36 bytes) 000421 1785394356.080154 mapread 0x0005ff thru 0x0011c9 (0xbcb bytes) 000422 1785394356.084918 punch from 0x000151 to 0x0011c9 (0x1079 bytes) 000423 1785394356.098190 punch from 0x00056c to 0x0011c9 (0xc5e bytes) 000424 1785394356.110984 punch from 0x000e60 to 0x0011c9 (0x36a bytes) 000425 1785394356.118494 trunc from 0x0011ca to 0x010bdc (0xfa13 bytes) 000426 1785394356.151800 read 0x009d75 thru 0x010bdc (0x6e68 bytes) 000427 1785394356.189571 mapread 0x0070e1 thru 0x00c0fe (0x501e bytes) 000430 1785394356.191704 fallocate 0x023814 thru 0x02e667 (0xae54 bytes) 000431 1785394356.211157 trunc from 0x010bdd to 0x00b91d (0x52bf bytes) 000432 1785394356.245451 read 0x006d9e thru 0x00b320 (0x4583 bytes) 000434 1785394356.246599 read 0x0059a0 thru 0x0067ef (0xe50 bytes) 000435 1785394356.247173 trunc from 0x00b91e to 0x030693 (0x24d76 bytes) 000436 1785394356.280837 write 0x0165b7 thru 0x01cb98 (0x65e2 bytes) 000437 1785394356.322458 trunc from 0x030694 to 0x016295 (0x1a3fe bytes) 000438 1785394356.361424 mapwrite 0x039c35 thru 0x03ffff (0x63cb bytes) 000439 1785394356.426454 read 0x028b06 thru 0x030188 (0x7683 bytes) 000440 1785394356.451789 fallocate 0x038601 thru 0x03ffff (0x79ff bytes) 000441 1785394356.463281 trunc from 0x030e5e to 0x0372d8 (0x647b bytes) 000442 1785394356.476230 trunc from 0x040000 to 0x03dfdf (0x2020 bytes) 000443 1785394356.531140 MIRROR_EXTEND to 2 mirrors 000444 1785394357.800315 mapread 0x01f468 thru 0x0288e2 (0x947b bytes) 000445 1785394357.856501 trunc from 0x03dfe0 to 0x03969b (0x4944 bytes) 000446 1785394357.965757 trunc from 0x03969c to 0x03e8ab (0x5210 bytes) 000447 1785394357.994704 write 0x01fe0e thru 0x02f403 (0xf5f6 bytes) 000448 1785394358.029513 trunc from 0x03e8ac to 0x00a615 (0x34296 bytes) 000449 1785394358.059213 mapread 0x0040fc thru 0x006f71 (0x2e76 bytes) 000450 1785394358.092320 punch from 0x003c82 to 0x00a615 (0x6994 bytes) 000451 1785394358.113756 read 0x000fc8 thru 0x00a615 (0x964e bytes) 000452 1785394358.124954 write 0x023275 thru 0x02a4df (0x726b bytes) 000453 1785394358.148465 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000455 1785394358.396478 mapread 0x009037 thru 0x0149a5 (0xb96f bytes) 000456 1785394358.425459 punch from 0x0054f9 to 0x012bb5 (0xd6bd bytes) 000457 1785394358.430116 read 0x021e78 thru 0x02a40c (0x8595 bytes) 000458 1785394358.438624 fallocate 0x033c16 thru 0x03b1dc (0x75c7 bytes) 000459 1785394358.443333 mapwrite 0x01f1b7 thru 0x0286af (0x94f9 bytes) 000460 1785394358.466562 fallocate 0x03ee49 thru 0x03ffff (0x11b7 bytes) 000461 1785394358.471602 fallocate 0x001562 thru 0x00d1fb (0xbc9a bytes) 000462 1785394358.482407 MIRROR_EXTEND to 2 mirrors 000464 1785394359.137430 mapwrite 0x02aae1 thru 0x037e2a (0xd34a bytes) 000465 1785394359.273271 trunc from 0x03b1dd to 0x03265b (0x8b81 bytes) 000466 1785394359.330535 mapwrite 0x0153cf thru 0x018ffc (0x3c2e bytes) 000467 1785394359.424422 write 0x02f225 thru 0x033cb3 (0x4a8f bytes) 000468 1785394359.427529 write 0x0097ca thru 0x00cb51 (0x3388 bytes) 000469 1785394359.462083 mapwrite 0x01fe67 thru 0x02ba0e (0xbba8 bytes) 000470 1785394359.546013 mapwrite 0x031a34 thru 0x03ede7 (0xd3b4 bytes) 000471 1785394359.680050 mapread 0x00093f thru 0x00d41d (0xcadf bytes) 000472 1785394359.695507 read 0x03e6ba thru 0x03ede7 (0x72e bytes) 000473 1785394359.696867 mapread 0x017873 thru 0x023277 (0xba05 bytes) 000474 1785394359.699257 punch from 0x0315d5 to 0x0398d3 (0x82ff bytes) 000475 1785394359.703596 trunc from 0x03ede8 to 0x0028bf (0x3c528 bytes) 000476 1785394359.720453 punch from 0x00112b to 0x0028bf (0x1795 bytes) 000477 1785394359.724357 mapwrite 0x01d548 thru 0x027fbc (0xaa75 bytes) 000478 1785394359.761538 write 0x005056 thru 0x0113f6 (0xc3a1 bytes) 000479 1785394359.768967 trunc from 0x00473a to 0x009fa7 (0x586e bytes) 000480 1785394359.778740 mapwrite 0x00f470 thru 0x00fe09 (0x99a bytes) 000482 1785394359.788568 trunc from 0x027fbd to 0x027a25 (0x597 bytes) 000483 1785394359.799789 trunc from 0x027a26 to 0x0221d0 (0x5855 bytes) 000485 1785394359.816991 punch from 0x010102 to 0x01333d (0x323c bytes) 000486 1785394359.822172 fallocate 0x03f9df thru 0x03ffff (0x621 bytes) 000487 1785394359.825992 trunc from 0x040000 to 0x016547 (0x29ab8 bytes) 000488 1785394359.839430 mapwrite 0x03868c thru 0x03cbfb (0x4570 bytes) 000489 1785394359.877131 write 0x00d0c9 thru 0x015d7d (0x8cb5 bytes) 000490 1785394359.884602 trunc from 0x01a945 to 0x01e2dd (0x3999 bytes) 000492 1785394359.889933 mapwrite 0x0395d1 thru 0x03ffff (0x6a2f bytes) 000493 1785394359.920630 write 0x01f163 thru 0x029011 (0x9eaf bytes) 000495 1785394359.930066 trunc from 0x040000 to 0x03370e (0xc8f1 bytes) 000496 1785394359.945789 trunc from 0x00416e to 0x0121e1 (0xe074 bytes) 000497 1785394359.958779 MIRROR_RESYNC 2 mirrors 000498 1785394360.610584 write 0x007d5a thru 0x00bcca (0x3f71 bytes) 000499 1785394360.663264 write 0x035ca2 thru 0x03e998 (0x8cf7 bytes) 000500 1785394360.666945 punch from 0x00ff7a to 0x01bdd0 (0xbe57 bytes) 000501 1785394360.671925 trunc from 0x03e999 to 0x03a8b8 (0x40e0 bytes) 000502 1785394360.733908 write 0x01417a thru 0x022e9b (0xed22 bytes) 000503 1785394360.765869 fallocate 0x02a9e0 thru 0x03135d (0x697e bytes) 000504 1785394360.793240 MIRROR_RESYNC 2 mirrors 000505 1785394361.313667 trunc from 0x01a090 to 0x028bc0 (0xeb31 bytes) 000506 1785394361.346377 read 0x0282f0 thru 0x036361 (0xe072 bytes) 000507 1785394361.356095 mapwrite 0x0199c2 thru 0x01baca (0x2109 bytes) 000509 1785394361.381443 mapwrite 0x009e13 thru 0x00e78b (0x4979 bytes) 000510 1785394361.411009 mapwrite 0x03437d thru 0x03f108 (0xad8c bytes) 000511 1785394361.461326 mapwrite 0x00818b thru 0x00fca0 (0x7b16 bytes) 000512 1785394361.481798 read 0x021fae thru 0x02ddec (0xbe3f bytes) 000513 1785394361.491590 write 0x00a0c7 thru 0x00bd00 (0x1c3a bytes) 000514 1785394361.497036 MIRROR_RESYNC 2 mirrors 000515 1785394362.161454 write 0x03f374 thru 0x03ffff (0xc8c bytes) 000517 1785394362.205106 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000518 1785394362.465402 trunc from 0x040000 to 0x034540 (0xbabf bytes) 000519 1785394362.560856 mapwrite 0x001ae6 thru 0x004c08 (0x3123 bytes) 000520 1785394362.627839 fallocate 0x002d7a thru 0x0049e6 (0x1c6d bytes) 000521 1785394362.640951 fallocate 0x03d5e0 thru 0x03e82d (0x124e bytes) 000522 1785394362.661367 mapwrite 0x00a912 thru 0x00bacc (0x11bb bytes) 000523 1785394362.705054 trunc from 0x00d5d0 to 0x0175dd (0xa00e bytes) 000524 1785394362.725329 trunc from 0x03e82e to 0x0177c9 (0x27064 bytes) 000525 1785394362.753713 read 0x0108f5 thru 0x0177c9 (0x6ed5 bytes) 000526 1785394362.782982 trunc from 0x0177ca to 0x00e263 (0x9566 bytes) 000527 1785394362.801540 trunc from 0x008f28 to 0x00a334 (0x140d bytes) 000528 1785394362.814278 trunc from 0x00da99 to 0x00e263 (0x7cb bytes) 000529 1785394362.820991 mapwrite 0x01497f thru 0x01b51e (0x6ba0 bytes) 000530 1785394362.918893 read 0x017e48 thru 0x01b51e (0x36d7 bytes) 000531 1785394362.921450 trunc from 0x005586 to 0x0134b8 (0xdf33 bytes) 000533 1785394362.940630 read 0x01ac00 thru 0x01b51e (0x91f bytes) 000534 1785394362.942103 write 0x006ac5 thru 0x00bc19 (0x5155 bytes) 000535 1785394362.957788 trunc from 0x004ee3 to 0x00581d (0x93b bytes) 000536 1785394362.978056 punch from 0x002fe8 to 0x00edac (0xbdc5 bytes) 000537 1785394363.000063 write 0x02c56b thru 0x03bcdc (0xf772 bytes) 000538 1785394363.004492 fallocate 0x02c4cb thru 0x030e18 (0x494e bytes) 000539 1785394363.024847 mapread 0x033f1d thru 0x03926f (0x5353 bytes) 000540 1785394363.030567 trunc from 0x03bcdd to 0x01b5fe (0x206de bytes) 000541 1785394363.075480 trunc from 0x009877 to 0x00f81e (0x5fa8 bytes) 000542 1785394363.082684 read 0x00a225 thru 0x00f97c (0x5758 bytes) 000543 1785394363.133080 fallocate 0x02c45c thru 0x034a3d (0x85e2 bytes) 000544 1785394363.142481 write 0x0048c6 thru 0x00fccb (0xb406 bytes) 000545 1785394363.144340 fallocate 0x010e37 thru 0x019441 (0x860b bytes) 000546 1785394363.158050 read 0x02ad5e thru 0x034a3d (0x9ce0 bytes) 000547 1785394363.209895 mapwrite 0x03d300 thru 0x03ffff (0x2d00 bytes) 000548 1785394363.333401 punch from 0x034d84 to 0x037315 (0x2592 bytes) 000549 1785394363.343828 mapread 0x018cc1 thru 0x024fbc (0xc2fc bytes) 000550 1785394363.345165 read 0x02eae9 thru 0x03bdf1 (0xd309 bytes) 000551 1785394363.365529 fallocate 0x015664 thru 0x01cf9b (0x7938 bytes) 000555 1785394363.393391 punch from 0x0112e0 to 0x013cd2 (0x29f3 bytes) 000556 1785394363.400501 trunc from 0x040000 to 0x00ee0a (0x311f5 bytes) 000557 1785394363.428146 read 0x00e367 thru 0x00ee0a (0xaa4 bytes) 000558 1785394363.429411 read 0x001e38 thru 0x005fe4 (0x41ad bytes) 000559 1785394363.437847 trunc from 0x00ee0b to 0x014073 (0x5269 bytes) 000560 1785394363.496709 write 0x0310f4 thru 0x0390cc (0x7fd9 bytes) 000561 1785394363.498165 mapwrite 0x03c7b7 thru 0x03ffff (0x3849 bytes) 000563 1785394363.598204 fallocate 0x034962 thru 0x03ba2e (0x70cd bytes) 000564 1785394363.613809 mapwrite 0x029930 thru 0x035d74 (0xc445 bytes) 000565 1785394363.679471 mapwrite 0x0356ce thru 0x03ffff (0xa932 bytes) 000566 1785394363.739311 write 0x01577c thru 0x0171e7 (0x1a6c bytes) 000567 1785394363.740299 punch from 0x020c3e to 0x02f7b3 (0xeb76 bytes) 000568 1785394363.755848 mapread 0x0131a2 thru 0x017a87 (0x48e6 bytes) 000569 1785394363.756778 read 0x028a84 thru 0x02c3a3 (0x3920 bytes) 000570 1785394363.792857 fallocate 0x010139 thru 0x01e75c (0xe624 bytes) 000571 1785394363.810788 mapread 0x00f75b thru 0x0187a9 (0x904f bytes) 000572 1785394363.811859 mapwrite 0x029b93 thru 0x02e0df (0x454d bytes) 000574 1785394363.873969 read 0x017b9f thru 0x01ba9f (0x3f01 bytes) 000575 1785394363.874705 mapread 0x0224ae thru 0x02f6e7 (0xd23a bytes) 000576 1785394363.879551 mapwrite 0x02b554 thru 0x02bda3 (0x850 bytes) 000577 1785394363.890806 mapread 0x03ba5e thru 0x03ffff (0x45a2 bytes) 000579 1785394363.901465 mapwrite 0x032795 thru 0x03ffff (0xd86b bytes) 000580 1785394363.971235 mapwrite 0x01373e thru 0x01cbff (0x94c2 bytes) 000581 1785394364.035731 read 0x0154e0 thru 0x01aa09 (0x552a bytes) 000582 1785394364.037689 mapwrite 0x027786 thru 0x02ef69 (0x77e4 bytes) 000583 1785394364.100506 trunc from 0x040000 to 0x000d95 (0x3f26a bytes) 000584 1785394364.141537 mapwrite 0x0031bf thru 0x0055d6 (0x2418 bytes) 000585 1785394364.219569 mapwrite 0x006ab5 thru 0x008ae0 (0x202c bytes) 000586 1785394364.269982 mapread 0x001c89 thru 0x001d1c (0x94 bytes) 000587 1785394364.271972 trunc from 0x008ae1 to 0x02a960 (0x21e80 bytes) 000588 1785394364.331527 read 0x006a19 thru 0x01286b (0xbe53 bytes) 000589 1785394364.346518 mapread 0x0027fa thru 0x0059a0 (0x31a7 bytes) 000590 1785394364.348406 fallocate 0x00ec6f thru 0x0169a0 (0x7d32 bytes) 000591 1785394364.359376 punch from 0x010175 to 0x01b2cb (0xb157 bytes) 000592 1785394364.372855 read 0x012ce4 thru 0x0131a4 (0x4c1 bytes) 000596 1785394364.387423 fallocate 0x036bdb thru 0x03ffff (0x9425 bytes) 000597 1785394364.397508 write 0x024810 thru 0x03205f (0xd850 bytes) 000598 1785394364.401391 read 0x02dbe7 thru 0x03205f (0x4479 bytes) 000599 1785394364.403481 mapwrite 0x00602e thru 0x00827d (0x2250 bytes) 000600 1785394364.489399 read 0x020c0f thru 0x02c6d4 (0xbac6 bytes) 000601 1785394364.491405 fallocate 0x02012e thru 0x02093a (0x80d bytes) 000602 1785394364.505478 trunc from 0x0139e5 to 0x01d449 (0x9a65 bytes) 000603 1785394364.526622 read 0x020ccf thru 0x02a216 (0x9548 bytes) 000604 1785394364.529183 fallocate 0x03683a thru 0x03ffff (0x97c6 bytes) 000605 1785394364.554487 mapwrite 0x018ec5 thru 0x021efb (0x9037 bytes) 000606 1785394364.618357 mapwrite 0x023942 thru 0x03044d (0xcb0c bytes) 000607 1785394364.684177 mapread 0x035710 thru 0x037c6a (0x255b bytes) 000609 1785394364.694347 trunc from 0x019c66 to 0x01d7ff (0x3b9a bytes) 000610 1785394364.708198 fallocate 0x029657 thru 0x02c9b9 (0x3363 bytes) 000611 1785394364.713288 mapread 0x033ce1 thru 0x03776f (0x3a8f bytes) 000612 1785394364.719353 read 0x000723 thru 0x002d2e (0x260c bytes) 000613 1785394364.719992 read 0x01e61f thru 0x022b98 (0x457a bytes) 000614 1785394364.720574 mapread 0x00a67c thru 0x0192eb (0xec70 bytes) 000615 1785394364.760368 trunc from 0x040000 to 0x011366 (0x2ec99 bytes) 000616 1785394364.810752 read 0x00d33b thru 0x011366 (0x402c bytes) 000617 1785394364.811356 mapwrite 0x006162 thru 0x0065cd (0x46c bytes) 000618 1785394364.840548 punch from 0x00639a to 0x0104ad (0xa114 bytes) 000620 1785394364.861970 punch from 0x00c985 to 0x011366 (0x49e2 bytes) 000621 1785394364.868743 mapread 0x00ee80 thru 0x011366 (0x24e7 bytes) 000622 1785394364.878532 trunc from 0x010f6c to 0x011366 (0x3fb bytes) 000623 1785394364.900290 trunc from 0x011367 to 0x0150fd (0x3d97 bytes) 000626 1785394364.941807 mapwrite 0x025729 thru 0x031096 (0xb96e bytes) 000627 1785394365.122027 write 0x02bd62 thru 0x0328a0 (0x6b3f bytes) 000628 1785394365.124033 trunc from 0x0328a1 to 0x01526d (0x1d633 bytes) 000629 1785394365.173842 mapwrite 0x0122aa thru 0x01b5bd (0x9314 bytes) 000630 1785394365.300244 trunc from 0x01b5be to 0x0292dc (0xdd1f bytes) 000631 1785394365.313188 mapwrite 0x00c60e thru 0x019cbb (0xd6ae bytes) 000633 1785394365.392132 trunc from 0x000b6a to 0x00d621 (0xcab8 bytes) 000634 1785394365.399284 mapwrite 0x0260ad thru 0x031434 (0xb388 bytes) 000635 1785394365.633318 mapread 0x01c76f thru 0x01d77a (0x100c bytes) 000637 1785394365.666436 trunc from 0x031435 to 0x02925d (0x81d7 bytes) 000638 1785394365.688137 mapwrite 0x024818 thru 0x027f30 (0x3719 bytes) 000639 1785394365.741620 write 0x003563 thru 0x0073f0 (0x3e8e bytes) 000640 1785394365.753170 mapread 0x01001d thru 0x017c4c (0x7c30 bytes) 000641 1785394365.756804 mapread 0x020137 thru 0x0260b0 (0x5f7a bytes) 000642 1785394365.830285 fallocate 0x02f34d thru 0x03bdd0 (0xca84 bytes) 000643 1785394365.848920 fallocate 0x00069d thru 0x004be4 (0x4548 bytes) 000644 1785394365.872944 read 0x010375 thru 0x016b70 (0x67fc bytes) 000645 1785394365.878020 mapwrite 0x02c796 thru 0x034838 (0x80a3 bytes) 000646 1785394365.990542 MIRROR_EXTEND to 2 mirrors 000647 1785394367.406260 mapwrite 0x03158d thru 0x0357c3 (0x4237 bytes) 000648 1785394367.571412 read 0x00497a thru 0x012979 (0xe000 bytes) 000649 1785394367.584813 fallocate 0x03a145 thru 0x03ffff (0x5ebb bytes) 000651 1785394367.606910 read 0x008bf0 thru 0x00919a (0x5ab bytes) 000652 1785394367.610376 fallocate 0x01591d thru 0x021f25 (0xc609 bytes) 000653 1785394367.627386 write 0x0065fc thru 0x00800a (0x1a0f bytes) 000655 1785394367.641884 write 0x0144fc thru 0x015f62 (0x1a67 bytes) 000656 1785394367.688592 read 0x0013e3 thru 0x0106d0 (0xf2ee bytes) 000658 1785394367.690515 fallocate 0x0017be thru 0x00e5ef (0xce32 bytes) 000659 1785394367.700869 trunc from 0x040000 to 0x009db1 (0x3624e bytes) 000660 1785394367.784553 mapwrite 0x015d4e thru 0x0211b3 (0xb466 bytes) 000661 1785394367.956406 punch from 0x020949 to 0x0211b3 (0x86b bytes) 000662 1785394367.978465 trunc from 0x0211b4 to 0x010eba (0x102f9 bytes) 000663 1785394368.013782 trunc from 0x004f36 to 0x009af7 (0x4bc2 bytes) 000664 1785394368.035402 MIRROR_RESYNC 2 mirrors 000665 1785394368.928449 trunc from 0x010ebb to 0x019709 (0x884f bytes) 000666 1785394368.986715 trunc from 0x01970a to 0x008a23 (0x10ce6 bytes) 000667 1785394369.010809 fallocate 0x01ee1c thru 0x01fe8e (0x1073 bytes) 000668 1785394369.026230 fallocate 0x02ab6b thru 0x033360 (0x87f6 bytes) 000669 1785394369.039238 mapwrite 0x0159b9 thru 0x025857 (0xfe9f bytes) 000670 1785394369.155304 write 0x039e45 thru 0x039fb3 (0x16f bytes) 000671 1785394369.156406 punch from 0x01b976 to 0x027b90 (0xc21b bytes) 000672 1785394369.167522 trunc from 0x0088d1 to 0x010b70 (0x82a0 bytes) 000673 1785394369.182484 fallocate 0x00e703 thru 0x019b32 (0xb430 bytes) 000674 1785394369.193005 read 0x01645d thru 0x01af38 (0x4adc bytes) 000675 1785394369.209227 MIRROR_RESYNC 2 mirrors 000678 1785394370.224308 trunc from 0x011863 to 0x0178ac (0x604a bytes) 000680 1785394370.285944 read 0x0047df thru 0x013a5c (0xf27e bytes) 000681 1785394370.314032 punch from 0x022354 to 0x028d28 (0x69d5 bytes) 000682 1785394370.320941 mapread 0x01e0da thru 0x0246d4 (0x65fb bytes) 000683 1785394370.369662 mapread 0x02a0e7 thru 0x038cd8 (0xebf2 bytes) 000684 1785394370.402031 punch from 0x01dc48 to 0x023a86 (0x5e3f bytes) 000685 1785394370.424583 trunc from 0x039fb4 to 0x01ca87 (0x1d52c bytes) 000686 1785394370.483236 trunc from 0x01ca88 to 0x01be99 (0xbee bytes) 000687 1785394370.521426 trunc from 0x013bdc to 0x01be99 (0x82be bytes) 000688 1785394370.525889 punch from 0x0051be to 0x00c018 (0x6e5b bytes) 000689 1785394370.531120 trunc from 0x00c512 to 0x01b655 (0xf144 bytes) 000690 1785394370.535792 write 0x03ca49 thru 0x03e561 (0x1b19 bytes) 000691 1785394370.539861 write 0x039008 thru 0x03ffff (0x6ff8 bytes) 000692 1785394370.545760 punch from 0x000239 to 0x00c32d (0xc0f5 bytes) 000693 1785394370.555189 mapread 0x034ec5 thru 0x03b0ae (0x61ea bytes) 000694 1785394370.572054 trunc from 0x030761 to 0x035f98 (0x5838 bytes) 000695 1785394370.580022 write 0x002609 thru 0x005e2c (0x3824 bytes) 000696 1785394370.614443 MIRROR_RESYNC 2 mirrors 000697 1785394371.781284 fallocate 0x011e20 thru 0x018c0f (0x6df0 bytes) 000699 1785394371.837671 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000700 1785394372.149591 mapread 0x00d652 thru 0x0100f5 (0x2aa4 bytes) 000701 1785394372.197823 write 0x02f0a6 thru 0x031ac9 (0x2a24 bytes) 000702 1785394372.242332 trunc from 0x023d39 to 0x02e902 (0xabca bytes) 000703 1785394372.250878 fallocate 0x01c5d6 thru 0x0218a4 (0x52cf bytes) 000704 1785394372.255008 trunc from 0x040000 to 0x003cb5 (0x3c34a bytes) 000705 1785394372.299735 read 0x001d72 thru 0x003cb5 (0x1f44 bytes) 000706 1785394372.320473 MIRROR_EXTEND to 2 mirrors 000707 1785394373.352379 trunc from 0x0019df to 0x003cb5 (0x22d7 bytes) 000708 1785394373.442118 trunc from 0x003cb6 to 0x017a3d (0x13d88 bytes) 000709 1785394373.538947 mapwrite 0x00581f thru 0x013b67 (0xe349 bytes) 000710 1785394373.657680 mapread 0x00a2a5 thru 0x012af5 (0x8851 bytes) 000711 1785394373.665204 write 0x02f51c thru 0x03e8bc (0xf3a1 bytes) 000712 1785394373.676111 trunc from 0x031f76 to 0x036843 (0x48ce bytes) 000713 1785394373.744000 trunc from 0x03e8bd to 0x026321 (0x1859b bytes) 000714 1785394373.768253 punch from 0x0208ab to 0x026321 (0x5a77 bytes) 000715 1785394373.775454 write 0x030da4 thru 0x036af3 (0x5d50 bytes) 000716 1785394373.783176 trunc from 0x036af4 to 0x00187b (0x35278 bytes) 000717 1785394373.801465 fallocate 0x000ede thru 0x00eb66 (0xdc89 bytes) 000719 1785394373.816248 mapwrite 0x033c4d thru 0x035d9d (0x2151 bytes) 000721 1785394373.885008 trunc from 0x035d9e to 0x0368d8 (0xb3b bytes) 000722 1785394373.927345 write 0x00a0d5 thru 0x00dc92 (0x3bbe bytes) 000723 1785394373.950468 trunc from 0x0368d9 to 0x020c10 (0x15cc8 bytes) 000724 1785394373.996986 read 0x012cea thru 0x020c10 (0xdf27 bytes) 000725 1785394374.011209 mapwrite 0x03afb5 thru 0x03f520 (0x456c bytes) 000726 1785394374.181062 read 0x011e08 thru 0x011e45 (0x3e bytes) 000727 1785394374.213267 MIRROR_RESYNC 2 mirrors 000728 1785394375.134885 fallocate 0x0214f0 thru 0x028649 (0x715a bytes) 000729 1785394375.199016 mapread 0x0207c5 thru 0x02bf66 (0xb7a2 bytes) 000730 1785394375.269234 mapwrite 0x01aaef thru 0x025b5a (0xb06c bytes) 000731 1785394375.437262 write 0x035b90 thru 0x03ffff (0xa470 bytes) 000732 1785394375.448858 mapread 0x01a2a8 thru 0x028532 (0xe28b bytes) 000733 1785394375.450147 mapread 0x03eea5 thru 0x03ffff (0x115b bytes) 000734 1785394375.461836 trunc from 0x026cd0 to 0x0313dd (0xa70e bytes) 000735 1785394375.479356 write 0x00b73d thru 0x011c11 (0x64d5 bytes) 000736 1785394375.523356 trunc from 0x034703 to 0x03c1cb (0x7ac9 bytes) 000737 1785394375.547907 mapwrite 0x02e72c thru 0x037597 (0x8e6c bytes) 000738 1785394375.680562 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc1602614 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3b6889c9 Mirror 3: 0x3b6889c9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000739 1785394377.634005 trunc from 0x040000 to 0x00e2bf (0x31d40 bytes) 000740 1785394377.766716 mapread 0x0090af thru 0x00e2bf (0x5211 bytes) 000741 1785394377.849048 punch from 0x00e04a to 0x00e2bf (0x276 bytes) 000742 1785394377.856257 read 0x008307 thru 0x00e2bf (0x5fb9 bytes) 000743 1785394377.888738 trunc from 0x000c5e to 0x00e2bf (0xd662 bytes) 000744 1785394377.910274 fallocate 0x0325bd thru 0x03a3ac (0x7df0 bytes) 000745 1785394377.951601 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a3ad): Mirror 3: 0x32ecb610 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a3ad): Mirror 3: 0x32ecb610 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a3ad: skipped 000747 1785394378.603337 punch from 0x00a5da to 0x00acab (0x6d2 bytes) 000748 1785394378.616617 punch from 0x019686 to 0x027754 (0xe0cf bytes) 000749 1785394378.625802 read 0x037806 thru 0x03a3ac (0x2ba7 bytes) 000750 1785394378.635455 punch from 0x0177ca to 0x019c2c (0x2463 bytes) 000752 1785394378.658793 mapread 0x022d7f thru 0x024ccb (0x1f4d bytes) 000753 1785394378.728997 punch from 0x0301e5 to 0x03a3ac (0xa1c8 bytes) 000754 1785394378.733622 mapwrite 0x028d4f thru 0x0313d1 (0x8683 bytes) 000755 1785394378.829223 fallocate 0x02ae01 thru 0x030479 (0x5679 bytes) 000756 1785394378.846406 fallocate 0x01da3b thru 0x0251e5 (0x77ab bytes) 000757 1785394378.867334 punch from 0x0066ef to 0x00e8b7 (0x81c9 bytes) 000758 1785394378.882169 trunc from 0x03a3ad to 0x026259 (0x14153 bytes) 000759 1785394378.936589 trunc from 0x00c618 to 0x00db06 (0x14ef bytes) 000760 1785394378.950982 trunc from 0x02625a to 0x03aa05 (0x147ac bytes) 000761 1785394378.987110 write 0x02fe79 thru 0x03a2e3 (0xa46b bytes) 000764 1785394379.010060 MIRROR_RESYNC 2 mirrors 000765 1785394380.008077 trunc from 0x03aa06 to 0x039aaf (0xf56 bytes) 000766 1785394380.073665 punch from 0x01f3af to 0x029ae6 (0xa738 bytes) 000767 1785394380.081476 read 0x02d3ad thru 0x03569f (0x82f3 bytes) 000768 1785394380.095238 mapwrite 0x01025e thru 0x018046 (0x7de9 bytes) 000769 1785394380.129669 trunc from 0x01e3d3 to 0x02a3f0 (0xc01e bytes) 000770 1785394380.133283 read 0x0388f2 thru 0x039aaf (0x11be bytes) 000771 1785394380.140529 write 0x003d75 thru 0x010c90 (0xcf1c bytes) 000772 1785394380.142143 trunc from 0x039ab0 to 0x0255a6 (0x14509 bytes) 000773 1785394380.154856 read 0x01d533 thru 0x0255a6 (0x8074 bytes) 000774 1785394380.155552 fallocate 0x008150 thru 0x009038 (0xee9 bytes) 000775 1785394380.160656 mapwrite 0x034d54 thru 0x03d266 (0x8513 bytes) 000776 1785394380.214234 punch from 0x00bc89 to 0x00dc1d (0x1f95 bytes) 000777 1785394380.226733 MIRROR_RESYNC 2 mirrors 000779 1785394381.183371 mapwrite 0x0284da thru 0x036d9c (0xe8c3 bytes) 000780 1785394381.478494 mapwrite 0x003e75 thru 0x0049e3 (0xb6f bytes) 000781 1785394381.629242 write 0x017e6c thru 0x01ae8d (0x3022 bytes) 000783 1785394381.663618 read 0x00b5d9 thru 0x00bb17 (0x53f bytes) 000784 1785394381.671902 fallocate 0x02b9af thru 0x034400 (0x8a52 bytes) 000785 1785394381.696143 trunc from 0x03d267 to 0x039cbb (0x35ab bytes) 000786 1785394381.785116 mapread 0x035847 thru 0x039cbb (0x4475 bytes) 000787 1785394381.799388 write 0x001bd8 thru 0x00a1d8 (0x8601 bytes) 000788 1785394381.817708 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x39cbc): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39cbc): Mirror 3: 0xd7da7297 (stale) CRC-32 checksum value for chunk [0, 0x39cbc): Mirror 1: 0xd0f8c0ca Mirror 4: 0xd0f8c0ca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39cbc: skipped 000789 1785394383.572333 trunc from 0x007c3d to 0x01598f (0xdd53 bytes) 000790 1785394383.650209 trunc from 0x010379 to 0x01bef6 (0xbb7e bytes) 000792 1785394383.660028 trunc from 0x039cbc to 0x030eb6 (0x8e05 bytes) 000793 1785394383.702596 trunc from 0x023ad9 to 0x030eb6 (0xd3de bytes) 000794 1785394383.707502 mapwrite 0x02ff6d thru 0x035761 (0x57f5 bytes) 000795 1785394383.815284 punch from 0x007c8d to 0x00a910 (0x2c84 bytes) 000796 1785394383.822952 write 0x0392a9 thru 0x03fc29 (0x6981 bytes) 000797 1785394383.824355 read 0x03e06b thru 0x03fc29 (0x1bbf bytes) 000799 1785394383.829783 trunc from 0x0065fe to 0x00e293 (0x7c96 bytes) 000800 1785394383.840413 read 0x009452 thru 0x016fe3 (0xdb92 bytes) 000801 1785394383.874873 read 0x01977e thru 0x022b1f (0x93a2 bytes) 000802 1785394383.877608 trunc from 0x01b5fc to 0x01c5fd (0x1002 bytes) 000803 1785394383.890529 mapwrite 0x00b3a3 thru 0x016c13 (0xb871 bytes) 000804 1785394383.953343 trunc from 0x03fc2a to 0x01f80c (0x2041d bytes) 000805 1785394384.013061 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f80d): Mirror 4: 0x4448c73 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f80d): Mirror 4: 0x4448c73 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f80d: skipped 000807 1785394384.531413 trunc from 0x01f80d to 0x01e903 (0xf09 bytes) 000808 1785394384.575627 MIRROR_RESYNC 2 mirrors 000810 1785394385.583918 write 0x003e69 thru 0x00e8be (0xaa56 bytes) 000812 1785394385.648061 trunc from 0x008891 to 0x00892e (0x9e bytes) 000813 1785394385.664235 trunc from 0x01e904 to 0x0395b2 (0x1acaf bytes) 000815 1785394385.712016 punch from 0x00b78f to 0x00d307 (0x1b79 bytes) 000816 1785394385.727934 mapwrite 0x01595c thru 0x01fcbc (0xa361 bytes) 000817 1785394385.808810 punch from 0x023894 to 0x023dda (0x547 bytes) 000818 1785394385.818819 mapread 0x01ed8b thru 0x02c44f (0xd6c5 bytes) 000819 1785394385.850565 trunc from 0x0395b3 to 0x01f614 (0x19f9e bytes) 000820 1785394385.880805 punch from 0x005cef to 0x00a5a5 (0x48b7 bytes) 000821 1785394385.897944 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000822 1785394386.131404 punch from 0x01c131 to 0x01f614 (0x34e4 bytes) 000824 1785394386.153893 mapread 0x01810d thru 0x01c13c (0x4030 bytes) 000825 1785394386.201425 read 0x01bf1c thru 0x01f614 (0x36f9 bytes) 000826 1785394386.202352 punch from 0x00a5e3 to 0x014e35 (0xa853 bytes) 000827 1785394386.223234 punch from 0x018e86 to 0x01920a (0x385 bytes) 000828 1785394386.237133 trunc from 0x01f615 to 0x014170 (0xb4a4 bytes) 000829 1785394386.264313 trunc from 0x00b36d to 0x010ac7 (0x575b bytes) 000831 1785394386.280055 read 0x006e02 thru 0x0087e4 (0x19e3 bytes) 000832 1785394386.324774 MIRROR_EXTEND to 2 mirrors 000833 1785394387.202336 write 0x001461 thru 0x006711 (0x52b1 bytes) 000834 1785394387.280370 mapread 0x007363 thru 0x014170 (0xce0e bytes) 000835 1785394387.338204 trunc from 0x014171 to 0x03bb89 (0x27a19 bytes) 000836 1785394387.443810 fallocate 0x0084ef thru 0x008d18 (0x82a bytes) 000837 1785394387.462783 punch from 0x035346 to 0x03bb89 (0x6844 bytes) 000838 1785394387.477070 trunc from 0x0023dc to 0x0105e4 (0xe209 bytes) 000839 1785394387.524928 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000840 1785394387.764770 write 0x03f535 thru 0x03ffff (0xacb bytes) 000841 1785394387.774890 trunc from 0x040000 to 0x0379f7 (0x8608 bytes) 000842 1785394387.818100 trunc from 0x0379f8 to 0x01d4bb (0x1a53c bytes) 000843 1785394387.851434 trunc from 0x00a3e5 to 0x015969 (0xb585 bytes) 000844 1785394387.860803 write 0x01e401 thru 0x0256d8 (0x72d8 bytes) 000847 1785394387.869176 mapread 0x01200c thru 0x015831 (0x3826 bytes) 000848 1785394387.911991 mapwrite 0x014c26 thru 0x015fee (0x13c9 bytes) 000849 1785394387.981472 mapwrite 0x00e2a4 thru 0x011c0a (0x3967 bytes) 000850 1785394388.012478 mapwrite 0x0263cb thru 0x02ca40 (0x6676 bytes) 000851 1785394388.112124 trunc from 0x02ca41 to 0x03774b (0xad0b bytes) 000852 1785394388.139319 read 0x0247a6 thru 0x02db76 (0x93d1 bytes) 000855 1785394388.157616 read 0x010356 thru 0x019cbd (0x9968 bytes) 000856 1785394388.159137 punch from 0x009b17 to 0x00e5b4 (0x4a9e bytes) 000857 1785394388.175357 trunc from 0x01ef65 to 0x021ba7 (0x2c43 bytes) 000858 1785394388.187565 read 0x009277 thru 0x018719 (0xf4a3 bytes) 000859 1785394388.222448 trunc from 0x00e2cd to 0x0137aa (0x54de bytes) 000861 1785394388.242773 write 0x037e11 thru 0x03ac01 (0x2df1 bytes) 000863 1785394388.248364 read 0x00c588 thru 0x00c8fa (0x373 bytes) 000864 1785394388.250736 punch from 0x0158be to 0x01cf85 (0x76c8 bytes) 000865 1785394388.264930 trunc from 0x001cdb to 0x003f86 (0x22ac bytes) 000867 1785394388.280011 mapread 0x01bf86 thru 0x022f36 (0x6fb1 bytes) 000868 1785394388.315005 read 0x0054a0 thru 0x0089aa (0x350b bytes) 000869 1785394388.317754 trunc from 0x03ac02 to 0x02c532 (0xe6cf bytes) 000870 1785394388.333146 trunc from 0x02c533 to 0x03552a (0x8ff8 bytes) 000871 1785394388.403030 MIRROR_EXTEND to 2 mirrors 000872 1785394389.689775 read 0x027a91 thru 0x02a026 (0x2596 bytes) 000873 1785394389.703348 write 0x028509 thru 0x037aca (0xf5c2 bytes) 000874 1785394389.773764 fallocate 0x00b921 thru 0x011cfc (0x63dc bytes) 000875 1785394389.783060 fallocate 0x00e360 thru 0x013da4 (0x5a45 bytes) 000876 1785394389.791767 write 0x01de8d thru 0x029ffa (0xc16e bytes) 000877 1785394389.810109 fallocate 0x024b33 thru 0x028f97 (0x4465 bytes) 000878 1785394389.839878 punch from 0x00a767 to 0x017742 (0xcfdc bytes) 000880 1785394389.856600 fallocate 0x01992b thru 0x023c19 (0xa2ef bytes) 000881 1785394389.869252 trunc from 0x037acb to 0x022e7c (0x14c4e bytes) 000882 1785394389.939892 MIRROR_RESYNC 2 mirrors 000883 1785394391.388453 mapread 0x015733 thru 0x0176a1 (0x1f6f bytes) 000884 1785394391.457793 trunc from 0x0065c7 to 0x013214 (0xcc4e bytes) 000885 1785394391.547618 read 0x0194db thru 0x01c236 (0x2d5c bytes) 000887 1785394391.603065 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000888 1785394391.878024 read 0x018e4b thru 0x01b8e2 (0x2a98 bytes) 000889 1785394391.889040 fallocate 0x01bcae thru 0x027e6a (0xc1bd bytes) 000890 1785394391.911441 mapwrite 0x013013 thru 0x019a15 (0x6a03 bytes) 000891 1785394392.043561 mapwrite 0x02a674 thru 0x02ced5 (0x2862 bytes) 000892 1785394392.150525 mapwrite 0x03c412 thru 0x03cc78 (0x867 bytes) 000896 1785394392.212374 mapwrite 0x01f10f thru 0x02ba47 (0xc939 bytes) 000897 1785394392.284480 mapread 0x00f5e1 thru 0x0141a4 (0x4bc4 bytes) 000898 1785394392.287764 trunc from 0x013421 to 0x0182bd (0x4e9d bytes) 000899 1785394392.315731 read 0x0178d4 thru 0x02106c (0x9799 bytes) 000900 1785394392.334148 write 0x005e63 thru 0x00dc1a (0x7db8 bytes) 000901 1785394392.337284 punch from 0x0344a9 to 0x03cc78 (0x87d0 bytes) 000902 1785394392.348138 mapwrite 0x00ff0d thru 0x012f23 (0x3017 bytes) 000904 1785394392.382331 punch from 0x003ede to 0x0061c7 (0x22ea bytes) 000905 1785394392.401622 mapread 0x01d185 thru 0x026896 (0x9712 bytes) 000906 1785394392.404499 punch from 0x005a0d to 0x00d40c (0x7a00 bytes) 000907 1785394392.415243 fallocate 0x0055d8 thru 0x012278 (0xcca1 bytes) 000908 1785394392.431387 fallocate 0x028f2d thru 0x030770 (0x7844 bytes) 000910 1785394392.457494 write 0x032f1f thru 0x033c2d (0xd0f bytes) 000911 1785394392.459794 mapread 0x0158ea thru 0x01af0e (0x5625 bytes) 000912 1785394392.462048 read 0x035e3f thru 0x037a50 (0x1c12 bytes) 000914 1785394392.494621 fallocate 0x02fec8 thru 0x035f94 (0x60cd bytes) 000916 1785394392.506419 trunc from 0x03cc79 to 0x01ef82 (0x1dcf6 bytes) 000917 1785394392.562317 trunc from 0x01ef83 to 0x019af1 (0x5491 bytes) 000920 1785394392.595307 mapread 0x0093a5 thru 0x0126c1 (0x931d bytes) 000921 1785394392.596830 trunc from 0x00cb3c to 0x019af1 (0xcfb6 bytes) 000922 1785394392.612556 write 0x0175a7 thru 0x0261c2 (0xec1c bytes) 000923 1785394392.627606 punch from 0x001383 to 0x009b88 (0x8806 bytes) 000924 1785394392.633667 punch from 0x01152a to 0x021175 (0xfc4c bytes) 000925 1785394392.721571 trunc from 0x015d37 to 0x017833 (0x1afd bytes) 000926 1785394392.760929 trunc from 0x00344c to 0x0060e6 (0x2c9b bytes) 000927 1785394392.787385 trunc from 0x011ddc to 0x020385 (0xe5aa bytes) 000929 1785394392.803146 fallocate 0x00c7dd thru 0x00e929 (0x214d bytes) 000930 1785394392.842294 write 0x001341 thru 0x006fe4 (0x5ca4 bytes) 000931 1785394392.890152 read 0x01d460 thru 0x0261c2 (0x8d63 bytes) 000932 1785394392.942246 mapwrite 0x007352 thru 0x012b48 (0xb7f7 bytes) 000933 1785394393.001067 fallocate 0x005e37 thru 0x00ffc4 (0xa18e bytes) 000934 1785394393.024317 write 0x003e11 thru 0x0137a7 (0xf997 bytes) 000935 1785394393.030142 read 0x013afa thru 0x018bba (0x50c1 bytes) 000936 1785394393.033102 mapread 0x0255f1 thru 0x0261c2 (0xbd2 bytes) 000937 1785394393.037992 trunc from 0x0261c3 to 0x00d182 (0x19040 bytes) 000938 1785394393.114158 punch from 0x00b067 to 0x00d182 (0x211c bytes) 000939 1785394393.143294 read 0x00073d thru 0x00a28f (0x9b53 bytes) 000940 1785394393.144059 read 0x008bc1 thru 0x00d182 (0x45c2 bytes) 000941 1785394393.164606 fallocate 0x015eea thru 0x01b779 (0x5890 bytes) 000943 1785394393.185918 trunc from 0x01b77a to 0x006482 (0x152f7 bytes) 000944 1785394393.229251 trunc from 0x006483 to 0x003b22 (0x2960 bytes) 000945 1785394393.288083 read 0x0031a0 thru 0x003b22 (0x983 bytes) 000946 1785394393.289911 fallocate 0x032ef0 thru 0x03c588 (0x9699 bytes) 000947 1785394393.303746 punch from 0x001acb to 0x00265b (0xb91 bytes) 000948 1785394393.313403 fallocate 0x037034 thru 0x03f84b (0x8818 bytes) 000950 1785394393.323896 mapwrite 0x03f2dd thru 0x03ffff (0xd23 bytes) 000951 1785394393.396255 mapwrite 0x00ec52 thru 0x015e38 (0x71e7 bytes) 000953 1785394393.416108 mapread 0x011748 thru 0x019636 (0x7eef bytes) 000954 1785394393.417968 trunc from 0x008f8a to 0x00ce3a (0x3eb1 bytes) 000955 1785394393.424256 write 0x01b464 thru 0x0221f6 (0x6d93 bytes) 000956 1785394393.426621 fallocate 0x000443 thru 0x007d47 (0x7905 bytes) 000957 1785394393.432169 punch from 0x015993 to 0x01861a (0x2c88 bytes) 000958 1785394393.437505 fallocate 0x039864 thru 0x039c97 (0x434 bytes) 000959 1785394393.443157 fallocate 0x00da20 thru 0x018530 (0xab11 bytes) 000960 1785394393.448145 mapwrite 0x019045 thru 0x025359 (0xc315 bytes) 000961 1785394393.467815 read 0x01346d thru 0x0176f3 (0x4287 bytes) 000962 1785394393.477774 trunc from 0x037371 to 0x03ffff (0x8c8f bytes) 000963 1785394393.482901 write 0x018e1a thru 0x026063 (0xd24a bytes) 000965 1785394393.487073 mapread 0x006598 thru 0x0163d8 (0xfe41 bytes) 000966 1785394393.489364 trunc from 0x040000 to 0x0063cf (0x39c30 bytes) 000967 1785394393.504090 mapwrite 0x0051ce thru 0x007b3c (0x296f bytes) 000968 1785394393.543045 fallocate 0x0304d3 thru 0x035f06 (0x5a34 bytes) 000969 1785394393.547854 read 0x0022c9 thru 0x00745d (0x5195 bytes) 000970 1785394393.548910 fallocate 0x0353b0 thru 0x03f7d6 (0xa427 bytes) 000971 1785394393.554070 fallocate 0x0250ad thru 0x0310d3 (0xc027 bytes) 000972 1785394393.559120 punch from 0x0068dd to 0x011c5d (0xb381 bytes) 000973 1785394393.565239 trunc from 0x0310d4 to 0x02cf02 (0x41d1 bytes) 000974 1785394393.579263 trunc from 0x024622 to 0x0251e4 (0xbc3 bytes) 000977 1785394393.586358 read 0x01ac54 thru 0x01b34f (0x6fc bytes) 000979 1785394393.598128 read 0x024d13 thru 0x025750 (0xa3e bytes) 000980 1785394393.599245 write 0x003392 thru 0x0085cb (0x523a bytes) 000981 1785394393.601134 fallocate 0x03eda4 thru 0x03f240 (0x49d bytes) 000982 1785394393.606480 mapwrite 0x0050dd thru 0x010d54 (0xbc78 bytes) 000983 1785394393.631817 trunc from 0x016404 to 0x024d30 (0xe92d bytes) 000984 1785394393.637369 read 0x01ba8e thru 0x023d7c (0x82ef bytes) 000985 1785394393.648207 mapread 0x00adf0 thru 0x0110c5 (0x62d6 bytes) 000986 1785394393.649917 punch from 0x027492 to 0x0308b3 (0x9422 bytes) 000987 1785394393.654123 write 0x02e868 thru 0x03188c (0x3025 bytes) 000988 1785394393.666125 mapread 0x027ad4 thru 0x02c142 (0x466f bytes) 000989 1785394393.694128 write 0x01f3af thru 0x02b480 (0xc0d2 bytes) 000991 1785394393.698063 trunc from 0x025ccd to 0x02a565 (0x4899 bytes) 000992 1785394393.714151 mapwrite 0x01736f thru 0x0191f5 (0x1e87 bytes) 000993 1785394393.729999 trunc from 0x03f241 to 0x034716 (0xab2a bytes) 000996 1785394393.752279 fallocate 0x036576 thru 0x038b37 (0x25c2 bytes) 000997 1785394393.757943 mapread 0x035c58 thru 0x038b37 (0x2ee0 bytes) 000998 1785394393.846464 mapwrite 0x01ba67 thru 0x01df88 (0x2522 bytes) 000999 1785394393.873043 punch from 0x0295cf to 0x031541 (0x7f73 bytes) 001000 1785394393.908372 write 0x013881 thru 0x01891f (0x509f bytes) All operations completed A-OK! PASS 70a (74s) == sanity-flr test 71: check mirror extend parent fid ==== 02:53:23 (1785394403) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x139:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (19s) == sanity-flr test 72: check append on FLR file ========== 02:53:42 (1785394422) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.258157 s, 8.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161645 s, 6.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (12s) == sanity-flr test 200a: stress test ===================== 02:53:53 (1785394433) Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre2 Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8656448 .. Extending file size to 5785312 .. Extending file size to 6271648 .. Extending file size to 7174784 .. Extending file size to 7530496 .. Extending file size to 8389792 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8483168 .. Extending file size to 8978752 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 61152 61153 61155 61156 61158 done 192.168.204.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg430-client.virtnet /mnt/lustre2 (opts:) 192.168.204.130@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg430-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (104s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 02:55:37 (1785394537) Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre2 Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2466112 .. Extending file size to 5642272 .. done Extending file size to 3349728 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4095712 .. Extending file size to 4303584 .. Extending file size to 4835264 .. Extending file size to 6489952 .. Extending file size to 6686784 .. Extending file size to 7674688 .. Extending file size to 7053088 .. Extending file size to 7227776 .. Extending file size to 7735680 .. Extending file size to 7964608 .. Extending file size to 8991488 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8278784 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8983104 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62484 62485 62488 62489 62490 done 192.168.204.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg430-client.virtnet /mnt/lustre2 (opts:) 192.168.204.130@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg430-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (89s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 02:57:06 (1785394626) Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2292 1269012 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2028 1269276 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24372 3582648 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18740 3588280 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43112 7170928 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2292 1269012 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2028 1269276 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24372 3582648 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18740 3588280 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43112 7170928 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.242071 s, 8.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.293567 s, 7.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xb6c6bdff (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg430-client.virtnet /mnt/lustre2 (opts:) PASS 200c (33s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 02:57:41 (1785394661) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0706724 s, 29.7 MB/s PASS 202 (10s) == sanity-flr test 203: mirror file preserve mirror ID === 02:57:51 (1785394671) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x8c:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x122:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x122:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x123:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x142:0x0] } PASS 203 (12s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 02:58:03 (1785394683) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.10413 s, 20.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x19f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x19f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1a0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (13s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 02:58:16 (1785394696) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.143255 s, 14.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x124:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x125:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x143:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x124:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x125:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x143:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x144:0x0] } PASS 204b (11s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 02:58:27 (1785394707) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.130753 s, 16.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1a0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1a1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1a0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1a1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1a2:0x0] } PASS 204c (37s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 02:59:04 (1785394744) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg430-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12094 s, 8.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1a1:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1a4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1a3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1a1:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1a4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1a3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1a2:0x0] } PASS 204d (62s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 03:00:07 (1785394807) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg430-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1a4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.134373 s, 15.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1a4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1a5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1a4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1a5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (54s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 03:01:01 (1785394861) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg430-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg430-server: Pool lustre.test_204f created Adding targets to pool oleg430-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg430-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.148601 s, 14.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1a5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1a6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1a5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1a6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg430-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg430-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg430-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 03:02:23 (1785394943) PASS 205a (12s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 03:02:36 (1785394956) PASS 205b (14s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 03:02:50 (1785394970) oleg430-server: Pool lustre.test_206 created Adding targets to pool oleg430-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg430-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg430-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg430-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg430-server: Pool lustre.test_206 destroyed PASS 206 (37s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 03:03:27 (1785395007) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181507 s, 5.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694712 s, 15.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x29c315fe Mirror 3: 0x29c315fe PASS 207 (16s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 03:03:43 (1785395023) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 03:03:48 (1785395028) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 03:03:52 (1785395032) Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0561026 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x22d0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xc00000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xc00000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xc00000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1920000 Read this (4 bytes): BBBB 192.168.204.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg430-client.virtnet /mnt/lustre2 (opts:) PASS 209a (54s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 03:04:46 (1785395086) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0366905 s, 223 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 03:04:56 (1785395096) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051637 s, 20.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa7 lmm_fid: [0x200000405:0xa7:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa7 lmm_fid: [0x200000405:0xa7:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611303 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 03:04:58.000000000 -0400 Modify: 2026-07-30 03:04:58.000000000 -0400 Change: 2026-07-30 03:05:00.000000000 -0400 Birth: 2026-07-30 03:04:58.000000000 -0400 PASS 210a (11s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 03:05:06 (1785395106) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632222 s, 16.6 MB/s oleg430-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg430-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg430-server: O/280000401/d14/14e: File not found by ext2_lookup PASS 210b (25s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 03:05:32 (1785395132) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.183191 s, 224 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0181734 s, 225 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 30 03:05 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xac:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 30 03:05 /mnt/lustre/f211.sanity-flr PASS 211 (12s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 03:05:43 (1785395143) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 03:05:53 (1785395153) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (12s) == sanity-flr test complete, duration 2303 sec =========== 03:06:05 (1785395165) === sanity-flr: start cleanup 03:06:07 (1785395167) === === sanity-flr: finish cleanup 03:06:20 (1785395180) ===