-----============= acceptance-small: sanity-flr ============----- Sat Sep 12 22:11:28 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg240-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 22:11:50 (1789265510) === oleg240-client.virtnet: executing check_config_client /mnt/lustre oleg240-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg240-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88abd26b9000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88abd26b9000.idle_timeout=debug disable quota as required oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 22:12:35 (1789265555) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8234] == sanity-flr test 0a: lfs mirror create with -N option == 22:12:38 (1789265558) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 22:12:56 (1789265576) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 22:13:00 (1789265580) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 22:13:03 (1789265583) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 22:13:32 (1789265612) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (5s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 22:13:36 (1789265616) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 22:13:41 (1789265621) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88abd26b9000.stats=clear osc.lustre-OST0001-osc-ffff88abd26b9000.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 22:13:52 (1789265632) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 22:14:04 (1789265644) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 22:14:14 (1789265654) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 22:14:18 (1789265658) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 22:14:26 (1789265666) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (12s) == sanity-flr test 2: create components from existing files ========================================================== 22:14:38 (1789265678) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 22:14:47 (1789265687) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 22:14:56 (1789265696) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 22:15:05 (1789265705) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 22:15:15 (1789265715) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 22:15:22 (1789265722) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 1.60269 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082016 s, 12.8 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 22:15:33 (1789265733) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.17327 s, 30.3 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 22:15:40 (1789265740) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 22:15:51 (1789265751) Stopping /mnt/lustre-ost1 (opts:) on oleg240-server oleg240-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88abd26b9000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88abd26b9000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg240-server: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg240-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg240-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg240-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg240-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg240-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg240-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg240-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg240-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg240-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg240-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg240-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg240-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg240-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg240-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg240-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg240-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg240-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg240-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg240-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg240-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg240-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg240-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg240-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg240-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg240-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg240-server oleg240-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88abd26b9000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88abd26b9000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg240-server: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg240-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (133s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 22:18:05 (1789265885) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.394114 s, 18.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg240-server oleg240-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88abd26b9000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88abd26b9000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg240-server: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg240-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (40s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 22:18:45 (1789265925) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg240-server oleg240-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88abd26b9000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88abd26b9000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg240-server: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg240-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg240-server oleg240-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88abd26b9000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88abd26b9000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg240-server: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg240-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (87s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 22:20:12 (1789266012) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.63294 s, 14.4 MB/s -rw-r--r-- 1 root root 52428800 Sep 12 22:20 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.354s user 0m0.000s sys 0m0.190s Stopping /mnt/lustre-ost1 (opts:) on oleg240-server oleg240-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88abd26b9000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88abd26b9000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.812s user 0m0.000s sys 0m0.182s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg240-server: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg240-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg240-server oleg240-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88abd26b9000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88abd26b9000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.240s user 0m0.001s sys 0m0.176s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg240-server: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg240-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg240-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88abd26b9000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88abd26b9000.ost_server_uuid in FULL state after 0 sec PASS 33b (98s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 22:21:50 (1789266110) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 22:21:53 (1789266113) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 22:21:56 (1789266116) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 22:21:59 (1789266119) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 22:22:06 (1789266126) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.652301 s, 19.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 22:22:18 (1789266138) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501678 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460764 s, 22.8 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 22:22:25 (1789266145) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 22:22:34 (1789266154) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00163559 s, 626 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00165505 s, 619 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00319872 s, 320 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00200506 s, 511 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 22:22:44 (1789266164) create test files with size 15754158 .. cedadf0418d5d2c2dbdbef1b58ff8455 - 03ddbecc6f5fcf831cb939fff4120c78 - 0730a283df01189313df1660367819c7 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg240-client.virtnet /mnt/lustre (opts:) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 22:23:09 (1789266189) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 22:23:23 (1789266203) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 22:23:30 (1789266210) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103194 s, 10.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb6:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb7:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053598 s, 19.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xba:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 22:23:39 (1789266219) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0659484 s, 31.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.241654 s, 17.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.134928 s, 15.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 22:23:54 (1789266234) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 22:23:57 (1789266237) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 22:24:00 (1789266240) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278845 s, 37.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 22:24:09 (1789266249) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.278312 s, 11.3 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 22:24:20 (1789266260) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc4:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc4:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 22:24:30 (1789266270) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.468948 s, 22.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg240-client.virtnet /mnt/lustre (opts:) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 22:24:42 (1789266282) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff88abd3f3f800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.174272 s, 60.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg240-client.virtnet /mnt/lustre (opts:) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff88abc8bed800.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 22:24:58 (1789266298) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (32s) == sanity-flr test 46: Verify setstripe --copy option ==== 22:25:30 (1789266330) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (32s) == sanity-flr test 47: Verify mirror obj alloc =========== 22:26:02 (1789266362) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 22:26:04 (1789266364) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.216955 s, 14.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.093295 s, 33.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x281f0be4 (nosync,stale) Mirror 2: 0x281f0be4 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 0101f50f89950c41ce02afbbdf558dec - mirror 1 checksum: e6be7479d3006ec36ea61cc2e81dee6f - mirror 2 checksum: 0101f50f89950c41ce02afbbdf558dec - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-12T22:26:07-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-12T22:26:07-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 0101f50f89950c41ce02afbbdf558dec - mirror 1 checksum: e6be7479d3006ec36ea61cc2e81dee6f - mirror 2 checksum: e6be7479d3006ec36ea61cc2e81dee6f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-12T22:26:07-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-12T22:26:07-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } PASS 48 (9s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 22:26:13 (1789266373) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372678 s, 28.1 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 314368.. 315391: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 22:26:20 (1789266380) Failing mds1 on oleg240-server Stopping /mnt/lustre-mds1 (opts:) on oleg240-server 22:26:32 (1789266392) shut down facet: mds1 facet_host: oleg240-server facet_failover_host: oleg240-server Failover mds1 to oleg240-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-MDT0000 22:27:03 (1789266423) targets are mounted 22:27:03 (1789266423) facet_failover done oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246567 s, 42.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } PASS 50A (60s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 22:27:20 (1789266440) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0406592 s, 504 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.024281 s, 843 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 22:27:30 (1789266450) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0567502 s, 361 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0130857 s, 1.6 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 22:27:39 (1789266459) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00756639 s, 2.2 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 22:27:48 (1789266468) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0433166 s, 473 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 22:27:57 (1789266477) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.016323 s, 251 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.102143 s, 40.1 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 22:28:06 (1789266486) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 22:28:27 (1789266507) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 22:28:58 (1789266538) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 22:29:24 (1789266564) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 22:29:50 (1789266590) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 22:29:59 (1789266599) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57594 mirror split pid 57595 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7019: 57595 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7019: 57595 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57594 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57595 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 22:31:08 (1789266668) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 6670 000001 1789266670.570313 MIRROR_EXTEND to 2 mirrors skipping zero size read fallocating to largest ever: 0x396fe 000003 1789266671.040236 fallocate 0x02c540 thru 0x0396fd (0xd1be bytes) 000004 1789266671.109089 read 0x01854f thru 0x01bbac (0x365e bytes) 000005 1789266671.124513 write 0x00c70d thru 0x00ff25 (0x3819 bytes) 000007 1789266671.160701 write 0x03c04b thru 0x03e4cf (0x2485 bytes) 000008 1789266671.166733 mapread 0x03aff0 thru 0x03e4cf (0x34e0 bytes) 000009 1789266671.189575 mapwrite 0x036f40 thru 0x03b1d8 (0x4299 bytes) 000010 1789266671.288439 mapwrite 0x00d127 thru 0x017c3c (0xab16 bytes) 000011 1789266671.359623 trunc from 0x01d9f1 to 0x01f40d (0x1a1d bytes) 000012 1789266671.364986 trunc from 0x0047a6 to 0x00ae90 (0x66eb bytes) 000013 1789266671.375649 read 0x036888 thru 0x037ee5 (0x165e bytes) 000014 1789266671.380125 trunc from 0x02ddbc to 0x032c9f (0x4ee4 bytes) 000015 1789266671.385257 fallocate 0x0167c8 thru 0x01a4fb (0x3d34 bytes) 000016 1789266671.391577 write 0x022d0e thru 0x029c78 (0x6f6b bytes) 000018 1789266671.415540 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e4d0): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e4d0): Mirror 2: 0xea131ab5 (stale) CRC-32 checksum value for chunk [0, 0x3e4d0): Mirror 1: 0xe541448b Mirror 3: 0xe541448b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e4d0: skipped 000019 1789266672.628433 trunc from 0x01044e to 0x01ebe4 (0xe797 bytes) 000020 1789266672.659156 read 0x02685f thru 0x0360aa (0xf84c bytes) 000021 1789266672.667932 punch from 0x002b09 to 0x011383 (0xe87b bytes) 000022 1789266672.673345 trunc from 0x007e84 to 0x00c114 (0x4291 bytes) 000023 1789266672.681015 write 0x00b821 thru 0x013df2 (0x85d2 bytes) 000024 1789266672.695997 write 0x03f3ca thru 0x03ffff (0xc36 bytes) 000026 1789266672.697276 trunc from 0x040000 to 0x00a3b8 (0x35c47 bytes) 000027 1789266672.712691 fallocate 0x0004c6 thru 0x0071c8 (0x6d03 bytes) 000028 1789266672.717271 read 0x0087b1 thru 0x009466 (0xcb6 bytes) 000029 1789266672.724219 read 0x0030bf thru 0x00a3b8 (0x72fa bytes) 000031 1789266672.725243 mapwrite 0x02e028 thru 0x03510b (0x70e4 bytes) 000032 1789266672.797499 trunc from 0x03510c to 0x011080 (0x2408b bytes) 000034 1789266672.833518 mapread 0x00beaf thru 0x00f836 (0x3988 bytes) truncating to largest ever: 0x39730 000035 1789266672.834752 trunc from 0x011081 to 0x03972f (0x286af bytes) 000036 1789266672.864774 mapread 0x02bcdf thru 0x03972f (0xda51 bytes) 000037 1789266672.953332 mapwrite 0x0158c9 thru 0x01ea61 (0x9199 bytes) 000038 1789266673.024055 mapwrite 0x006dc3 thru 0x00e49f (0x76dd bytes) 000040 1789266673.052494 read 0x006e36 thru 0x0089e1 (0x1bac bytes) 000041 1789266673.054463 trunc from 0x0155a0 to 0x02301b (0xda7c bytes) 000042 1789266673.062293 trunc from 0x015728 to 0x02396a (0xe243 bytes) 000044 1789266673.077621 punch from 0x011389 to 0x01ea61 (0xd6d9 bytes) 000045 1789266673.086780 read 0x00f5a4 thru 0x01aea7 (0xb904 bytes) 000046 1789266673.100476 mapwrite 0x006e31 thru 0x00df33 (0x7103 bytes) 000047 1789266673.121757 trunc from 0x039730 to 0x03069c (0x9093 bytes) 000048 1789266673.138692 read 0x00de2e thru 0x0143a4 (0x6577 bytes) 000049 1789266673.139559 fallocate 0x028cdf thru 0x02c5d1 (0x38f3 bytes) 000050 1789266673.149253 trunc from 0x00d807 to 0x01cfa4 (0xf79e bytes) 000051 1789266673.154247 write 0x0213ca thru 0x02bafa (0xa731 bytes) 000052 1789266673.159510 mapwrite 0x02dfa7 thru 0x033f88 (0x5fe2 bytes) 000053 1789266673.229044 mapwrite 0x02e584 thru 0x038a81 (0xa4fe bytes) 000054 1789266673.258578 write 0x021722 thru 0x0291bd (0x7a9c bytes) 000055 1789266673.259410 trunc from 0x000b0b to 0x0060e2 (0x55d8 bytes) truncating to largest ever: 0x3a85a 000056 1789266673.264467 trunc from 0x038a82 to 0x03a859 (0x1dd8 bytes) 000057 1789266673.278871 MIRROR_RESYNC 3 mirrors 000058 1789266674.324544 fallocate 0x01eccb thru 0x027db2 (0x90e8 bytes) 000060 1789266674.407461 write 0x004659 thru 0x00bb0f (0x74b7 bytes) 000061 1789266674.426872 mapwrite 0x01cc12 thru 0x02b19c (0xe58b bytes) 000063 1789266674.499538 write 0x0143df thru 0x01f141 (0xad63 bytes) 000064 1789266674.505508 trunc from 0x032242 to 0x032bd8 (0x997 bytes) 000067 1789266674.510331 write 0x01b1f6 thru 0x021b57 (0x6962 bytes) 000068 1789266674.512885 trunc from 0x03a85a to 0x02de8d (0xc9cc bytes) 000070 1789266674.544311 punch from 0x01d7a6 to 0x021b9d (0x43f8 bytes) 000071 1789266674.561697 mapread 0x026a8e thru 0x029c87 (0x31fa bytes) 000073 1789266674.564340 mapwrite 0x03429f thru 0x03ffff (0xbd61 bytes) 000074 1789266674.648416 trunc from 0x010d5e to 0x013ae7 (0x2d8a bytes) 000076 1789266674.661986 MIRROR_RESYNC 3 mirrors 000078 1789266675.809035 mapread 0x0375fd thru 0x03ffff (0x8a03 bytes) 000079 1789266675.838397 MIRROR_RESYNC 3 mirrors 000080 1789266676.471579 MIRROR_EXTEND to 4 mirrors 000081 1789266677.285353 write 0x02cf01 thru 0x031acd (0x4bcd bytes) 000082 1789266677.329580 write 0x010f16 thru 0x019632 (0x871d bytes) 000083 1789266677.343063 fallocate 0x00408f thru 0x006c65 (0x2bd7 bytes) 000084 1789266677.352084 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3766f04c (stale) Mirror 3: 0x3766f04c (stale) Mirror 4: 0x3766f04c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2e575cc1 Mirror 5: 0x2e575cc1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000085 1789266679.031560 punch from 0x02fd90 to 0x03bea7 (0xc118 bytes) 000086 1789266679.099348 write 0x015a22 thru 0x01a4ac (0x4a8b bytes) 000087 1789266679.132682 mapread 0x02ae86 thru 0x02ee6c (0x3fe7 bytes) 000089 1789266679.192640 write 0x0164e1 thru 0x021c28 (0xb748 bytes) 000090 1789266679.200116 punch from 0x005252 to 0x010889 (0xb638 bytes) 000091 1789266679.214808 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3766f04c (stale) Mirror 3: 0x3766f04c (stale) Mirror 5: 0x2e575cc1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3766f04c (stale) Mirror 3: 0x3766f04c (stale) Mirror 5: 0x2e575cc1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000092 1789266679.799389 mapwrite 0x0281b4 thru 0x02acfa (0x2b47 bytes) 000093 1789266679.829698 mapwrite 0x026e81 thru 0x036a22 (0xfba2 bytes) 000094 1789266679.858319 read 0x03b4e1 thru 0x03ffff (0x4b1f bytes) 000096 1789266679.864966 punch from 0x02ad6a to 0x02c2ef (0x1586 bytes) 000097 1789266679.870344 mapwrite 0x011dee thru 0x01ce5a (0xb06d bytes) 000098 1789266679.897947 mapwrite 0x0103cc thru 0x0154a4 (0x50d9 bytes) 000099 1789266679.912827 punch from 0x038345 to 0x03ffff (0x7cbb bytes) 000100 1789266679.920403 mapwrite 0x003635 thru 0x012ca8 (0xf674 bytes) 000102 1789266679.962989 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3766f04c (stale) Mirror 3: 0x3766f04c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3766f04c (stale) Mirror 3: 0x3766f04c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000103 1789266680.417230 punch from 0x00e37a to 0x01cc21 (0xe8a8 bytes) 000104 1789266680.426308 mapread 0x0312d1 thru 0x03db66 (0xc896 bytes) truncating to largest ever: 0x3fb47 000105 1789266680.476293 trunc from 0x040000 to 0x03fb46 (0x4b9 bytes) 000106 1789266680.502177 punch from 0x02b386 to 0x031dae (0x6a29 bytes) 000107 1789266680.508011 mapwrite 0x022eb1 thru 0x028341 (0x5491 bytes) 000108 1789266680.527016 fallocate 0x02bdc4 thru 0x037c49 (0xbe86 bytes) 000109 1789266680.531314 write 0x03c194 thru 0x03ffff (0x3e6c bytes) 000110 1789266680.532630 fallocate 0x01a96f thru 0x026c1d (0xc2af bytes) 000111 1789266680.539427 trunc from 0x027e38 to 0x035a73 (0xdc3c bytes) 000112 1789266680.545576 punch from 0x0032ea to 0x004cbc (0x19d3 bytes) 000114 1789266680.552015 trunc from 0x02cd3f to 0x02d1f8 (0x4ba bytes) 000115 1789266680.557128 mapread 0x02019f thru 0x02a3b4 (0xa216 bytes) zero_range to largest ever: 0x40000 000116 1789266680.567240 trunc from 0x03f1cb to 0x03ffff (0xe35 bytes) 000117 1789266680.576643 read 0x00faf6 thru 0x016974 (0x6e7f bytes) 000118 1789266680.585305 mapread 0x03308e thru 0x03474d (0x16c0 bytes) 000119 1789266680.594577 fallocate 0x008165 thru 0x011cd9 (0x9b75 bytes) 000120 1789266680.599375 punch from 0x0335d5 to 0x03b653 (0x807f bytes) 000122 1789266680.608658 MIRROR_RESYNC 3 mirrors 000123 1789266681.659853 fallocate 0x03ca6d thru 0x03ffff (0x3593 bytes) 000124 1789266681.730363 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3bde1bf1 (stale) Mirror 3: 0x3bde1bf1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3bde1bf1 Mirror 4: 0x3bde1bf1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000125 1789266682.581598 mapwrite 0x01684b thru 0x0206b5 (0x9e6b bytes) 000126 1789266682.702452 punch from 0x016de2 to 0x018ab3 (0x1cd2 bytes) 000127 1789266682.712431 mapread 0x002f47 thru 0x011606 (0xe6c0 bytes) 000128 1789266682.744143 trunc from 0x003f2a to 0x00bf69 (0x8040 bytes) 000129 1789266682.750386 trunc from 0x040000 to 0x000f82 (0x3f07d bytes) 000130 1789266682.786048 mapread 0x00066d thru 0x000f82 (0x916 bytes) 000131 1789266682.792457 punch from 0x00076d to 0x000f82 (0x816 bytes) 000132 1789266682.797692 mapread 0x0004ba thru 0x000f82 (0xac9 bytes) 000133 1789266682.809661 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf83): Mirror 2: 0x41abdfdc (stale) Mirror 3: 0x41abdfdc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf83): Mirror 2: 0x41abdfdc (stale) Mirror 3: 0x41abdfdc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf83: skipped 000134 1789266683.096594 trunc from 0x000f83 to 0x00c894 (0xb912 bytes) 000135 1789266683.120007 punch from 0x003a3c to 0x0057a2 (0x1d67 bytes) 000136 1789266683.125407 read 0x00b4a0 thru 0x00c894 (0x13f5 bytes) 000137 1789266683.131154 mapwrite 0x00127c thru 0x00d57c (0xc301 bytes) 000138 1789266683.192230 write 0x03fd91 thru 0x03ffff (0x26f bytes) 000139 1789266683.199595 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3bde1bf1 (stale) Mirror 3: 0x3bde1bf1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc4e9ba5e Mirror 4: 0xc4e9ba5e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000140 1789266683.981962 trunc from 0x02a37e to 0x037e65 (0xdae8 bytes) 000141 1789266684.035140 read 0x02d3e4 thru 0x032ee9 (0x5b06 bytes) 000142 1789266684.045382 write 0x029b34 thru 0x0337d2 (0x9c9f bytes) 000143 1789266684.064129 trunc from 0x040000 to 0x0077f6 (0x38809 bytes) 000145 1789266684.121205 fallocate 0x01ba11 thru 0x02489f (0x8e8f bytes) 000146 1789266684.128152 mapwrite 0x01b0fd thru 0x020a83 (0x5987 bytes) 000147 1789266684.173903 mapread 0x013652 thru 0x01b63b (0x7fea bytes) 000148 1789266684.188090 trunc from 0x00f22a to 0x0113c7 (0x219e bytes) 000149 1789266684.195052 trunc from 0x0160f9 to 0x0208be (0xa7c6 bytes) 000150 1789266684.201120 mapread 0x0072fc thru 0x011bc5 (0xa8ca bytes) 000151 1789266684.221595 trunc from 0x0042f9 to 0x008b9c (0x48a4 bytes) 000153 1789266684.228518 punch from 0x00c34f to 0x01b268 (0xef1a bytes) 000154 1789266684.234606 punch from 0x002a02 to 0x00fe40 (0xd43f bytes) 000155 1789266684.241334 fallocate 0x00dcd8 thru 0x00eb4e (0xe77 bytes) 000156 1789266684.252263 MIRROR_RESYNC 4 mirrors 000157 1789266685.116054 punch from 0x005d5b to 0x00d6a6 (0x794c bytes) 000158 1789266685.183278 punch from 0x01e567 to 0x02489f (0x6339 bytes) 000159 1789266685.191324 trunc from 0x0248a0 to 0x003313 (0x2158c bytes) 000160 1789266685.259518 trunc from 0x0017e8 to 0x003313 (0x1b2c bytes) 000161 1789266685.271623 trunc from 0x003314 to 0x02a0b9 (0x26da6 bytes) 000163 1789266685.308067 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2a0ba): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a0ba): Mirror 2: 0x128887ba (stale) Mirror 3: 0x128887ba (stale) Mirror 4: 0x128887ba (stale) CRC-32 checksum value for chunk [0, 0x2a0ba): Mirror 1: 0x7c2b8e9e Mirror 5: 0x7c2b8e9e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a0ba: skipped 000164 1789266686.304669 write 0x00ebdc thru 0x014e25 (0x624a bytes) 000165 1789266686.378207 trunc from 0x02a0ba to 0x019069 (0x11050 bytes) 000167 1789266686.398598 punch from 0x011375 to 0x01365d (0x22e9 bytes) 000168 1789266686.412351 fallocate 0x022e31 thru 0x03169a (0xe86a bytes) 000169 1789266686.417564 fallocate 0x00e484 thru 0x014323 (0x5ea0 bytes) 000170 1789266686.422583 read 0x002dd4 thru 0x008b54 (0x5d81 bytes) 000171 1789266686.434570 MIRROR_RESYNC 5 mirrors 000173 1789266687.669737 read 0x028425 thru 0x02b2f1 (0x2ecd bytes) 000174 1789266687.677379 write 0x03e228 thru 0x03f264 (0x103d bytes) 000175 1789266687.709800 fallocate 0x0322e3 thru 0x03919c (0x6eba bytes) 000176 1789266687.720271 MIRROR_RESYNC 5 mirrors 000177 1789266689.119125 read 0x03bf95 thru 0x03f264 (0x32d0 bytes) 000178 1789266689.128176 trunc from 0x02c389 to 0x039b73 (0xd7eb bytes) 000179 1789266689.163366 fallocate 0x000130 thru 0x00e7c1 (0xe692 bytes) 000180 1789266689.169751 write 0x029ecb thru 0x033718 (0x984e bytes) 000181 1789266689.178531 read 0x0316f2 thru 0x03314b (0x1a5a bytes) 000182 1789266689.179396 trunc from 0x025935 to 0x031cd6 (0xc3a2 bytes) 000184 1789266689.199011 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f265): Mirror 3: 0x46a1254e (stale) Mirror 4: 0x46a1254e (stale) Mirror 5: 0x46a1254e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f265): Mirror 3: 0x46a1254e (stale) Mirror 4: 0x46a1254e (stale) Mirror 5: 0x46a1254e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f265: skipped 000186 1789266689.664087 read 0x00ec38 thru 0x015631 (0x69fa bytes) 000187 1789266689.676887 trunc from 0x00ab0c to 0x01647a (0xb96f bytes) 000188 1789266689.683992 fallocate 0x01b85c thru 0x023822 (0x7fc7 bytes) 000189 1789266689.690528 trunc from 0x03f265 to 0x019f6a (0x252fa bytes) 000190 1789266689.724588 read 0x004da8 thru 0x01305e (0xe2b7 bytes) 000191 1789266689.741486 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x19f6b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19f6b): Mirror 3: 0x5293da23 (stale) Mirror 4: 0x5293da23 (stale) Mirror 5: 0x5293da23 (stale) CRC-32 checksum value for chunk [0, 0x19f6b): Mirror 1: 0xb069175c Mirror 6: 0xb069175c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19f6b: skipped 000192 1789266690.689646 mapread 0x014efa thru 0x019f6a (0x5071 bytes) 000193 1789266690.746262 mapread 0x00306c thru 0x0080fa (0x508f bytes) 000194 1789266690.781912 mapread 0x004004 thru 0x00788a (0x3887 bytes) 000196 1789266690.817659 mapread 0x007a70 thru 0x00f885 (0x7e16 bytes) 000197 1789266690.834936 read 0x013a6f thru 0x014036 (0x5c8 bytes) 000198 1789266690.845688 mapread 0x014bb9 thru 0x019f6a (0x53b2 bytes) 000199 1789266690.860692 mapwrite 0x01724c thru 0x021692 (0xa447 bytes) 000200 1789266690.981138 mapread 0x018ff1 thru 0x021692 (0x86a2 bytes) 000201 1789266690.982161 mapread 0x00beab thru 0x00cbb7 (0xd0d bytes) 000202 1789266690.997041 fallocate 0x000eec thru 0x003e09 (0x2f1e bytes) 000203 1789266691.003360 trunc from 0x014576 to 0x019ed9 (0x5964 bytes) 000204 1789266691.014960 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21693): Mirror 4: 0xe0ec368f (stale) Mirror 5: 0xe0ec368f (stale) Mirror 6: 0x4c44966c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21693): Mirror 4: 0xe0ec368f (stale) Mirror 5: 0xe0ec368f (stale) Mirror 6: 0x4c44966c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21693: skipped 000205 1789266691.411504 mapwrite 0x02307e thru 0x029a5c (0x69df bytes) 000206 1789266691.484832 MIRROR_RESYNC 4 mirrors 000207 1789266692.510574 read 0x00961d thru 0x0168c6 (0xd2aa bytes) 000209 1789266692.521312 mapwrite 0x005ded thru 0x014c1c (0xee30 bytes) 000210 1789266692.627577 trunc from 0x020351 to 0x020968 (0x618 bytes) 000211 1789266692.633027 read 0x00a35a thru 0x00c80f (0x24b6 bytes) 000212 1789266692.633578 punch from 0x020f8a to 0x024294 (0x330b bytes) 000213 1789266692.639389 mapwrite 0x00076d thru 0x00869d (0x7f31 bytes) 000214 1789266692.668375 write 0x039b57 thru 0x03ffff (0x64a9 bytes) 000215 1789266692.669852 fallocate 0x03c14b thru 0x03ffff (0x3eb5 bytes) 000216 1789266692.680329 MIRROR_RESYNC 4 mirrors 000217 1789266693.707310 write 0x011b24 thru 0x019866 (0x7d43 bytes) 000218 1789266693.750225 fallocate 0x022a64 thru 0x026c18 (0x41b5 bytes) 000219 1789266693.755883 trunc from 0x0118ed to 0x01daf6 (0xc20a bytes) 000220 1789266693.778323 fallocate 0x011929 thru 0x012162 (0x83a bytes) 000221 1789266693.783110 mapwrite 0x01c6ad thru 0x01f64e (0x2fa2 bytes) 000222 1789266693.817385 trunc from 0x03c383 to 0x03ebe6 (0x2864 bytes) 000223 1789266693.823189 trunc from 0x00158a to 0x00fef8 (0xe96f bytes) 000224 1789266693.829079 write 0x003cd5 thru 0x012878 (0xeba4 bytes) 000225 1789266693.842506 trunc from 0x0271b9 to 0x02d195 (0x5fdd bytes) 000226 1789266693.848090 trunc from 0x040000 to 0x024e88 (0x1b177 bytes) 000227 1789266693.872968 mapread 0x002a86 thru 0x00f23f (0xc7ba bytes) 000228 1789266693.880297 trunc from 0x01db67 to 0x024e88 (0x7322 bytes) 000229 1789266693.885334 mapwrite 0x023c3f thru 0x0328a2 (0xec64 bytes) 000230 1789266693.948917 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x328a3): Mirror 4: 0x524f07ea (stale) Mirror 6: 0x524f07ea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x328a3): Mirror 4: 0x524f07ea (stale) Mirror 6: 0x524f07ea (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x328a3: skipped 000231 1789266694.296020 MIRROR_RESYNC 3 mirrors 000232 1789266695.359447 MIRROR_RESYNC 3 mirrors 000233 1789266695.780268 MIRROR_RESYNC 3 mirrors 000234 1789266696.229282 mapread 0x0260d3 thru 0x0273ad (0x12db bytes) 000236 1789266696.243261 punch from 0x00b1cc to 0x015a05 (0xa83a bytes) 000237 1789266696.292237 trunc from 0x0328a3 to 0x03fab2 (0xd210 bytes) 000238 1789266696.335087 MIRROR_RESYNC 3 mirrors 000239 1789266697.885158 write 0x00e576 thru 0x012912 (0x439d bytes) 000240 1789266697.936673 trunc from 0x0067e7 to 0x00c5e3 (0x5dfd bytes) 000242 1789266697.945200 read 0x00e233 thru 0x016c9f (0x8a6d bytes) 000243 1789266697.956006 fallocate 0x00970c thru 0x013daa (0xa69f bytes) 000244 1789266697.963111 mapread 0x033a62 thru 0x03682d (0x2dcc bytes) 000245 1789266697.999170 mapread 0x022c6c thru 0x023bb1 (0xf46 bytes) 000246 1789266698.024112 trunc from 0x03fab3 to 0x015e1b (0x29c97 bytes) 000247 1789266698.081282 mapwrite 0x01ccd0 thru 0x024b3a (0x7e6b bytes) 000248 1789266698.167060 mapread 0x01a746 thru 0x02121a (0x6ad5 bytes) 000249 1789266698.169868 fallocate 0x037bf0 thru 0x038d26 (0x1137 bytes) 000250 1789266698.183494 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24b3b): Mirror 6: 0xd1426547 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24b3b): Mirror 6: 0xd1426547 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24b3b: skipped 000251 1789266698.525485 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000252 1789266698.648460 mapread 0x0211f6 thru 0x024b3a (0x3945 bytes) 000253 1789266698.662833 mapwrite 0x010951 thru 0x01c6f5 (0xbda5 bytes) 000254 1789266698.687955 punch from 0x0060ff to 0x00c903 (0x6805 bytes) 000255 1789266698.693234 read 0x00275f thru 0x003ded (0x168f bytes) 000256 1789266698.700006 punch from 0x00686b to 0x00c82d (0x5fc3 bytes) 000257 1789266698.704326 fallocate 0x00b2e8 thru 0x013300 (0x8019 bytes) 000258 1789266698.712921 fallocate 0x0006a4 thru 0x009f71 (0x98ce bytes) 000259 1789266698.720124 read 0x00670a thru 0x013e48 (0xd73f bytes) 000261 1789266698.737469 mapread 0x0136e6 thru 0x021ba8 (0xe4c3 bytes) 000263 1789266698.742156 trunc from 0x019096 to 0x020950 (0x78bb bytes) 000264 1789266698.751080 punch from 0x00ec5c to 0x014982 (0x5d27 bytes) 000266 1789266698.761443 mapwrite 0x015423 thru 0x021fcd (0xcbab bytes) 000267 1789266698.888992 fallocate 0x032483 thru 0x03ffff (0xdb7d bytes) 000268 1789266698.899111 trunc from 0x024b3b to 0x00df2c (0x16c0e bytes) 000269 1789266698.960676 punch from 0x001429 to 0x00b3e9 (0x9fc1 bytes) 000270 1789266698.977485 fallocate 0x024c54 thru 0x027828 (0x2bd5 bytes) 000271 1789266698.997253 trunc from 0x01410a to 0x01e4bd (0xa3b4 bytes) 000272 1789266699.015354 mapwrite 0x028e57 thru 0x02ec01 (0x5dab bytes) 000274 1789266699.112092 read 0x02791e thru 0x02ec01 (0x72e4 bytes) 000275 1789266699.114047 mapwrite 0x0106ac thru 0x0146b7 (0x400c bytes) 000276 1789266699.165116 mapwrite 0x036dc9 thru 0x03ffff (0x9237 bytes) 000278 1789266699.261555 read 0x00ee03 thru 0x01c931 (0xdb2f bytes) 000279 1789266699.267205 trunc from 0x040000 to 0x011bba (0x2e445 bytes) 000280 1789266699.301703 write 0x018c00 thru 0x0228e2 (0x9ce3 bytes) 000282 1789266699.309008 mapwrite 0x0279db thru 0x027eec (0x512 bytes) 000283 1789266699.359479 read 0x01c0d3 thru 0x02625f (0xa18d bytes) 000284 1789266699.360428 mapread 0x004899 thru 0x00a88d (0x5ff5 bytes) 000285 1789266699.362448 punch from 0x01d960 to 0x02346a (0x5b0b bytes) 000286 1789266699.367944 trunc from 0x027eed to 0x00090c (0x275e0 bytes) 000287 1789266699.433118 read 0x0002d8 thru 0x00090c (0x635 bytes) 000288 1789266699.434519 read 0x000760 thru 0x00090c (0x1ad bytes) 000289 1789266699.435069 punch from 0x0006f7 to 0x00090c (0x216 bytes) 000291 1789266699.452431 mapwrite 0x03dc9f thru 0x03ffff (0x2361 bytes) 000292 1789266699.538336 trunc from 0x02d5db to 0x02e7c5 (0x11eb bytes) 000293 1789266699.544910 trunc from 0x0217cd to 0x0238a0 (0x20d4 bytes) 000294 1789266699.550241 fallocate 0x005afe thru 0x0083f7 (0x28fa bytes) 000295 1789266699.558780 mapwrite 0x01dafd thru 0x02430c (0x6810 bytes) 000296 1789266699.601106 mapwrite 0x0300d9 thru 0x035c50 (0x5b78 bytes) 000297 1789266699.621074 trunc from 0x040000 to 0x004a57 (0x3b5a8 bytes) 000298 1789266699.639928 write 0x02d4ee thru 0x02ee1b (0x192e bytes) 000299 1789266699.641123 punch from 0x004113 to 0x012ff1 (0xeedf bytes) 000301 1789266699.649288 write 0x012184 thru 0x01a07c (0x7ef9 bytes) 000302 1789266699.666455 punch from 0x015efe to 0x017507 (0x160a bytes) 000303 1789266699.685791 write 0x01d785 thru 0x01e2a5 (0xb21 bytes) 000305 1789266699.701272 trunc from 0x02ee1c to 0x014829 (0x1a5f2 bytes) 000306 1789266699.719968 mapwrite 0x00e2ab thru 0x011d4f (0x3aa5 bytes) 000307 1789266699.766427 fallocate 0x0376e7 thru 0x03ffff (0x8919 bytes) 000308 1789266699.774038 fallocate 0x019248 thru 0x0281fb (0xefb4 bytes) 000309 1789266699.786396 MIRROR_EXTEND to 2 mirrors 000310 1789266700.708041 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000312 1789266700.837506 mapread 0x0147ff thru 0x0188a3 (0x40a5 bytes) 000313 1789266700.859433 write 0x021fe9 thru 0x029bb4 (0x7bcc bytes) 000314 1789266700.888814 write 0x030711 thru 0x0375db (0x6ecb bytes) 000316 1789266700.898769 trunc from 0x013c21 to 0x016034 (0x2414 bytes) 000317 1789266700.903661 trunc from 0x040000 to 0x024c72 (0x1b38d bytes) 000318 1789266700.924094 trunc from 0x024c73 to 0x031ec6 (0xd254 bytes) 000319 1789266700.943864 fallocate 0x007cd5 thru 0x008179 (0x4a5 bytes) 000320 1789266700.953613 MIRROR_EXTEND to 2 mirrors 000321 1789266701.825174 mapread 0x01782a thru 0x022964 (0xb13b bytes) 000322 1789266701.863332 trunc from 0x031ec7 to 0x01b3d6 (0x16af0 bytes) 000323 1789266701.948059 trunc from 0x011285 to 0x01b3d6 (0xa152 bytes) 000324 1789266701.955899 mapwrite 0x030188 thru 0x03a8a0 (0xa719 bytes) 000325 1789266702.037281 trunc from 0x03a8a1 to 0x02a767 (0x10139 bytes) 000326 1789266702.065982 read 0x005b75 thru 0x00b399 (0x5825 bytes) 000327 1789266702.084821 read 0x012cdd thru 0x015a95 (0x2db9 bytes) 000328 1789266702.121043 mapread 0x00219a thru 0x00f5a8 (0xd40f bytes) 000329 1789266702.122082 punch from 0x015ae0 to 0x018689 (0x2baa bytes) 000330 1789266702.126851 fallocate 0x002a1b thru 0x006610 (0x3bf6 bytes) 000332 1789266702.144751 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2a768): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a768): Mirror 2: 0xcf283529 (stale) CRC-32 checksum value for chunk [0, 0x2a768): Mirror 1: 0x4474d74b Mirror 3: 0x4474d74b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a768: skipped 000334 1789266703.310897 read 0x023e0a thru 0x02a767 (0x695e bytes) 000335 1789266703.326613 write 0x0273a0 thru 0x02f707 (0x8368 bytes) 000336 1789266703.391611 mapwrite 0x02e4f9 thru 0x039d87 (0xb88f bytes) 000337 1789266703.529328 mapread 0x00a2d0 thru 0x0136e4 (0x9415 bytes) 000338 1789266703.555512 punch from 0x02f7b4 to 0x031ec9 (0x2716 bytes) 000339 1789266703.563154 trunc from 0x01c61f to 0x01dfd8 (0x19ba bytes) 000340 1789266703.568114 trunc from 0x033ffc to 0x039d87 (0x5d8c bytes) 000341 1789266703.574077 trunc from 0x039d88 to 0x020263 (0x19b24 bytes) 000342 1789266703.591444 punch from 0x012558 to 0x01f2d1 (0xcd7a bytes) 000344 1789266703.599039 mapread 0x002def thru 0x008c1f (0x5e31 bytes) 000345 1789266703.605161 write 0x020b69 thru 0x02989c (0x8d34 bytes) 000346 1789266703.613240 trunc from 0x003a06 to 0x006ba8 (0x31a3 bytes) 000347 1789266703.618518 trunc from 0x02989d to 0x01c7ee (0xd0ae bytes) 000348 1789266703.637961 fallocate 0x013f5d thru 0x01fd53 (0xbdf7 bytes) 000349 1789266703.646080 fallocate 0x03b810 thru 0x03ffff (0x47f0 bytes) 000350 1789266703.652021 write 0x020126 thru 0x02e04e (0xdf29 bytes) 000351 1789266703.654535 mapread 0x02596d thru 0x02e04e (0x86e2 bytes) 000352 1789266703.656298 mapread 0x00d781 thru 0x012d48 (0x55c8 bytes) 000353 1789266703.672339 trunc from 0x02e04f to 0x001667 (0x2c9e7 bytes) 000354 1789266703.693266 read 0x000f83 thru 0x001667 (0x6e5 bytes) 000355 1789266703.701125 trunc from 0x001668 to 0x03bda7 (0x3a740 bytes) 000357 1789266703.721203 fallocate 0x019de4 thru 0x01ed24 (0x4f41 bytes) 000358 1789266703.728316 punch from 0x02656c to 0x030711 (0xa1a6 bytes) 000359 1789266703.735070 trunc from 0x029bf9 to 0x0365d7 (0xc9df bytes) 000360 1789266703.757604 read 0x012501 thru 0x01a024 (0x7b24 bytes) 000362 1789266703.787274 punch from 0x0070b3 to 0x0162f1 (0xf23f bytes) 000363 1789266703.795496 write 0x037c24 thru 0x03e878 (0x6c55 bytes) 000364 1789266703.803471 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3e879): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e879): Mirror 2: 0xbac47e64 (stale) Mirror 3: 0xa0abc103 (stale) CRC-32 checksum value for chunk [0, 0x3e879): Mirror 1: 0x5de53480 Mirror 4: 0x5de53480 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e879: skipped 000365 1789266705.076235 mapread 0x010647 thru 0x01f215 (0xebcf bytes) 000368 1789266705.171298 fallocate 0x015057 thru 0x023b76 (0xeb20 bytes) 000370 1789266705.227936 write 0x020bb0 thru 0x02ce0d (0xc25e bytes) 000371 1789266705.253462 mapread 0x0239c8 thru 0x03141c (0xda55 bytes) 000372 1789266705.271256 trunc from 0x003423 to 0x010b2e (0xd70c bytes) 000373 1789266705.277923 trunc from 0x03aec9 to 0x03e878 (0x39b0 bytes) 000374 1789266705.285154 mapread 0x0332fc thru 0x038627 (0x532c bytes) 000375 1789266705.300452 read 0x00bf27 thru 0x016cc2 (0xad9c bytes) 000376 1789266705.319214 trunc from 0x038d6b to 0x03e878 (0x5b0e bytes) 000377 1789266705.325183 write 0x03567d thru 0x035bd0 (0x554 bytes) 000378 1789266705.326168 mapread 0x003df3 thru 0x004ebf (0x10cd bytes) 000379 1789266705.335102 write 0x01ea1e thru 0x020610 (0x1bf3 bytes) 000380 1789266705.342219 punch from 0x03b28f to 0x03e878 (0x35ea bytes) 000381 1789266705.354109 mapread 0x0174b1 thru 0x021f34 (0xaa84 bytes) 000382 1789266705.370343 read 0x01fa97 thru 0x0222f9 (0x2863 bytes) 000383 1789266705.371064 mapread 0x021cbf thru 0x0309dd (0xed1f bytes) 000384 1789266705.373089 punch from 0x00c3f0 to 0x00e010 (0x1c21 bytes) 000385 1789266705.378361 mapwrite 0x02d749 thru 0x030ae2 (0x339a bytes) 000387 1789266705.447151 write 0x00fbfd thru 0x01892e (0x8d32 bytes) 000388 1789266705.448407 fallocate 0x0306c5 thru 0x030e2d (0x769 bytes) 000389 1789266705.461191 punch from 0x032433 to 0x036d81 (0x494f bytes) 000390 1789266705.474416 mapread 0x03e1d9 thru 0x03e878 (0x6a0 bytes) 000391 1789266705.513152 read 0x002f9a thru 0x00c1bc (0x9223 bytes) 000392 1789266705.520481 fallocate 0x037a06 thru 0x03cbc4 (0x51bf bytes) 000394 1789266705.532127 write 0x031898 thru 0x0367e1 (0x4f4a bytes) 000395 1789266705.540139 trunc from 0x03e879 to 0x01954f (0x25329 bytes) 000396 1789266705.596361 mapread 0x0123ca thru 0x016ad8 (0x470f bytes) 000397 1789266705.597685 fallocate 0x00f9c1 thru 0x011c4c (0x228c bytes) 000398 1789266705.607331 write 0x034a3c thru 0x038d04 (0x42c9 bytes) 000399 1789266705.608578 punch from 0x0121e6 to 0x01f988 (0xd7a3 bytes) 000400 1789266705.645293 mapread 0x01d992 thru 0x0267b9 (0x8e28 bytes) 000401 1789266705.657049 write 0x0371be thru 0x03ffff (0x8e42 bytes) 000403 1789266705.658904 trunc from 0x01944e to 0x026f8e (0xdb41 bytes) 000404 1789266705.671473 MIRROR_RESYNC 4 mirrors 000405 1789266707.146172 trunc from 0x040000 to 0x02cd34 (0x132cb bytes) 000406 1789266707.190259 punch from 0x01cd15 to 0x02b9c8 (0xecb4 bytes) 000407 1789266707.196958 trunc from 0x01b582 to 0x024bcd (0x964c bytes) 000409 1789266707.203293 mapwrite 0x01f780 thru 0x02f75c (0xffdd bytes) 000410 1789266707.282776 write 0x00f738 thru 0x00f94a (0x213 bytes) 000411 1789266707.292279 mapread 0x00c1e4 thru 0x00caec (0x909 bytes) 000412 1789266707.308874 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f75d): Mirror 2: 0x36d89365 (stale) Mirror 4: 0x36d89365 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f75d): Mirror 2: 0x36d89365 (stale) Mirror 4: 0x36d89365 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f75d: skipped 000413 1789266707.835044 trunc from 0x02f75d to 0x02192d (0xde2f bytes) 000414 1789266707.889182 mapread 0x01813b thru 0x01c46a (0x4330 bytes) 000415 1789266707.929421 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2192e): Mirror 4: 0xf5939e0b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2192e): Mirror 4: 0xf5939e0b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2192e: skipped 000416 1789266708.362354 write 0x00497c thru 0x00ab10 (0x6195 bytes) 000418 1789266708.408422 trunc from 0x012799 to 0x015b0d (0x3375 bytes) 000419 1789266708.415340 mapwrite 0x01a1ee thru 0x021aa1 (0x78b4 bytes) 000420 1789266708.497438 read 0x01c06c thru 0x021aa1 (0x5a36 bytes) 000421 1789266708.498785 mapwrite 0x018afa thru 0x0225ff (0x9b06 bytes) 000422 1789266708.555053 trunc from 0x01a6a0 to 0x0225ff (0x7f60 bytes) 000423 1789266708.561195 read 0x02030b thru 0x0210ba (0xdb0 bytes) 000424 1789266708.570935 read 0x01c5fa thru 0x0225ff (0x6006 bytes) 000425 1789266708.579528 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x22600): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22600): Mirror 4: 0x557fa8c5 (stale) CRC-32 checksum value for chunk [0, 0x22600): Mirror 1: 0x93fff8d0 Mirror 5: 0x93fff8d0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22600: skipped 000426 1789266710.056919 write 0x032ae4 thru 0x03ffff (0xd51c bytes) 000427 1789266710.104210 trunc from 0x025c7f to 0x027659 (0x19db bytes) 000428 1789266710.112691 trunc from 0x040000 to 0x01f655 (0x209aa bytes) 000429 1789266710.139105 read 0x006ae0 thru 0x00fd08 (0x9229 bytes) 000430 1789266710.147000 mapread 0x008a13 thru 0x0098b8 (0xea6 bytes) 000431 1789266710.147921 trunc from 0x0042ca to 0x008333 (0x406a bytes) 000432 1789266710.158857 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1f656): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f656): Mirror 4: 0x693dda33 (stale) Mirror 5: 0xe2211750 (stale) CRC-32 checksum value for chunk [0, 0x1f656): Mirror 1: 0x5a557e3b Mirror 6: 0x5a557e3b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f656: skipped 000433 1789266711.295380 mapwrite 0x0066f0 thru 0x010f84 (0xa895 bytes) 000435 1789266711.401085 read 0x008a6a thru 0x0107b5 (0x7d4c bytes) 000436 1789266711.401688 write 0x00447f thru 0x007c8e (0x3810 bytes) 000437 1789266711.408259 read 0x01b7e0 thru 0x01f655 (0x3e76 bytes) 000440 1789266711.434826 fallocate 0x00dc81 thru 0x017936 (0x9cb6 bytes) 000441 1789266711.447527 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f656): Mirror 4: 0x693dda33 (stale) Mirror 6: 0x5a557e3b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f656): Mirror 4: 0x693dda33 (stale) Mirror 6: 0x5a557e3b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f656: skipped 000442 1789266711.935309 mapwrite 0x0312d9 thru 0x03e677 (0xd39f bytes) 000443 1789266712.134379 trunc from 0x0074d5 to 0x007b86 (0x6b2 bytes) 000444 1789266712.159285 punch from 0x02bf4f to 0x02d00c (0x10be bytes) 000445 1789266712.188022 read 0x0277b5 thru 0x02b6f9 (0x3f45 bytes) 000446 1789266712.231762 write 0x006bfa thru 0x015324 (0xe72b bytes) 000447 1789266712.255162 punch from 0x036700 to 0x03e677 (0x7f78 bytes) 000448 1789266712.271845 write 0x02ab41 thru 0x02e700 (0x3bc0 bytes) 000449 1789266712.273550 fallocate 0x0141bc thru 0x0235ed (0xf432 bytes) 000450 1789266712.285595 trunc from 0x03e678 to 0x02dff9 (0x1067e bytes) 000451 1789266712.316241 mapread 0x0019b5 thru 0x0030da (0x1726 bytes) 000453 1789266712.334754 trunc from 0x02dffa to 0x02c60f (0x19ea bytes) 000454 1789266712.381852 read 0x0182be thru 0x02028c (0x7fcf bytes) 000455 1789266712.406764 write 0x0164af thru 0x025c4a (0xf79c bytes) 000456 1789266712.408275 trunc from 0x005b48 to 0x00e938 (0x8df1 bytes) 000457 1789266712.450348 read 0x00264c thru 0x00876f (0x6124 bytes) 000458 1789266712.478790 punch from 0x0266e9 to 0x029e13 (0x372b bytes) 000460 1789266712.487140 read 0x02743d thru 0x029bfa (0x27be bytes) 000461 1789266712.510072 mapread 0x00e5f0 thru 0x00f3dc (0xded bytes) 000462 1789266712.513984 write 0x022070 thru 0x02c553 (0xa4e4 bytes) 000463 1789266712.518158 mapwrite 0x01a60d thru 0x027eb1 (0xd8a5 bytes) 000464 1789266712.590532 read 0x01eb14 thru 0x026808 (0x7cf5 bytes) 000465 1789266712.591253 fallocate 0x01d884 thru 0x0212b5 (0x3a32 bytes) 000466 1789266712.602114 mapwrite 0x027249 thru 0x02fa1a (0x87d2 bytes) 000468 1789266712.711660 mapread 0x00f150 thru 0x00fbd5 (0xa86 bytes) 000469 1789266712.712521 punch from 0x029b37 to 0x02c2b7 (0x2781 bytes) 000470 1789266712.727273 punch from 0x023693 to 0x02c09a (0x8a08 bytes) 000471 1789266712.748530 write 0x033cac thru 0x0362ae (0x2603 bytes) 000472 1789266712.755352 trunc from 0x02a3f4 to 0x035287 (0xae94 bytes) 000473 1789266712.786652 read 0x0285af thru 0x031fd1 (0x9a23 bytes) 000474 1789266712.817443 trunc from 0x0362af to 0x02a5c8 (0xbce6 bytes) 000475 1789266712.856178 trunc from 0x02a5c9 to 0x03953f (0xef77 bytes) 000476 1789266712.886226 write 0x026e90 thru 0x02d823 (0x6994 bytes) 000477 1789266712.902386 fallocate 0x01f7e8 thru 0x02e311 (0xeb2a bytes) 000478 1789266712.909961 trunc from 0x039540 to 0x0221a4 (0x1739b bytes) 000479 1789266712.947191 trunc from 0x0221a5 to 0x00f448 (0x12d5c bytes) 000480 1789266712.988285 trunc from 0x006d8f to 0x00f448 (0x86ba bytes) 000481 1789266712.995241 mapread 0x0099ad thru 0x00f448 (0x5a9c bytes) 000482 1789266713.100969 write 0x024c8f thru 0x02599a (0xd0c bytes) 000483 1789266713.103195 write 0x037705 thru 0x0383e4 (0xce0 bytes) 000484 1789266713.104265 punch from 0x0143ef to 0x01cc3a (0x884c bytes) 000485 1789266713.112389 read 0x01462a thru 0x023783 (0xf15a bytes) 000486 1789266713.131929 read 0x01c71e thru 0x02821a (0xbafd bytes) 000487 1789266713.133988 mapread 0x030686 thru 0x0383e4 (0x7d5f bytes) 000488 1789266713.134877 mapwrite 0x0039a6 thru 0x0059c7 (0x2022 bytes) 000489 1789266713.167464 punch from 0x034e3e to 0x0383e4 (0x35a7 bytes) 000490 1789266713.174210 mapread 0x015f58 thru 0x01cd1f (0x6dc8 bytes) 000491 1789266713.179336 write 0x02a76f thru 0x02cd11 (0x25a3 bytes) 000492 1789266713.181248 trunc from 0x01f474 to 0x0227e3 (0x3370 bytes) 000493 1789266713.191368 read 0x02ae75 thru 0x02d99a (0x2b26 bytes) 000494 1789266713.192493 read 0x02dc52 thru 0x0383e4 (0xa793 bytes) 000495 1789266713.213673 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x383e5): Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x383e5): Mirror 4: 0x18b1f065 (stale) Mirror 6: 0xa639e4e0 (stale) CRC-32 checksum value for chunk [0, 0x383e5): Mirror 1: 0x8d51371e Mirror 7: 0x8d51371e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x383e5: skipped 000496 1789266714.492702 mapread 0x01b81a thru 0x01e23a (0x2a21 bytes) 000497 1789266714.541699 trunc from 0x00de8d to 0x012cd2 (0x4e46 bytes) 000498 1789266714.640740 write 0x035963 thru 0x03ffff (0xa69d bytes) 000499 1789266714.701983 trunc from 0x03fd72 to 0x03ffff (0x28e bytes) 000500 1789266714.750125 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x7b7faf41 (stale) Mirror 6: 0x3e889e70 (stale) Mirror 7: 0x821968d9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa21edb45 Mirror 8: 0xa21edb45 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000501 1789266716.446784 punch from 0x03755b to 0x03d434 (0x5eda bytes) 000502 1789266716.538428 punch from 0x02091c to 0x022acf (0x21b4 bytes) 000503 1789266716.545768 mapread 0x019b82 thru 0x028541 (0xe9c0 bytes) 000504 1789266716.586845 punch from 0x00a49d to 0x0167ed (0xc351 bytes) 000505 1789266716.594417 write 0x01a9be thru 0x022566 (0x7ba9 bytes) 000507 1789266716.606855 punch from 0x00a465 to 0x012e4f (0x89eb bytes) 000509 1789266716.630305 MIRROR_RESYNC 5 mirrors 000510 1789266719.682133 mapread 0x03d347 thru 0x03ffff (0x2cb9 bytes) 000511 1789266719.732105 write 0x0178bc thru 0x01c986 (0x50cb bytes) 000513 1789266719.800324 read 0x03d0b4 thru 0x03ffff (0x2f4c bytes) 000514 1789266719.816799 read 0x02bb41 thru 0x035408 (0x98c8 bytes) 000515 1789266719.881860 mapwrite 0x01c75a thru 0x01c75b (0x2 bytes) 000516 1789266719.922436 trunc from 0x00dcee to 0x0190cc (0xb3df bytes) 000517 1789266719.928402 write 0x0181e2 thru 0x0273c2 (0xf1e1 bytes) 000518 1789266719.952584 read 0x008288 thru 0x00d25d (0x4fd6 bytes) 000521 1789266719.989232 trunc from 0x03a4f6 to 0x03ffff (0x5b0a bytes) 000522 1789266720.031580 write 0x000027 thru 0x0061f9 (0x61d3 bytes) 000523 1789266720.032912 trunc from 0x040000 to 0x027c15 (0x183ea bytes) 000524 1789266720.077443 trunc from 0x00a340 to 0x00d3d0 (0x3091 bytes) 000525 1789266720.099251 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x27c16): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27c16): Mirror 4: 0xfcc1ec47 (stale) Mirror 6: 0xfcc1ec47 (stale) Mirror 7: 0xfcc1ec47 (stale) Mirror 8: 0xfcc1ec47 (stale) CRC-32 checksum value for chunk [0, 0x27c16): Mirror 1: 0x2014fc77 Mirror 9: 0x2014fc77 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27c16: skipped 000526 1789266721.877922 punch from 0x015a95 to 0x01a954 (0x4ec0 bytes) 000527 1789266722.001416 trunc from 0x0273b6 to 0x027c15 (0x860 bytes) 000529 1789266722.013649 read 0x00ee4d thru 0x01d9f8 (0xebac bytes) 000531 1789266722.053446 read 0x027487 thru 0x027c15 (0x78f bytes) 000533 1789266722.088701 trunc from 0x019d85 to 0x021191 (0x740d bytes) 000534 1789266722.108893 punch from 0x016285 to 0x021c9f (0xba1b bytes) 000535 1789266722.118687 write 0x0087fb thru 0x013581 (0xad87 bytes) 000536 1789266722.147904 trunc from 0x01414b to 0x01b8c3 (0x7779 bytes) 000538 1789266722.177022 mapwrite 0x03a5da thru 0x03c715 (0x213c bytes) 000539 1789266722.290816 punch from 0x004b4b to 0x010a41 (0xbef7 bytes) 000540 1789266722.307559 read 0x00dd3e thru 0x0138b4 (0x5b77 bytes) 000541 1789266722.342887 write 0x015f10 thru 0x0226b1 (0xc7a2 bytes) 000542 1789266722.344506 read 0x031075 thru 0x035e4f (0x4ddb bytes) 000543 1789266722.360484 trunc from 0x03c716 to 0x033606 (0x910f bytes) 000544 1789266722.410789 read 0x01c934 thru 0x01ca59 (0x126 bytes) 000545 1789266722.411427 read 0x011bdc thru 0x01a2b2 (0x86d7 bytes) 000546 1789266722.412017 write 0x00e2c8 thru 0x0175fe (0x9337 bytes) 000547 1789266722.416425 mapwrite 0x00e7ff thru 0x0159e6 (0x71e8 bytes) 000548 1789266722.515618 mapread 0x0180f0 thru 0x0246fe (0xc60f bytes) 000549 1789266722.533882 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x33607): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33607): Mirror 4: 0x838aca52 (stale) Mirror 6: 0x838aca52 (stale) Mirror 7: 0x838aca52 (stale) Mirror 8: 0x838aca52 (stale) Mirror 9: 0x7512b7d5 (stale) CRC-32 checksum value for chunk [0, 0x33607): Mirror 1: 0x27370ca1 Mirror 10: 0x27370ca1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33607: skipped 000550 1789266724.369052 mapread 0x0315ab thru 0x033606 (0x205c bytes) 000551 1789266724.431633 punch from 0x00e739 to 0x015c75 (0x753d bytes) 000552 1789266724.535602 trunc from 0x033607 to 0x006403 (0x2d203 bytes) 000553 1789266724.597608 fallocate 0x028699 thru 0x03049e (0x7e06 bytes) 000554 1789266724.611581 read 0x001d81 thru 0x004b07 (0x2d87 bytes) 000555 1789266724.621543 read 0x0034f7 thru 0x006403 (0x2f0d bytes) 000556 1789266724.645261 mapread 0x005e5f thru 0x006403 (0x5a5 bytes) 000557 1789266724.647524 mapwrite 0x0229c8 thru 0x024af3 (0x212c bytes) 000558 1789266724.722887 read 0x01b533 thru 0x024af3 (0x95c1 bytes) 000560 1789266724.727788 mapwrite 0x0323c1 thru 0x03468b (0x22cb bytes) 000561 1789266724.825836 MIRROR_RESYNC 7 mirrors 000562 1789266727.608322 mapread 0x007720 thru 0x016345 (0xec26 bytes) 000563 1789266727.696215 mapwrite 0x014cb8 thru 0x01b0b0 (0x63f9 bytes) 000564 1789266727.797637 write 0x036b65 thru 0x03ffff (0x949b bytes) 000566 1789266727.801823 mapwrite 0x034d84 thru 0x03e6c6 (0x9943 bytes) 000567 1789266727.834237 fallocate 0x015da1 thru 0x0178c6 (0x1b26 bytes) 000568 1789266727.839838 mapread 0x019dbe thru 0x01ed9a (0x4fdd bytes) 000569 1789266727.853669 mapread 0x01b17a thru 0x01d05c (0x1ee3 bytes) 000570 1789266727.860772 MIRROR_RESYNC 7 mirrors 000571 1789266730.938041 trunc from 0x0300e5 to 0x03cbdb (0xcaf7 bytes) 000572 1789266730.976178 MIRROR_RESYNC 7 mirrors 000573 1789266733.846569 mapread 0x03b4e0 thru 0x03ffff (0x4b20 bytes) 000574 1789266733.870011 mapread 0x039348 thru 0x03ffff (0x6cb8 bytes) 000575 1789266733.871138 write 0x03d94d thru 0x03ffff (0x26b3 bytes) 000576 1789266733.925962 punch from 0x007d5f to 0x00c8cb (0x4b6d bytes) 000577 1789266733.933585 write 0x02f0ca thru 0x03cefc (0xde33 bytes) 000578 1789266733.946946 trunc from 0x040000 to 0x0116c4 (0x2e93b bytes) 000580 1789266734.006178 read 0x0035af thru 0x00f05f (0xbab1 bytes) 000581 1789266734.031698 mapread 0x00a469 thru 0x0116c4 (0x725c bytes) 000582 1789266734.041873 mapwrite 0x03e772 thru 0x03ffff (0x188e bytes) 000583 1789266734.150184 read 0x02dd75 thru 0x030e88 (0x3114 bytes) 000584 1789266734.171660 write 0x0313a6 thru 0x039660 (0x82bb bytes) 000585 1789266734.173492 trunc from 0x01fc36 to 0x02f1b3 (0xf57e bytes) 000586 1789266734.196928 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8c1add58 (stale) Mirror 6: 0x8c1add58 (stale) Mirror 7: 0x8c1add58 (stale) Mirror 8: 0x8c1add58 (stale) Mirror 9: 0x8c1add58 (stale) Mirror 10: 0x8c1add58 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdfd2ffe8 Mirror 11: 0xdfd2ffe8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000587 1789266735.049255 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8c1add58 (stale) Mirror 6: 0x8c1add58 (stale) Mirror 7: 0x8c1add58 (stale) Mirror 8: 0x8c1add58 (stale) Mirror 9: 0x8c1add58 (stale) Mirror 10: 0x8c1add58 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdfd2ffe8 Mirror 11: 0xdfd2ffe8 Mirror 12: 0xdfd2ffe8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000588 1789266736.470887 mapread 0x0180ea thru 0x027042 (0xef59 bytes) 000589 1789266736.529183 mapwrite 0x02747a thru 0x030ab6 (0x963d bytes) 000590 1789266736.794311 trunc from 0x02c922 to 0x0343f4 (0x7ad3 bytes) 000591 1789266736.808006 trunc from 0x040000 to 0x02ecfe (0x11301 bytes) 000592 1789266736.928913 mapwrite 0x032a33 thru 0x03c0cf (0x969d bytes) 000595 1789266737.104280 write 0x00b0d4 thru 0x00eab0 (0x39dd bytes) 000596 1789266737.144118 write 0x0141d1 thru 0x019b3e (0x596e bytes) 000597 1789266737.187540 trunc from 0x03c0d0 to 0x03b47d (0xc52 bytes) 000598 1789266737.223644 mapwrite 0x007d91 thru 0x00fe4e (0x80be bytes) 000601 1789266737.299177 mapwrite 0x026cc3 thru 0x030179 (0x94b7 bytes) 000602 1789266737.323651 read 0x026271 thru 0x02ae13 (0x4ba3 bytes) 000603 1789266737.324903 write 0x00d781 thru 0x00ea77 (0x12f7 bytes) 000604 1789266737.327960 trunc from 0x03b47e to 0x003920 (0x37b5d bytes) 000605 1789266737.349870 trunc from 0x003921 to 0x027325 (0x23a05 bytes) 000606 1789266737.386263 trunc from 0x004469 to 0x00a5a9 (0x6141 bytes) 000607 1789266737.394568 trunc from 0x005e77 to 0x00c43f (0x65c9 bytes) 000608 1789266737.400760 fallocate 0x017e5b thru 0x0186b2 (0x858 bytes) 000609 1789266737.407527 mapread 0x01abf5 thru 0x021be1 (0x6fed bytes) 000610 1789266737.417652 fallocate 0x039fc7 thru 0x03e6b6 (0x46f0 bytes) 000611 1789266737.422915 punch from 0x010ce1 to 0x01aeed (0xa20d bytes) 000612 1789266737.434562 trunc from 0x027326 to 0x024f65 (0x23c0 bytes) 000613 1789266737.463004 read 0x00bc85 thru 0x00c55c (0x8d8 bytes) 000615 1789266737.490537 fallocate 0x003dec thru 0x00e509 (0xa71e bytes) 000616 1789266737.496283 punch from 0x0142e5 to 0x0216b3 (0xd3cf bytes) 000617 1789266737.502558 punch from 0x00ec90 to 0x01d966 (0xecd7 bytes) 000618 1789266737.513566 trunc from 0x00cd82 to 0x0181bb (0xb43a bytes) 000619 1789266737.533023 punch from 0x020491 to 0x022b7f (0x26ef bytes) 000621 1789266737.548597 trunc from 0x024f66 to 0x03d2d9 (0x18374 bytes) 000623 1789266737.575370 trunc from 0x03d2da to 0x0034c7 (0x39e12 bytes) 000625 1789266737.600886 write 0x007682 thru 0x012920 (0xb29f bytes) 000626 1789266737.604842 trunc from 0x0027bd to 0x010a73 (0xe2b7 bytes) 000627 1789266737.621822 write 0x00f350 thru 0x01d9bd (0xe66e bytes) 000628 1789266737.630199 read 0x015b3b thru 0x01d9bd (0x7e83 bytes) 000630 1789266737.631845 trunc from 0x01d9be to 0x01ab3c (0x2e81 bytes) 000631 1789266737.665484 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1ab3d): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ab3d): Mirror 4: 0xffca599b (stale) Mirror 6: 0xffca599b (stale) Mirror 7: 0xffca599b (stale) Mirror 8: 0xffca599b (stale) Mirror 9: 0xffca599b (stale) Mirror 10: 0xffca599b (stale) Mirror 11: 0x71c06337 (stale) Mirror 12: 0x71c06337 (stale) CRC-32 checksum value for chunk [0, 0x1ab3d): Mirror 1: 0x46119e6e Mirror 13: 0x46119e6e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ab3d: skipped 000632 1789266739.220788 trunc from 0x01ab3d to 0x022ded (0x82b1 bytes) 000633 1789266739.285250 write 0x012fe2 thru 0x014f6a (0x1f89 bytes) 000634 1789266739.296662 trunc from 0x01ee9a to 0x022ded (0x3f54 bytes) 000635 1789266739.305135 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x22dee): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22dee): Mirror 4: 0x41f242ad (stale) Mirror 6: 0x41f242ad (stale) Mirror 7: 0x41f242ad (stale) Mirror 8: 0x41f242ad (stale) Mirror 9: 0x41f242ad (stale) Mirror 10: 0x41f242ad (stale) Mirror 11: 0xc5e3900c (stale) Mirror 12: 0xc5e3900c (stale) Mirror 13: 0x55f4faee (stale) CRC-32 checksum value for chunk [0, 0x22dee): Mirror 1: 0xc403d379 Mirror 14: 0xc403d379 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22dee: skipped 000636 1789266741.160347 trunc from 0x016b2f to 0x020c3a (0xa10c bytes) 000637 1789266741.249433 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x22dee): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22dee): Mirror 4: 0x41f242ad (stale) Mirror 6: 0x41f242ad (stale) Mirror 7: 0x41f242ad (stale) Mirror 8: 0x41f242ad (stale) Mirror 9: 0x41f242ad (stale) Mirror 10: 0x41f242ad (stale) Mirror 11: 0xc5e3900c (stale) Mirror 12: 0xc5e3900c (stale) Mirror 13: 0x55f4faee (stale) Mirror 14: 0xc403d379 (stale) CRC-32 checksum value for chunk [0, 0x22dee): Mirror 1: 0xf916d0ff Mirror 15: 0xf916d0ff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22dee: skipped 000638 1789266742.966973 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x22dee): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22dee): Mirror 4: 0x41f242ad (stale) Mirror 6: 0x41f242ad (stale) Mirror 7: 0x41f242ad (stale) Mirror 8: 0x41f242ad (stale) Mirror 9: 0x41f242ad (stale) Mirror 10: 0x41f242ad (stale) Mirror 11: 0xc5e3900c (stale) Mirror 12: 0xc5e3900c (stale) Mirror 13: 0x55f4faee (stale) Mirror 14: 0xc403d379 (stale) CRC-32 checksum value for chunk [0, 0x22dee): Mirror 1: 0xf916d0ff Mirror 15: 0xf916d0ff Mirror 16: 0xf916d0ff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22dee: skipped 000639 1789266744.077469 fallocate 0x0362e5 thru 0x03d4a6 (0x71c2 bytes) 000640 1789266744.121638 write 0x02ecf3 thru 0x03c8f4 (0xdc02 bytes) 000641 1789266744.134368 MIRROR_SPLIT mirror 4 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c8f5): Mirror 4: 0xf686c667 (stale) Mirror 6: 0xf686c667 (stale) Mirror 7: 0xf686c667 (stale) Mirror 9: 0xf686c667 (stale) Mirror 10: 0xf686c667 (stale) Mirror 11: 0x1e3fd838 (stale) Mirror 12: 0x1e3fd838 (stale) Mirror 13: 0x1bb89a37 (stale) Mirror 14: 0x28f4c0dd (stale) Mirror 15: 0x3596baed (stale) Mirror 16: 0x3596baed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c8f5): Mirror 4: 0xf686c667 (stale) Mirror 6: 0xf686c667 (stale) Mirror 7: 0xf686c667 (stale) Mirror 9: 0xf686c667 (stale) Mirror 10: 0xf686c667 (stale) Mirror 11: 0x1e3fd838 (stale) Mirror 12: 0x1e3fd838 (stale) Mirror 13: 0x1bb89a37 (stale) Mirror 14: 0x28f4c0dd (stale) Mirror 15: 0x3596baed (stale) Mirror 16: 0x3596baed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c8f5: skipped 000643 1789266745.118637 mapread 0x034bd7 thru 0x03c8f4 (0x7d1e bytes) 000644 1789266745.160625 mapread 0x0256f0 thru 0x02db7c (0x848d bytes) 000645 1789266745.183774 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c8f5): Mirror 4: 0xf686c667 (stale) Mirror 6: 0xf686c667 (stale) Mirror 7: 0xf686c667 (stale) Mirror 9: 0xf686c667 (stale) Mirror 10: 0xf686c667 (stale) Mirror 11: 0x1e3fd838 (stale) Mirror 12: 0x1e3fd838 (stale) Mirror 13: 0x1bb89a37 (stale) Mirror 14: 0x28f4c0dd (stale) Mirror 16: 0x3596baed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c8f5): Mirror 4: 0xf686c667 (stale) Mirror 6: 0xf686c667 (stale) Mirror 7: 0xf686c667 (stale) Mirror 9: 0xf686c667 (stale) Mirror 10: 0xf686c667 (stale) Mirror 11: 0x1e3fd838 (stale) Mirror 12: 0x1e3fd838 (stale) Mirror 13: 0x1bb89a37 (stale) Mirror 14: 0x28f4c0dd (stale) Mirror 16: 0x3596baed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c8f5: skipped 000646 1789266745.878242 mapwrite 0x02dd15 thru 0x03d0bc (0xf3a8 bytes) 000647 1789266745.951933 write 0x03d12a thru 0x03e32b (0x1202 bytes) 000648 1789266745.954482 read 0x0379a5 thru 0x03e32b (0x6987 bytes) 000649 1789266745.955942 fallocate 0x03711b thru 0x03ffff (0x8ee5 bytes) 000650 1789266745.962940 trunc from 0x040000 to 0x0235ad (0x1ca52 bytes) 000651 1789266745.987666 trunc from 0x01c9c2 to 0x0235ad (0x6bec bytes) 000652 1789266745.995026 fallocate 0x02c93b thru 0x03792c (0xaff2 bytes) 000653 1789266746.001040 write 0x038a88 thru 0x03ffff (0x7578 bytes) 000654 1789266746.002923 mapwrite 0x02d488 thru 0x03d378 (0xfef1 bytes) 000655 1789266746.062992 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8c1add58 (stale) Mirror 7: 0x8c1add58 (stale) Mirror 9: 0x8c1add58 (stale) Mirror 10: 0x8c1add58 (stale) Mirror 11: 0xdfd2ffe8 (stale) Mirror 12: 0xdfd2ffe8 (stale) Mirror 13: 0xeb2d5a10 (stale) Mirror 14: 0x768caa67 (stale) Mirror 16: 0x5cabd6ca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8c1add58 (stale) Mirror 7: 0x8c1add58 (stale) Mirror 9: 0x8c1add58 (stale) Mirror 10: 0x8c1add58 (stale) Mirror 11: 0xdfd2ffe8 (stale) Mirror 12: 0xdfd2ffe8 (stale) Mirror 13: 0xeb2d5a10 (stale) Mirror 14: 0x768caa67 (stale) Mirror 16: 0x5cabd6ca (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000656 1789266746.659350 trunc from 0x02ae72 to 0x038b8c (0xdd1b bytes) 000657 1789266746.666844 trunc from 0x000e78 to 0x0092a8 (0x8431 bytes) 000658 1789266746.673408 punch from 0x0294bc to 0x031ebb (0x8a00 bytes) 000659 1789266746.680900 fallocate 0x00c6f4 thru 0x014804 (0x8111 bytes) 000660 1789266746.686684 trunc from 0x0083ee to 0x00e8a1 (0x64b4 bytes) 000661 1789266746.693785 write 0x030fbc thru 0x038325 (0x736a bytes) 000662 1789266746.711019 mapwrite 0x03fe6a thru 0x03ffff (0x196 bytes) 000663 1789266746.739153 write 0x023e9f thru 0x02e46c (0xa5ce bytes) 000664 1789266746.756874 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8c1add58 (stale) Mirror 7: 0x8c1add58 (stale) Mirror 9: 0x8c1add58 (stale) Mirror 10: 0x8c1add58 (stale) Mirror 11: 0xdfd2ffe8 (stale) Mirror 12: 0xdfd2ffe8 (stale) Mirror 13: 0xeb2d5a10 (stale) Mirror 14: 0x768caa67 (stale) Mirror 16: 0x5cabd6ca (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7c1b072d Mirror 17: 0x7c1b072d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000665 1789266748.011900 mapwrite 0x0007ba thru 0x00731e (0x6b65 bytes) 000666 1789266748.171655 trunc from 0x0158e6 to 0x021ce2 (0xc3fd bytes) 000667 1789266748.179567 punch from 0x029542 to 0x034e48 (0xb907 bytes) 000668 1789266748.186424 punch from 0x012ace to 0x01b40a (0x893d bytes) 000671 1789266748.194841 punch from 0x00df68 to 0x00f98b (0x1a24 bytes) 000672 1789266748.209193 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8c1add58 (stale) Mirror 7: 0x8c1add58 (stale) Mirror 9: 0x8c1add58 (stale) Mirror 10: 0x8c1add58 (stale) Mirror 11: 0xdfd2ffe8 (stale) Mirror 13: 0xeb2d5a10 (stale) Mirror 14: 0x768caa67 (stale) Mirror 16: 0x5cabd6ca (stale) Mirror 17: 0x7c1b072d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8c1add58 (stale) Mirror 7: 0x8c1add58 (stale) Mirror 9: 0x8c1add58 (stale) Mirror 10: 0x8c1add58 (stale) Mirror 11: 0xdfd2ffe8 (stale) Mirror 13: 0xeb2d5a10 (stale) Mirror 14: 0x768caa67 (stale) Mirror 16: 0x5cabd6ca (stale) Mirror 17: 0x7c1b072d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000674 1789266748.904459 punch from 0x0306da to 0x0355ac (0x4ed3 bytes) 000675 1789266748.915048 punch from 0x00b2bc to 0x014bac (0x98f1 bytes) 000676 1789266748.921903 trunc from 0x03ea0d to 0x03ffff (0x15f3 bytes) 000677 1789266748.929302 fallocate 0x01119b thru 0x017200 (0x6066 bytes) 000678 1789266748.941399 trunc from 0x040000 to 0x035c19 (0xa3e6 bytes) 000681 1789266748.983959 mapread 0x02021d thru 0x0270c3 (0x6ea7 bytes) 000682 1789266749.033175 fallocate 0x00a23e thru 0x010322 (0x60e5 bytes) 000683 1789266749.043438 mapread 0x035970 thru 0x035c19 (0x2aa bytes) 000684 1789266749.077970 punch from 0x016f27 to 0x01c6d1 (0x57ab bytes) 000686 1789266749.096860 trunc from 0x00fedb to 0x014846 (0x496c bytes) 000687 1789266749.121482 punch from 0x0161b2 to 0x01cfae (0x6dfd bytes) 000689 1789266749.153181 trunc from 0x035c1a to 0x02dd4d (0x7ecc bytes) 000690 1789266749.191227 mapwrite 0x000894 thru 0x008077 (0x77e4 bytes) 000691 1789266749.260030 mapwrite 0x033765 thru 0x03d620 (0x9ebc bytes) 000692 1789266749.337799 fallocate 0x02415c thru 0x028115 (0x3fba bytes) 000693 1789266749.358029 mapread 0x036128 thru 0x03d620 (0x74f9 bytes) 000694 1789266749.361938 mapread 0x035bf2 thru 0x036639 (0xa48 bytes) 000695 1789266749.368775 trunc from 0x03d621 to 0x03bfbb (0x1665 bytes) 000696 1789266749.403406 punch from 0x007e54 to 0x010dea (0x8f97 bytes) 000697 1789266749.413377 mapwrite 0x0395e4 thru 0x03ffff (0x6a1c bytes) 000698 1789266749.480452 punch from 0x03aeca to 0x03ffff (0x5136 bytes) 000699 1789266749.487086 read 0x014788 thru 0x01dbbc (0x9435 bytes) 000702 1789266749.504473 mapwrite 0x00149d thru 0x00ceb2 (0xba16 bytes) 000703 1789266749.536353 read 0x01bc64 thru 0x024da1 (0x913e bytes) 000704 1789266749.537155 fallocate 0x00a50b thru 0x00fcac (0x57a2 bytes) 000705 1789266749.546645 mapread 0x01012b thru 0x010d5a (0xc30 bytes) 000707 1789266749.549214 trunc from 0x0047c9 to 0x007176 (0x29ae bytes) 000708 1789266749.561194 trunc from 0x040000 to 0x036b9f (0x9460 bytes) 000709 1789266749.583006 fallocate 0x01902e thru 0x022036 (0x9009 bytes) 000710 1789266749.590179 punch from 0x0198f3 to 0x02927a (0xf988 bytes) 000711 1789266749.597453 fallocate 0x01e21c thru 0x0230d3 (0x4eb8 bytes) 000712 1789266749.604139 fallocate 0x0323fc thru 0x03bb9b (0x97a0 bytes) 000713 1789266749.610817 trunc from 0x021b04 to 0x028145 (0x6642 bytes) 000714 1789266749.616616 read 0x012f39 thru 0x0207fb (0xd8c3 bytes) 000715 1789266749.652155 MIRROR_RESYNC 10 mirrors 000716 1789266753.089781 punch from 0x00bcea to 0x00e87b (0x2b92 bytes) 000718 1789266753.135369 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x36ba0): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36ba0): Mirror 6: 0x77c4b301 (stale) Mirror 7: 0x77c4b301 (stale) Mirror 9: 0x77c4b301 (stale) Mirror 10: 0x77c4b301 (stale) Mirror 11: 0x77c4b301 (stale) Mirror 13: 0x77c4b301 (stale) Mirror 14: 0x77c4b301 (stale) Mirror 16: 0x77c4b301 (stale) Mirror 17: 0x77c4b301 (stale) CRC-32 checksum value for chunk [0, 0x36ba0): Mirror 1: 0x41bd3bc9 Mirror 18: 0x41bd3bc9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36ba0: skipped 000719 1789266754.617455 mapread 0x031af4 thru 0x036b9f (0x50ac bytes) 000720 1789266754.658108 trunc from 0x036ba0 to 0x012413 (0x2478c bytes) 000721 1789266754.736435 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x12414): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12414): Mirror 6: 0xb4fb0621 (stale) Mirror 7: 0xb4fb0621 (stale) Mirror 9: 0xb4fb0621 (stale) Mirror 10: 0xb4fb0621 (stale) Mirror 11: 0xb4fb0621 (stale) Mirror 13: 0xb4fb0621 (stale) Mirror 14: 0xb4fb0621 (stale) Mirror 16: 0xb4fb0621 (stale) Mirror 17: 0xb4fb0621 (stale) Mirror 18: 0xb3f80349 (stale) CRC-32 checksum value for chunk [0, 0x12414): Mirror 1: 0xb3f80349 Mirror 19: 0xb3f80349 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12414: skipped 000723 1789266756.306311 punch from 0x00de52 to 0x012413 (0x45c2 bytes) 000724 1789266756.375797 trunc from 0x012414 to 0x02f4c7 (0x1d0b4 bytes) 000725 1789266756.411660 mapread 0x02426d thru 0x02dad0 (0x9864 bytes) 000726 1789266756.470359 mapwrite 0x02ec2b thru 0x032410 (0x37e6 bytes) 000727 1789266756.543696 trunc from 0x0235b1 to 0x029300 (0x5d50 bytes) 000728 1789266756.570447 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32411): Mirror 6: 0x9cb6ae8c (stale) Mirror 7: 0x9cb6ae8c (stale) Mirror 9: 0x9cb6ae8c (stale) Mirror 10: 0x9cb6ae8c (stale) Mirror 11: 0x9cb6ae8c (stale) Mirror 13: 0x9cb6ae8c (stale) Mirror 14: 0x9cb6ae8c (stale) Mirror 16: 0x9cb6ae8c (stale) Mirror 17: 0x9cb6ae8c (stale) Mirror 18: 0xd5aae1d1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32411): Mirror 6: 0x9cb6ae8c (stale) Mirror 7: 0x9cb6ae8c (stale) Mirror 9: 0x9cb6ae8c (stale) Mirror 10: 0x9cb6ae8c (stale) Mirror 11: 0x9cb6ae8c (stale) Mirror 13: 0x9cb6ae8c (stale) Mirror 14: 0x9cb6ae8c (stale) Mirror 16: 0x9cb6ae8c (stale) Mirror 17: 0x9cb6ae8c (stale) Mirror 18: 0xd5aae1d1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32411: skipped 000729 1789266757.549519 mapwrite 0x025dbf thru 0x035b97 (0xfdd9 bytes) 000730 1789266757.726199 mapread 0x0225e8 thru 0x02e566 (0xbf7f bytes) 000731 1789266757.730241 read 0x012440 thru 0x01e3d1 (0xbf92 bytes) 000734 1789266757.749182 punch from 0x016f90 to 0x020752 (0x97c3 bytes) 000735 1789266757.770277 mapread 0x00ce66 thru 0x00e637 (0x17d2 bytes) 000736 1789266757.790272 trunc from 0x0344f8 to 0x035b97 (0x16a0 bytes) 000737 1789266757.804103 write 0x00c609 thru 0x016138 (0x9b30 bytes) 000739 1789266757.824440 trunc from 0x035b98 to 0x011150 (0x24a47 bytes) 000740 1789266757.850503 write 0x03cb48 thru 0x03d382 (0x83b bytes) 000741 1789266757.851900 trunc from 0x03d383 to 0x01e7fd (0x1eb85 bytes) 000742 1789266757.889784 trunc from 0x00088f to 0x00640b (0x5b7d bytes) 000743 1789266757.899043 trunc from 0x01e7fe to 0x02454d (0x5d50 bytes) 000744 1789266757.923369 punch from 0x0135b3 to 0x0179e3 (0x4431 bytes) 000745 1789266757.935963 fallocate 0x01aace thru 0x023d3d (0x9270 bytes) 000746 1789266757.955237 trunc from 0x01ce9f to 0x02454d (0x76af bytes) 000749 1789266757.977937 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2454e): Mirror 6: 0xdb73e1a9 (stale) Mirror 7: 0xdb73e1a9 (stale) Mirror 9: 0xdb73e1a9 (stale) Mirror 10: 0xdb73e1a9 (stale) Mirror 13: 0xdb73e1a9 (stale) Mirror 14: 0xdb73e1a9 (stale) Mirror 16: 0xdb73e1a9 (stale) Mirror 17: 0xdb73e1a9 (stale) Mirror 18: 0x4f1a7b6e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2454e): Mirror 6: 0xdb73e1a9 (stale) Mirror 7: 0xdb73e1a9 (stale) Mirror 9: 0xdb73e1a9 (stale) Mirror 10: 0xdb73e1a9 (stale) Mirror 13: 0xdb73e1a9 (stale) Mirror 14: 0xdb73e1a9 (stale) Mirror 16: 0xdb73e1a9 (stale) Mirror 17: 0xdb73e1a9 (stale) Mirror 18: 0x4f1a7b6e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2454e: skipped 000750 1789266758.809282 read 0x0004ae thru 0x006ffa (0x6b4d bytes) 000751 1789266758.840147 punch from 0x00734e to 0x00a078 (0x2d2b bytes) 000752 1789266758.851617 mapwrite 0x0365fc thru 0x03ffff (0x9a04 bytes) 000753 1789266759.017035 mapwrite 0x005d65 thru 0x01387f (0xdb1b bytes) 000754 1789266759.120509 read 0x002102 thru 0x011a51 (0xf950 bytes) 000755 1789266759.124656 write 0x018d88 thru 0x01de9b (0x5114 bytes) 000756 1789266759.134625 mapread 0x028ebe thru 0x02efd3 (0x6116 bytes) 000757 1789266759.147596 mapwrite 0x0303cb thru 0x03c086 (0xbcbc bytes) 000758 1789266759.196532 mapread 0x0352b9 thru 0x03ffff (0xad47 bytes) 000759 1789266759.203626 mapread 0x03599d thru 0x03afda (0x563e bytes) 000760 1789266759.215527 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x61371026 (stale) Mirror 7: 0x61371026 (stale) Mirror 9: 0x61371026 (stale) Mirror 10: 0x61371026 (stale) Mirror 13: 0x61371026 (stale) Mirror 14: 0x61371026 (stale) Mirror 16: 0x61371026 (stale) Mirror 17: 0x61371026 (stale) Mirror 18: 0x228a0bc6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x209fca3 Mirror 19: 0x209fca3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000762 1789266760.999958 write 0x035cad thru 0x03ffff (0xa353 bytes) 000763 1789266761.074679 trunc from 0x039b67 to 0x03ffff (0x6499 bytes) 000764 1789266761.109671 mapread 0x00c8aa thru 0x01448a (0x7be1 bytes) 000765 1789266761.148350 mapwrite 0x0009c5 thru 0x005070 (0x46ac bytes) 000767 1789266761.209152 trunc from 0x00db96 to 0x01d322 (0xf78d bytes) 000768 1789266761.220952 mapwrite 0x0320f0 thru 0x0352cb (0x31dc bytes) 000769 1789266761.245265 trunc from 0x040000 to 0x008f22 (0x370dd bytes) 000771 1789266761.294402 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x8f23): Mirror 6: 0xbffe78c3 (stale) Mirror 7: 0xbffe78c3 (stale) Mirror 9: 0xbffe78c3 (stale) Mirror 10: 0xbffe78c3 (stale) Mirror 13: 0xbffe78c3 (stale) Mirror 14: 0xbffe78c3 (stale) Mirror 16: 0xbffe78c3 (stale) Mirror 17: 0xbffe78c3 (stale) Mirror 19: 0xe5095b16 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8f23): Mirror 6: 0xbffe78c3 (stale) Mirror 7: 0xbffe78c3 (stale) Mirror 9: 0xbffe78c3 (stale) Mirror 10: 0xbffe78c3 (stale) Mirror 13: 0xbffe78c3 (stale) Mirror 14: 0xbffe78c3 (stale) Mirror 16: 0xbffe78c3 (stale) Mirror 17: 0xbffe78c3 (stale) Mirror 19: 0xe5095b16 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8f23: skipped 000772 1789266761.958344 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x8f23): Mirror 6: 0xbffe78c3 (stale) Mirror 7: 0xbffe78c3 (stale) Mirror 9: 0xbffe78c3 (stale) Mirror 10: 0xbffe78c3 (stale) Mirror 14: 0xbffe78c3 (stale) Mirror 16: 0xbffe78c3 (stale) Mirror 17: 0xbffe78c3 (stale) Mirror 19: 0xe5095b16 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8f23): Mirror 6: 0xbffe78c3 (stale) Mirror 7: 0xbffe78c3 (stale) Mirror 9: 0xbffe78c3 (stale) Mirror 10: 0xbffe78c3 (stale) Mirror 14: 0xbffe78c3 (stale) Mirror 16: 0xbffe78c3 (stale) Mirror 17: 0xbffe78c3 (stale) Mirror 19: 0xe5095b16 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8f23: skipped 000773 1789266762.564475 read 0x001d46 thru 0x002006 (0x2c1 bytes) 000774 1789266762.571494 read 0x000991 thru 0x008f22 (0x8592 bytes) 000775 1789266762.591526 MIRROR_RESYNC 9 mirrors 000776 1789266763.960607 punch from 0x0037d9 to 0x00473a (0xf62 bytes) 000777 1789266764.002240 read 0x0087c8 thru 0x008f22 (0x75b bytes) 000778 1789266764.006459 mapread 0x005a5c thru 0x008f22 (0x34c7 bytes) 000779 1789266764.035623 trunc from 0x008f23 to 0x027edd (0x1efbb bytes) 000780 1789266764.071276 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27ede): Mirror 6: 0xdba34845 (stale) Mirror 7: 0xdba34845 (stale) Mirror 9: 0xdba34845 (stale) Mirror 10: 0xdba34845 (stale) Mirror 14: 0xdba34845 (stale) Mirror 17: 0xdba34845 (stale) Mirror 19: 0xdba34845 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27ede): Mirror 6: 0xdba34845 (stale) Mirror 7: 0xdba34845 (stale) Mirror 9: 0xdba34845 (stale) Mirror 10: 0xdba34845 (stale) Mirror 14: 0xdba34845 (stale) Mirror 17: 0xdba34845 (stale) Mirror 19: 0xdba34845 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27ede: skipped 000781 1789266764.538282 trunc from 0x00c6ef to 0x010e6a (0x477c bytes) 000782 1789266764.549292 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27ede): Mirror 7: 0xdba34845 (stale) Mirror 9: 0xdba34845 (stale) Mirror 10: 0xdba34845 (stale) Mirror 14: 0xdba34845 (stale) Mirror 17: 0xdba34845 (stale) Mirror 19: 0xdba34845 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27ede): Mirror 7: 0xdba34845 (stale) Mirror 9: 0xdba34845 (stale) Mirror 10: 0xdba34845 (stale) Mirror 14: 0xdba34845 (stale) Mirror 17: 0xdba34845 (stale) Mirror 19: 0xdba34845 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27ede: skipped 000783 1789266765.159310 write 0x02e87d thru 0x034fa7 (0x672b bytes) 000784 1789266765.169452 write 0x03d8ff thru 0x03ffff (0x2701 bytes) 000785 1789266765.171877 trunc from 0x036af3 to 0x03ffff (0x950d bytes) 000786 1789266765.185323 read 0x01d03b thru 0x02b230 (0xe1f6 bytes) 000787 1789266765.193418 write 0x01eb67 thru 0x02ba09 (0xcea3 bytes) 000788 1789266765.195055 read 0x028994 thru 0x02a233 (0x18a0 bytes) 000789 1789266765.196201 trunc from 0x01a705 to 0x029ff8 (0xf8f4 bytes) 000790 1789266765.211148 mapread 0x02da0d thru 0x02dcb8 (0x2ac bytes) 000791 1789266765.222831 trunc from 0x040000 to 0x01e4a8 (0x21b57 bytes) 000792 1789266765.244859 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1e4a9): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e4a9): Mirror 7: 0x54d888a7 (stale) Mirror 9: 0x54d888a7 (stale) Mirror 10: 0x54d888a7 (stale) Mirror 14: 0x54d888a7 (stale) Mirror 17: 0x54d888a7 (stale) Mirror 19: 0x54d888a7 (stale) CRC-32 checksum value for chunk [0, 0x1e4a9): Mirror 1: 0x8ed02f5f Mirror 20: 0x8ed02f5f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e4a9: skipped 000793 1789266766.632925 trunc from 0x01e4a9 to 0x001fc1 (0x1c4e7 bytes) 000794 1789266766.752167 punch from 0x001690 to 0x001fc1 (0x932 bytes) 000795 1789266766.767489 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1fc2): Mirror 7: 0x255b91fe (stale) Mirror 10: 0x255b91fe (stale) Mirror 14: 0x255b91fe (stale) Mirror 17: 0x255b91fe (stale) Mirror 19: 0x255b91fe (stale) Mirror 20: 0x255b91fe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fc2): Mirror 7: 0x255b91fe (stale) Mirror 10: 0x255b91fe (stale) Mirror 14: 0x255b91fe (stale) Mirror 17: 0x255b91fe (stale) Mirror 19: 0x255b91fe (stale) Mirror 20: 0x255b91fe (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fc2: skipped 000796 1789266767.148331 write 0x01d4a8 thru 0x02d2aa (0xfe03 bytes) 000797 1789266767.156449 fallocate 0x028eba thru 0x037a37 (0xeb7e bytes) 000798 1789266767.163100 read 0x0153b9 thru 0x01ebf0 (0x9838 bytes) 000800 1789266767.177526 mapread 0x005ed2 thru 0x012d69 (0xce98 bytes) 000802 1789266767.200342 punch from 0x02106e to 0x022f84 (0x1f17 bytes) 000803 1789266767.216860 punch from 0x02a7f0 to 0x02d2aa (0x2abb bytes) 000804 1789266767.222684 mapwrite 0x0386af thru 0x03f46a (0x6dbc bytes) 000805 1789266767.269847 mapwrite 0x01e52c thru 0x01edfa (0x8cf bytes) 000806 1789266767.294254 trunc from 0x03f46b to 0x031687 (0xdde3 bytes) 000807 1789266767.316980 read 0x01cae2 thru 0x01e0e4 (0x1603 bytes) 000810 1789266767.325194 MIRROR_RESYNC 7 mirrors 000811 1789266769.279677 write 0x020154 thru 0x028e04 (0x8cb1 bytes) 000812 1789266769.370247 trunc from 0x0251d1 to 0x025f93 (0xdc3 bytes) 000813 1789266769.403444 read 0x011c96 thru 0x02123e (0xf5a9 bytes) 000814 1789266769.421526 mapwrite 0x02e6f6 thru 0x030960 (0x226b bytes) 000815 1789266769.479261 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31688): Mirror 7: 0xbc1c4bd9 (stale) Mirror 14: 0xbc1c4bd9 (stale) Mirror 17: 0xbc1c4bd9 (stale) Mirror 19: 0xbc1c4bd9 (stale) Mirror 20: 0xbc1c4bd9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31688): Mirror 7: 0xbc1c4bd9 (stale) Mirror 14: 0xbc1c4bd9 (stale) Mirror 17: 0xbc1c4bd9 (stale) Mirror 19: 0xbc1c4bd9 (stale) Mirror 20: 0xbc1c4bd9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31688: skipped 000816 1789266770.166330 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31688): Mirror 14: 0xbc1c4bd9 (stale) Mirror 17: 0xbc1c4bd9 (stale) Mirror 19: 0xbc1c4bd9 (stale) Mirror 20: 0xbc1c4bd9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31688): Mirror 14: 0xbc1c4bd9 (stale) Mirror 17: 0xbc1c4bd9 (stale) Mirror 19: 0xbc1c4bd9 (stale) Mirror 20: 0xbc1c4bd9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31688: skipped 000817 1789266770.743589 trunc from 0x027be1 to 0x027f0d (0x32d bytes) 000818 1789266770.749817 write 0x03e025 thru 0x03ffff (0x1fdb bytes) 000819 1789266770.756495 trunc from 0x040000 to 0x037b5b (0x84a4 bytes) 000820 1789266770.778879 mapwrite 0x02ba48 thru 0x035e51 (0xa40a bytes) 000821 1789266770.824180 write 0x031de0 thru 0x033483 (0x16a4 bytes) 000822 1789266770.827066 write 0x03c7cb thru 0x03ffff (0x3835 bytes) 000823 1789266770.828596 fallocate 0x02e855 thru 0x039432 (0xabde bytes) 000825 1789266770.834638 write 0x03f47d thru 0x03ffff (0xb83 bytes) 000826 1789266770.835290 punch from 0x017f05 to 0x023d39 (0xbe35 bytes) 000827 1789266770.841654 read 0x039de4 thru 0x03ffff (0x621c bytes) 000829 1789266770.849583 trunc from 0x007787 to 0x00d8fb (0x6175 bytes) 000830 1789266770.860476 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x9c1f0602 (stale) Mirror 17: 0x9c1f0602 (stale) Mirror 19: 0x9c1f0602 (stale) Mirror 20: 0x9c1f0602 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4db0ba31 Mirror 21: 0x4db0ba31 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000831 1789266772.492682 punch from 0x003a9b to 0x00a4c9 (0x6a2f bytes) 000832 1789266772.592048 write 0x00d05b thru 0x01b90c (0xe8b2 bytes) 000833 1789266772.630454 trunc from 0x0113f1 to 0x01232b (0xf3b bytes) 000834 1789266772.656935 write 0x02e91d thru 0x033a9b (0x517f bytes) 000835 1789266772.673642 mapwrite 0x00b1a0 thru 0x00eec0 (0x3d21 bytes) 000836 1789266772.703676 trunc from 0x0315d6 to 0x03ffff (0xea2a bytes) 000837 1789266772.717968 mapread 0x01a012 thru 0x01f149 (0x5138 bytes) 000838 1789266772.733885 trunc from 0x040000 to 0x000588 (0x3fa77 bytes) 000840 1789266772.762350 read 0x000402 thru 0x000588 (0x187 bytes) 000841 1789266772.768179 punch from 0x000327 to 0x000588 (0x262 bytes) 000842 1789266772.772820 mapwrite 0x0243b7 thru 0x026778 (0x23c2 bytes) 000843 1789266772.813478 read 0x01bf01 thru 0x026778 (0xa878 bytes) 000844 1789266772.825062 trunc from 0x026779 to 0x006d6e (0x1fa0a bytes) 000845 1789266772.852797 mapwrite 0x022c5f thru 0x02656b (0x390d bytes) 000846 1789266772.918653 read 0x006cf4 thru 0x015f00 (0xf20d bytes) 000847 1789266772.929649 trunc from 0x010ed0 to 0x014ddd (0x3f0e bytes) 000848 1789266772.935530 trunc from 0x02656c to 0x025d63 (0x808 bytes) 000849 1789266772.955750 mapwrite 0x01e8cf thru 0x0246d8 (0x5e0a bytes) 000851 1789266772.978436 trunc from 0x025d64 to 0x0071f4 (0x1eb6f bytes) 000853 1789266773.004156 trunc from 0x004b86 to 0x0071f4 (0x266f bytes) 000854 1789266773.014079 mapread 0x003eac thru 0x0071f4 (0x3349 bytes) 000855 1789266773.040491 mapread 0x0056c7 thru 0x0071f4 (0x1b2e bytes) 000856 1789266773.041798 punch from 0x001c46 to 0x0071f4 (0x55af bytes) 000857 1789266773.048168 read 0x002c4a thru 0x0071f4 (0x45ab bytes) 000858 1789266773.065143 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x71f5): Mirror 14: 0x7c1923ef (stale) Mirror 17: 0x7c1923ef (stale) Mirror 19: 0x7c1923ef (stale) Mirror 21: 0x7c1923ef (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x71f5): Mirror 14: 0x7c1923ef (stale) Mirror 17: 0x7c1923ef (stale) Mirror 19: 0x7c1923ef (stale) Mirror 21: 0x7c1923ef (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x71f5: skipped 000860 1789266773.530849 mapwrite 0x013987 thru 0x019743 (0x5dbd bytes) 000861 1789266773.599179 mapread 0x0175db thru 0x019743 (0x2169 bytes) 000862 1789266773.600264 trunc from 0x00f5c1 to 0x01853a (0x8f7a bytes) 000864 1789266773.622267 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19744): Mirror 14: 0x932e92f3 (stale) Mirror 17: 0x932e92f3 (stale) Mirror 21: 0x932e92f3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19744): Mirror 14: 0x932e92f3 (stale) Mirror 17: 0x932e92f3 (stale) Mirror 21: 0x932e92f3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19744: skipped 000865 1789266774.008362 fallocate 0x0286ed thru 0x0377ea (0xf0fe bytes) 000866 1789266774.016670 punch from 0x01a497 to 0x0274b6 (0xd020 bytes) 000867 1789266774.022288 fallocate 0x00ff50 thru 0x01bb65 (0xbc16 bytes) 000868 1789266774.028632 mapread 0x00f84b thru 0x01c12c (0xc8e2 bytes) 000869 1789266774.063688 fallocate 0x03f33e thru 0x03ffff (0xcc2 bytes) 000870 1789266774.070359 mapread 0x030af5 thru 0x036787 (0x5c93 bytes) 000872 1789266774.086058 read 0x03bcd6 thru 0x03ffff (0x432a bytes) 000873 1789266774.098426 trunc from 0x040000 to 0x018547 (0x27ab8 bytes) 000874 1789266774.125092 punch from 0x01549a to 0x018547 (0x30ae bytes) 000875 1789266774.131924 mapread 0x011e56 thru 0x018547 (0x66f2 bytes) 000876 1789266774.141414 mapread 0x002238 thru 0x0073ac (0x5175 bytes) 000877 1789266774.154155 mapwrite 0x0147f3 thru 0x015ebf (0x16cd bytes) 000878 1789266774.168154 trunc from 0x0032f9 to 0x00428e (0xf96 bytes) 000879 1789266774.173798 mapwrite 0x03eb99 thru 0x03ffff (0x1467 bytes) 000880 1789266774.210345 mapread 0x03862e thru 0x03cde5 (0x47b8 bytes) 000881 1789266774.211854 mapwrite 0x01c215 thru 0x0232c5 (0x70b1 bytes) 000882 1789266774.243119 read 0x008958 thru 0x010265 (0x790e bytes) 000883 1789266774.244677 mapwrite 0x00517e thru 0x00cec2 (0x7d45 bytes) 000884 1789266774.263626 punch from 0x01d657 to 0x021fcf (0x4979 bytes) 000885 1789266774.270590 mapread 0x00d330 thru 0x01936c (0xc03d bytes) 000886 1789266774.271878 fallocate 0x00b4ec thru 0x017421 (0xbf36 bytes) 000887 1789266774.278691 mapread 0x02d726 thru 0x030f40 (0x381b bytes) 000888 1789266774.290434 fallocate 0x03d1aa thru 0x03f36a (0x21c1 bytes) 000889 1789266774.296243 trunc from 0x040000 to 0x014dbc (0x2b243 bytes) 000890 1789266774.318769 trunc from 0x014dbd to 0x02307b (0xe2bf bytes) 000891 1789266774.345252 mapread 0x00a8d8 thru 0x016baa (0xc2d3 bytes) 000892 1789266774.357626 punch from 0x000e54 to 0x002307 (0x14b4 bytes) 000893 1789266774.363630 mapwrite 0x008f11 thru 0x00d3d7 (0x44c7 bytes) 000894 1789266774.383919 mapwrite 0x01921f thru 0x025dc9 (0xcbab bytes) 000895 1789266774.490840 fallocate 0x0221d3 thru 0x0295d6 (0x7404 bytes) 000896 1789266774.497338 write 0x01fbb4 thru 0x022aa6 (0x2ef3 bytes) 000897 1789266774.498875 mapread 0x01c98b thru 0x01ffe3 (0x3659 bytes) 000898 1789266774.500636 trunc from 0x006fe2 to 0x0100c9 (0x90e8 bytes) 000899 1789266774.505930 mapwrite 0x010883 thru 0x014a21 (0x419f bytes) 000900 1789266774.528605 trunc from 0x025dca to 0x024e67 (0xf62 bytes) 000901 1789266774.544187 mapwrite 0x01bd45 thru 0x01ddef (0x20ab bytes) 000903 1789266774.562135 MIRROR_RESYNC 4 mirrors 000904 1789266776.134741 punch from 0x023809 to 0x024e67 (0x165f bytes) 000905 1789266776.179426 mapread 0x004c02 thru 0x011d89 (0xd188 bytes) 000906 1789266776.225133 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x24e68): Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24e68): Mirror 14: 0xefc2f809 (stale) Mirror 17: 0xefc2f809 (stale) Mirror 21: 0xefc2f809 (stale) CRC-32 checksum value for chunk [0, 0x24e68): Mirror 1: 0x2958d894 Mirror 22: 0x2958d894 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24e68: skipped 000907 1789266777.435365 trunc from 0x024e68 to 0x000fef (0x23e78 bytes) 000908 1789266777.509800 trunc from 0x000ff0 to 0x023174 (0x22185 bytes) 000909 1789266777.545024 write 0x017cd7 thru 0x01d936 (0x5c60 bytes) 000910 1789266777.558345 mapread 0x0223e5 thru 0x023174 (0xd90 bytes) 000911 1789266777.578581 punch from 0x020588 to 0x023174 (0x2bed bytes) 000912 1789266777.587250 mapwrite 0x03b878 thru 0x03ffff (0x4788 bytes) 000913 1789266777.706557 read 0x006d5c thru 0x012940 (0xbbe5 bytes) 000914 1789266777.762507 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x5458172e (stale) Mirror 21: 0x5458172e (stale) Mirror 22: 0xd59b7c60 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x5458172e (stale) Mirror 21: 0x5458172e (stale) Mirror 22: 0xd59b7c60 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000915 1789266778.454578 read 0x03a7e2 thru 0x03ffff (0x581e bytes) 000916 1789266778.466143 punch from 0x011e45 to 0x021c3a (0xfdf6 bytes) 000917 1789266778.471304 write 0x01f43a thru 0x02264e (0x3215 bytes) 000918 1789266778.495323 mapwrite 0x012d0e thru 0x01b376 (0x8669 bytes) 000920 1789266778.572249 punch from 0x027ecb to 0x02d9f2 (0x5b28 bytes) 000921 1789266778.589895 trunc from 0x03d179 to 0x03ffff (0x2e87 bytes) 000922 1789266778.615657 fallocate 0x00b149 thru 0x01075d (0x5615 bytes) 000923 1789266778.628510 trunc from 0x00a01f to 0x018bff (0xebe1 bytes) 000924 1789266778.646619 mapread 0x02896e thru 0x02e272 (0x5905 bytes) 000926 1789266778.676685 mapread 0x01a54c thru 0x024612 (0xa0c7 bytes) 000927 1789266778.691167 mapread 0x03a86a thru 0x03ffff (0x5796 bytes) 000928 1789266778.703680 trunc from 0x040000 to 0x01ce6a (0x23195 bytes) 000929 1789266778.730974 write 0x0233fe thru 0x02ae7b (0x7a7e bytes) 000931 1789266778.735394 read 0x01a2d2 thru 0x01a364 (0x93 bytes) 000932 1789266778.736247 write 0x0350e4 thru 0x036bc0 (0x1add bytes) 000934 1789266778.745532 read 0x02cbb2 thru 0x02f5f3 (0x2a42 bytes) 000935 1789266778.766482 mapwrite 0x02c2b5 thru 0x0397ba (0xd506 bytes) 000936 1789266778.849113 trunc from 0x0397bb to 0x0023e6 (0x373d4 bytes) 000937 1789266778.868242 fallocate 0x00198b thru 0x00e4f3 (0xcb69 bytes) 000938 1789266778.877240 trunc from 0x000308 to 0x00bccc (0xb9c5 bytes) 000939 1789266778.884052 fallocate 0x000fb7 thru 0x006a49 (0x5a93 bytes) 000940 1789266778.888862 mapread 0x00008f thru 0x00e04a (0xdfbc bytes) 000942 1789266778.962868 read 0x003ab7 thru 0x00e4f3 (0xaa3d bytes) 000943 1789266778.964371 mapread 0x00c552 thru 0x00e4f3 (0x1fa2 bytes) 000944 1789266778.965389 fallocate 0x036c87 thru 0x036e12 (0x18c bytes) 000945 1789266778.970474 trunc from 0x036e13 to 0x0323f0 (0x4a22 bytes) 000946 1789266778.994132 fallocate 0x03e264 thru 0x03f393 (0x1130 bytes) 000947 1789266778.998833 read 0x01ecc3 thru 0x01fb89 (0xec7 bytes) 000948 1789266779.010494 write 0x003619 thru 0x00d95a (0xa342 bytes) 000949 1789266779.012189 mapread 0x009d54 thru 0x01167f (0x792c bytes) 000950 1789266779.013950 trunc from 0x0091a8 to 0x00ad18 (0x1b71 bytes) 000951 1789266779.034476 MIRROR_RESYNC 4 mirrors 000952 1789266780.317872 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000953 1789266780.775217 trunc from 0x0323f1 to 0x02b56d (0x6e83 bytes) 000954 1789266780.936421 read 0x003d9d thru 0x007fe6 (0x424a bytes) 000955 1789266780.962433 punch from 0x005a22 to 0x005cd7 (0x2b6 bytes) 000956 1789266780.969067 write 0x01254d thru 0x01b147 (0x8bfb bytes) 000957 1789266780.981037 mapread 0x017c51 thru 0x01f17c (0x752c bytes) 000958 1789266781.006874 fallocate 0x03d099 thru 0x03ffff (0x2f67 bytes) 000960 1789266781.017655 mapread 0x021cb0 thru 0x024c76 (0x2fc7 bytes) 000961 1789266781.022224 punch from 0x037791 to 0x03ffff (0x886f bytes) 000962 1789266781.028376 mapread 0x0173f1 thru 0x0203e9 (0x8ff9 bytes) 000963 1789266781.029495 trunc from 0x040000 to 0x0279bd (0x18642 bytes) 000964 1789266781.069486 trunc from 0x00ee0a to 0x010c68 (0x1e5f bytes) 000965 1789266781.097793 MIRROR_RESYNC 3 mirrors 000966 1789266782.158356 trunc from 0x0279be to 0x010e31 (0x16b8c bytes) 000967 1789266782.207038 write 0x0111ee thru 0x011728 (0x53b bytes) 000968 1789266782.208675 trunc from 0x000dad to 0x00ba43 (0xac97 bytes) 000969 1789266782.220326 MIRROR_RESYNC 3 mirrors 000970 1789266783.185191 trunc from 0x008c1a to 0x011728 (0x8b0f bytes) 000971 1789266783.242556 write 0x00b311 thru 0x01a956 (0xf646 bytes) 000972 1789266783.258307 write 0x02c797 thru 0x0387e6 (0xc050 bytes) 000973 1789266783.271679 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x387e7): Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x387e7): Mirror 21: 0xac6dadef (stale) Mirror 22: 0xac6dadef (stale) CRC-32 checksum value for chunk [0, 0x387e7): Mirror 1: 0x4cb427de Mirror 23: 0x4cb427de lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x387e7: skipped 000975 1789266784.641545 fallocate 0x00d46c thru 0x00e816 (0x13ab bytes) 000976 1789266784.688150 trunc from 0x014d53 to 0x022a23 (0xdcd1 bytes) 000977 1789266784.698346 write 0x021753 thru 0x030381 (0xec2f bytes) 000980 1789266784.719452 fallocate 0x01d6d1 thru 0x029377 (0xbca7 bytes) 000982 1789266784.726070 punch from 0x025ce5 to 0x030104 (0xa420 bytes) 000983 1789266784.743046 punch from 0x01b9bb to 0x0205f6 (0x4c3c bytes) 000984 1789266784.750100 punch from 0x027519 to 0x02d62c (0x6114 bytes) 000985 1789266784.755831 mapread 0x005838 thru 0x0065cf (0xd98 bytes) 000986 1789266784.779258 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x387e7): Mirror 21: 0xac6dadef (stale) Mirror 22: 0xac6dadef (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x387e7): Mirror 21: 0xac6dadef (stale) Mirror 22: 0xac6dadef (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x387e7: skipped 000987 1789266785.388777 write 0x024236 thru 0x02f3f2 (0xb1bd bytes) 000988 1789266785.415626 mapread 0x02b5bb thru 0x0387e6 (0xd22c bytes) 000989 1789266785.458485 read 0x01b25d thru 0x0212a7 (0x604b bytes) 000991 1789266785.481229 write 0x02795e thru 0x02c530 (0x4bd3 bytes) 000992 1789266785.484688 mapwrite 0x031a2e thru 0x0368b3 (0x4e86 bytes) 000993 1789266785.514234 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x387e7): Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x387e7): Mirror 21: 0xac6dadef (stale) Mirror 22: 0xac6dadef (stale) CRC-32 checksum value for chunk [0, 0x387e7): Mirror 1: 0x997f3bf9 Mirror 23: 0x997f3bf9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x387e7: skipped 000994 1789266786.677487 mapread 0x02218d thru 0x026e8e (0x4d02 bytes) 000995 1789266786.727633 fallocate 0x02858f thru 0x02974f (0x11c1 bytes) 000996 1789266786.787461 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x387e7): Mirror 22: 0xac6dadef (stale) Mirror 23: 0x997f3bf9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x387e7): Mirror 22: 0xac6dadef (stale) Mirror 23: 0x997f3bf9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x387e7: skipped 000998 1789266787.314725 trunc from 0x01f413 to 0x0235bf (0x41ad bytes) 001000 1789266787.330518 read 0x019ae3 thru 0x01afa9 (0x14c7 bytes) All operations completed A-OK! PASS 70a (126s) == sanity-flr test 71: check mirror extend parent fid ==== 22:33:14 (1789266794) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x186:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (13s) == sanity-flr test 72: check append on FLR file ========== 22:33:27 (1789266807) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x139:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.237794 s, 8.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x139:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118305 s, 8.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x139:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11a:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x139:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11a:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13b:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 22:33:37 (1789266817) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre2 Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7295296 .. Extending file size to 3066144 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8221216 .. Extending file size to 6544640 .. Extending file size to 7656832 .. Extending file size to 8556768 .. Extending file size to 8792832 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9282752 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9193632 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61402 61403 61404 61405 61407 done 192.168.202.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre2 (opts:) 192.168.202.140@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (85s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 22:35:02 (1789266902) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre2 Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7171616 .. Extending file size to 6155616 .. Extending file size to 7631680 .. Extending file size to 8055232 .. Extending file size to 8442976 .. Extending file size to 8360544 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8828960 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9050592 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9189024 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8371776 .. Waiting 63005 63007 63008 63009 63013 done 192.168.202.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre2 (opts:) 192.168.202.140@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (82s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 22:36:24 (1789266984) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2424 1268880 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1269244 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25724 3581296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17772 3589248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43496 7170544 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2424 1268880 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2060 1269244 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25724 3581296 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17772 3589248 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43496 7170544 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.116921 s, 17.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.289623 s, 7.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x215e88d3 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 22:36:43 (1789267003) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.168284 s, 12.5 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 22:36:52 (1789267012) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 22:36:59 (1789267019) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0708774 s, 29.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 22:37:07 (1789267027) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.10642 s, 19.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x152:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 22:37:16 (1789267036) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.180456 s, 11.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20e:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 22:37:50 (1789267070) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg240-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149725 s, 7.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x210:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x210:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20d:0x0] } PASS 204d (51s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 22:38:42 (1789267122) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg240-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x211:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x212:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.140645 s, 14.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x211:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x212:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x211:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x212:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (57s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 22:39:38 (1789267178) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg240-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg240-server: Pool lustre.test_204f created Adding targets to pool oleg240-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg240-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.134638 s, 15.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x153:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x153:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg240-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg240-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg240-server: Pool lustre.test_204f destroyed PASS 204f (78s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 22:40:56 (1789267256) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 22:41:05 (1789267265) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 22:41:13 (1789267273) oleg240-server: Pool lustre.test_206 created Adding targets to pool oleg240-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg240-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg240-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg240-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg240-server: Pool lustre.test_206 destroyed PASS 206 (31s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 22:41:44 (1789267304) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108208 s, 9.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284798 s, 36.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x6202c78 Mirror 3: 0x6202c78 PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 22:41:58 (1789267318) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 22:42:02 (1789267322) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 22:42:07 (1789267327) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0601178 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x23d0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x6c0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x6c0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x6c0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1500000 Read this (4 bytes): BBBB 192.168.202.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre2 (opts:) PASS 209a (41s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 22:42:48 (1789267368) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0381116 s, 215 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 22:42:57 (1789267377) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069422 s, 15.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc2 lmm_fid: [0x200000405:0xc2:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc2 lmm_fid: [0x200000405:0xc2:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15d:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611330 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-12 22:42:59.000000000 -0400 Modify: 2026-09-12 22:42:59.000000000 -0400 Change: 2026-09-12 22:43:00.000000000 -0400 Birth: 2026-09-12 22:42:58.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 22:43:06 (1789267386) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229775 s, 45.6 MB/s oleg240-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg240-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg240-server: O/280000401/d30/15e: File not found by ext2_lookup PASS 210b (25s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 22:43:31 (1789267411) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.238225 s, 172 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.019863 s, 206 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 12 22:43 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xc7:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 12 22:43 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 22:43:39 (1789267419) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 22:43:47 (1789267427) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1947 sec =========== 22:43:57 (1789267437) === sanity-flr: start cleanup 22:43:59 (1789267439) === === sanity-flr: finish cleanup 22:44:07 (1789267447) ===