-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 17:08:49 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg420-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 17:09:09 (1787864949) === oleg420-client.virtnet: executing check_config_client /mnt/lustre oleg420-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg420-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9bafd8ba1800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9bafd8ba1800.idle_timeout=debug disable quota as required oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 17:09:53 (1787864993) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8208] == sanity-flr test 0a: lfs mirror create with -N option == 17:09:56 (1787864996) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 17:10:13 (1787865013) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 17:10:16 (1787865016) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 17:10:20 (1787865020) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 17:10:45 (1787865045) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 17:10:48 (1787865048) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 17:10:51 (1787865051) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9bafd8ba1800.stats=clear osc.lustre-OST0001-osc-ffff9bafd8ba1800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 17:10:58 (1787865058) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 17:11:08 (1787865068) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 17:11:15 (1787865075) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 17:11:19 (1787865079) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 17:11:27 (1787865087) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 17:11:37 (1787865097) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 17:11:46 (1787865106) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 17:11:53 (1787865113) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 17:12:01 (1787865121) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 17:12:11 (1787865131) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 17:12:18 (1787865138) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.38483 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110553 s, 9.5 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 17:12:28 (1787865148) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.536514 s, 39.1 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 17:12:35 (1787865155) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (8s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 17:12:44 (1787865164) Stopping /mnt/lustre-ost1 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9bafd8ba1800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9bafd8ba1800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg420-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9bafd8ba1800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9bafd8ba1800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg420-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (60s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 17:13:43 (1787865223) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.297945 s, 17.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9bafd8ba1800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9bafd8ba1800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg420-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (45s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 17:14:28 (1787865268) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9bafd8ba1800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9bafd8ba1800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg420-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9bafd8ba1800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9bafd8ba1800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg420-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (75s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 17:15:43 (1787865343) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.1276 s, 16.8 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 17:15 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.011s user 0m0.000s sys 0m0.213s Stopping /mnt/lustre-ost1 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9bafd8ba1800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9bafd8ba1800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.060s user 0m0.000s sys 0m0.185s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg420-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9bafd8ba1800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9bafd8ba1800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.349s user 0m0.001s sys 0m0.175s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg420-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg420-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9bafd8ba1800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9bafd8ba1800.ost_server_uuid in FULL state after 0 sec PASS 33b (90s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 17:17:13 (1787865433) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 17:17:16 (1787865436) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 17:17:20 (1787865440) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 17:17:23 (1787865443) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 17:17:30 (1787865450) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.409264 s, 30.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 17:17:41 (1787865461) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269919 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301514 s, 34.8 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 17:17:47 (1787865467) PASS 36c (6s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 17:17:54 (1787865474) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00199313 s, 514 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00124971 s, 819 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000802868 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000825063 s, 1.2 MB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 17:18:02 (1787865482) create test files with size 15729212 .. c95b7154d7c6f77a50e4ab2fe6572d1c - f3145309a9c5c5b442866bda4b84d31e - 97ada716fdb956ba0764fc71c2dd5cdf - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg420-client.virtnet /mnt/lustre (opts:) Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 17:18:24 (1787865504) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 17:18:38 (1787865518) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 17:18:47 (1787865527) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794601 s, 13.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc5:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc6:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xa9:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301846 s, 34.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xaa:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xab:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc7:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 17:18:56 (1787865536) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.105376 s, 19.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.279852 s, 15.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.194272 s, 10.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 17:19:12 (1787865552) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 17:19:15 (1787865555) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 17:19:19 (1787865559) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746456 s, 14.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (10s) == sanity-flr test 43c: read prefer uncompressed mirror == 17:19:29 (1787865569) llite.lustre-ffff9bafc8bbc800.enable_compression=1 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (9s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 17:19:38 (1787865578) llite.lustre-ffff9bafc8bbc800.enable_compression=1 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688907 s, 15.2 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681585 s, 15.4 MB/s **verify components PASS 43d (11s) == sanity-flr test 44a: lfs mirror split check =========== 17:19:49 (1787865589) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.352193 s, 8.9 MB/s PASS 44a (12s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 17:20:01 (1787865601) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb8:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 17:20:12 (1787865612) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.35465 s, 29.6 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x8:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg420-client.virtnet /mnt/lustre (opts:) Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 17:20:25 (1787865625) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9bafea3a8800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.297415 s, 35.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg420-client.virtnet /mnt/lustre (opts:) Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9bafc41bc000.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 17:20:41 (1787865641) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (33s) == sanity-flr test 46: Verify setstripe --copy option ==== 17:21:15 (1787865675) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (31s) == sanity-flr test 47: Verify mirror obj alloc =========== 17:21:45 (1787865705) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 17:21:47 (1787865707) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.213796 s, 14.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.147879 s, 21.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x8270f795 (nosync,stale) Mirror 2: 0x8270f795 (nosync,stale) ** verify mirror 2 doesn't change original checksum: cfd541f289b82bf753663c5fe8429719 - mirror 1 checksum: 240556c8bef3c2e718e6749cb85bde13 - mirror 2 checksum: cfd541f289b82bf753663c5fe8429719 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T17:21:50-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xdf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T17:21:50-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xe0:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: cfd541f289b82bf753663c5fe8429719 - mirror 1 checksum: 240556c8bef3c2e718e6749cb85bde13 - mirror 2 checksum: 240556c8bef3c2e718e6749cb85bde13 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T17:21:50-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xdf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T17:21:50-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xe0:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 17:21:57 (1787865717) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356823 s, 29.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 17:22:04 (1787865724) Failing mds1 on oleg420-server Stopping /mnt/lustre-mds1 (opts:) on oleg420-server 17:22:17 (1787865737) shut down facet: mds1 facet_host: oleg420-server facet_failover_host: oleg420-server Failover mds1 to oleg420-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0000 17:22:39 (1787865759) targets are mounted 17:22:39 (1787865759) facet_failover done oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487264 s, 21.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xe2:0x0] } PASS 50A (51s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 17:22:55 (1787865775) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0590373 s, 347 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0290063 s, 706 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (11s) == sanity-flr test 50b: mirror rsync handles sparseness == 17:23:06 (1787865786) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0498389 s, 411 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0178977 s, 1.1 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (12s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 17:23:19 (1787865799) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0173521 s, 944 kB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 17:23:27 (1787865807) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0542085 s, 378 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 17:23:35 (1787865815) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0108152 s, 379 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0395459 s, 104 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 17:23:44 (1787865824) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 17:24:07 (1787865847) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 17:24:38 (1787865878) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 17:25:01 (1787865901) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 17:25:24 (1787865924) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 17:25:32 (1787865932) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 58832 mirror split pid 58834 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 58832 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 58834 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 17:26:42 (1787866002) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 6004 truncating to largest ever: 0x24a00 000001 1787866005.037299 trunc from 00000000 to 0x0249ff (0x24a00 bytes) 000002 1787866005.059071 trunc from 0x024a00 to 0x01b8fb (0x9104 bytes) 000003 1787866005.084402 mapread 0x0009df thru 0x00b4cd (0xaaef bytes) 000004 1787866005.123965 mapread 0x012051 thru 0x012596 (0x546 bytes) 000006 1787866005.145553 mapwrite 0x02ab04 thru 0x02b5b8 (0xab5 bytes) 000007 1787866005.224797 read 0x023293 thru 0x02b5b8 (0x8326 bytes) fallocating to largest ever: 0x289fd 000008 1787866005.228588 fallocate 0x020689 thru 0x0289fc (0x8374 bytes) 000009 1787866005.238040 write 0x02a21a thru 0x038755 (0xe53c bytes) 000010 1787866005.245858 mapwrite 0x00a5c4 thru 0x017e80 (0xd8bd bytes) 000011 1787866005.288995 mapread 0x0074ae thru 0x010b6d (0x96c0 bytes) 000012 1787866005.296210 mapwrite 0x0052e2 thru 0x014d3d (0xfa5c bytes) truncating to largest ever: 0x2be68 000013 1787866005.338251 trunc from 0x038756 to 0x02be67 (0xc8ee bytes) 000014 1787866005.359521 trunc from 0x0167c3 to 0x0242a5 (0xdae3 bytes) truncating to largest ever: 0x3d380 000015 1787866005.369068 trunc from 0x02be68 to 0x03d37f (0x11518 bytes) 000016 1787866005.394022 mapwrite 0x016adf thru 0x01a73f (0x3c61 bytes) 000017 1787866005.440598 mapwrite 0x03f8e4 thru 0x03ffff (0x71c bytes) 000018 1787866005.544770 punch from 0x02bb25 to 0x02dc0b (0x20e7 bytes) 000019 1787866005.587382 MIRROR_EXTEND to 2 mirrors zero_range to largest ever: 0x3fef5 000020 1787866006.988244 trunc from 0x03e979 to 0x03fef4 (0x157c bytes) 000021 1787866007.086985 read 0x0218fe thru 0x029a61 (0x8164 bytes) 000022 1787866007.104499 mapwrite 0x023926 thru 0x032719 (0xedf4 bytes) 000023 1787866007.149560 mapread 0x01495a thru 0x0149f0 (0x97 bytes) zero_range to largest ever: 0x40000 000024 1787866007.165047 trunc from 0x0316b4 to 0x03ffff (0xe94c bytes) 000025 1787866007.172838 trunc from 0x020ca0 to 0x02bbcf (0xaf30 bytes) 000026 1787866007.187850 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000027 1787866007.452303 fallocate 0x037129 thru 0x037e9d (0xd75 bytes) 000028 1787866007.460450 mapwrite 0x02acd4 thru 0x03902a (0xe357 bytes) 000029 1787866007.601552 trunc from 0x040000 to 0x03ff36 (0xc9 bytes) 000030 1787866007.632640 write 0x023387 thru 0x031020 (0xdc9a bytes) 000031 1787866007.646253 MIRROR_EXTEND to 2 mirrors 000032 1787866008.672117 trunc from 0x029a0a to 0x02a06c (0x663 bytes) 000033 1787866008.757796 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ff37): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ff37): Mirror 2: 0xee86e204 (stale) CRC-32 checksum value for chunk [0, 0x3ff37): Mirror 1: 0xc2a0a45d Mirror 3: 0xc2a0a45d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ff37: skipped 000035 1787866010.635038 trunc from 0x03ff37 to 0x029da1 (0x16195 bytes) 000036 1787866010.716388 fallocate 0x016478 thru 0x01d322 (0x6eab bytes) 000037 1787866010.733781 mapread 0x00d2ac thru 0x01826d (0xafc2 bytes) 000038 1787866010.782317 mapread 0x001544 thru 0x001fd7 (0xa94 bytes) 000039 1787866010.803538 write 0x032d66 thru 0x036749 (0x39e4 bytes) 000040 1787866010.816968 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3674a): Mirror 2: 0x4a6b8d4b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3674a): Mirror 2: 0x4a6b8d4b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3674a: skipped 000041 1787866011.138853 write 0x019d8d thru 0x0217eb (0x7a5f bytes) 000042 1787866011.158951 trunc from 0x03674a to 0x01d571 (0x191d8 bytes) 000043 1787866011.207072 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1d572): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d572): Mirror 2: 0xb0bc2222 (stale) CRC-32 checksum value for chunk [0, 0x1d572): Mirror 1: 0xc87b3c5d Mirror 3: 0xc87b3c5d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d572: skipped 000044 1787866012.828953 read 0x00e3d7 thru 0x01d0b3 (0xecdd bytes) 000045 1787866012.864581 punch from 0x01a530 to 0x01d571 (0x3042 bytes) 000046 1787866012.987572 mapwrite 0x038bfa thru 0x039669 (0xa70 bytes) 000047 1787866013.082753 mapread 0x027dbf thru 0x0346be (0xc900 bytes) 000048 1787866013.143109 mapread 0x0012c4 thru 0x010bfc (0xf939 bytes) 000049 1787866013.182706 trunc from 0x03966a to 0x03cb50 (0x34e7 bytes) 000050 1787866013.200239 read 0x025782 thru 0x02fb3a (0xa3b9 bytes) 000051 1787866013.202232 mapwrite 0x02ae9f thru 0x037a21 (0xcb83 bytes) 000052 1787866013.242816 mapwrite 0x03d48d thru 0x03ffff (0x2b73 bytes) 000053 1787866013.307439 punch from 0x01e54c to 0x01f895 (0x134a bytes) 000054 1787866013.313986 mapwrite 0x01e7f3 thru 0x01f493 (0xca1 bytes) 000055 1787866013.386892 MIRROR_RESYNC 3 mirrors 000056 1787866015.477708 read 0x02b3f2 thru 0x0314fa (0x6109 bytes) 000057 1787866015.501716 trunc from 0x040000 to 0x02c177 (0x13e88 bytes) 000058 1787866015.571412 MIRROR_RESYNC 3 mirrors 000059 1787866016.613259 read 0x00cba6 thru 0x015b95 (0x8ff0 bytes) 000060 1787866016.641617 trunc from 0x0087fa to 0x00b2cb (0x2ad2 bytes) 000061 1787866016.705222 fallocate 0x020523 thru 0x02cd34 (0xc812 bytes) 000062 1787866016.713229 trunc from 0x0267b6 to 0x028d2e (0x2579 bytes) 000063 1787866016.737992 MIRROR_RESYNC 3 mirrors 000064 1787866018.369293 mapwrite 0x005263 thru 0x00ed84 (0x9b22 bytes) 000065 1787866018.515002 MIRROR_RESYNC 3 mirrors 000066 1787866020.139119 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000067 1787866020.669281 trunc from 0x02c178 to 0x02585a (0x691d bytes) 000068 1787866020.801945 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2585b): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2585b): Mirror 3: 0x13fe8a20 (stale) CRC-32 checksum value for chunk [0, 0x2585b): Mirror 1: 0x13fe8a20 Mirror 4: 0x13fe8a20 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2585b: skipped 000069 1787866022.241315 write 0x0165d3 thru 0x0222df (0xbd0d bytes) 000070 1787866022.370348 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2585b): Mirror 3: 0x13fe8a20 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2585b): Mirror 3: 0x13fe8a20 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2585b: skipped 000071 1787866022.888719 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2585b): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2585b): Mirror 3: 0x13fe8a20 (stale) CRC-32 checksum value for chunk [0, 0x2585b): Mirror 1: 0x5bd97893 Mirror 4: 0x5bd97893 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2585b: skipped 000072 1787866024.073948 trunc from 0x02585b to 0x00e924 (0x16f36 bytes) 000073 1787866024.174924 read 0x00c0f3 thru 0x00e924 (0x2832 bytes) 000074 1787866024.182751 trunc from 0x00e925 to 0x033b77 (0x25253 bytes) 000075 1787866024.215809 trunc from 0x033b78 to 0x006e80 (0x2ccf7 bytes) 000076 1787866024.232428 punch from 0x00441d to 0x006e80 (0x2a64 bytes) 000077 1787866024.239393 mapwrite 0x03be01 thru 0x03ffff (0x41ff bytes) 000078 1787866024.302124 mapread 0x0064ef thru 0x00dce3 (0x77f5 bytes) 000079 1787866024.321545 fallocate 0x0033da thru 0x008ac8 (0x56ef bytes) 000080 1787866024.327130 mapread 0x010fbe thru 0x018b4a (0x7b8d bytes) 000081 1787866024.343594 mapwrite 0x00f422 thru 0x015070 (0x5c4f bytes) 000082 1787866024.373756 fallocate 0x02ed77 thru 0x02f691 (0x91b bytes) 000083 1787866024.382120 punch from 0x030ceb to 0x03d87d (0xcb93 bytes) 000084 1787866024.388926 punch from 0x016427 to 0x025df5 (0xf9cf bytes) 000085 1787866024.394653 punch from 0x03f740 to 0x03ffff (0x8c0 bytes) 000086 1787866024.402657 punch from 0x005ceb to 0x00f5a1 (0x98b7 bytes) 000087 1787866024.409649 mapread 0x02897c thru 0x03072b (0x7db0 bytes) 000088 1787866024.443337 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8b12a8dd (stale) Mirror 4: 0xd4031c17 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc9c7a963 Mirror 5: 0xc9c7a963 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000089 1787866026.101264 write 0x0370fb thru 0x03eda6 (0x7cac bytes) 000090 1787866026.195745 read 0x01ef49 thru 0x01f7ae (0x866 bytes) 000091 1787866026.210467 write 0x029b56 thru 0x031c15 (0x80c0 bytes) 000093 1787866026.230114 mapread 0x0253a2 thru 0x025652 (0x2b1 bytes) 000095 1787866026.244609 mapwrite 0x0340a2 thru 0x03b3b5 (0x7314 bytes) 000096 1787866026.331479 mapread 0x00e88b thru 0x017355 (0x8acb bytes) 000097 1787866026.360218 mapwrite 0x03f030 thru 0x03ffff (0xfd0 bytes) 000098 1787866026.387207 fallocate 0x03b5fe thru 0x03ffff (0x4a02 bytes) 000099 1787866026.398603 fallocate 0x022acf thru 0x03172f (0xec61 bytes) 000100 1787866026.414461 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8b12a8dd (stale) Mirror 4: 0xd4031c17 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8b12a8dd (stale) Mirror 4: 0xd4031c17 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000101 1787866026.987035 punch from 0x0274eb to 0x02f146 (0x7c5c bytes) 000102 1787866027.000014 punch from 0x03066e to 0x03ab71 (0xa504 bytes) 000103 1787866027.014324 fallocate 0x0358a9 thru 0x03dfb9 (0x8711 bytes) 000104 1787866027.022274 fallocate 0x00243f thru 0x01223f (0xfe01 bytes) 000105 1787866027.032881 read 0x02393a thru 0x02662f (0x2cf6 bytes) 000106 1787866027.041800 fallocate 0x01aa3c thru 0x023ae9 (0x90ae bytes) 000107 1787866027.047862 write 0x005b84 thru 0x00cc61 (0x70de bytes) 000108 1787866027.064472 fallocate 0x03340b thru 0x038f4b (0x5b41 bytes) 000109 1787866027.078619 punch from 0x000ef5 to 0x008c21 (0x7d2d bytes) 000112 1787866027.093925 mapwrite 0x03348f thru 0x03ffff (0xcb71 bytes) 000113 1787866027.184971 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xd4031c17 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xd4031c17 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000114 1787866027.689655 write 0x01a9ef thru 0x022ec0 (0x84d2 bytes) 000115 1787866027.712208 trunc from 0x037fa4 to 0x03ffff (0x805c bytes) 000116 1787866027.719836 trunc from 0x040000 to 0x01bd79 (0x24286 bytes) 000117 1787866027.744269 write 0x02eb0b thru 0x036103 (0x75f9 bytes) 000118 1787866027.748494 read 0x01ac69 thru 0x02a29c (0xf634 bytes) 000119 1787866027.758977 write 0x036aa7 thru 0x03f53b (0x8a95 bytes) 000120 1787866027.762279 punch from 0x03164c to 0x03b78e (0xa143 bytes) 000122 1787866027.778355 punch from 0x03ef34 to 0x03f53b (0x608 bytes) 000123 1787866027.786617 mapread 0x002061 thru 0x00f0a2 (0xd042 bytes) 000124 1787866027.848041 read 0x0299b8 thru 0x034610 (0xac59 bytes) 000125 1787866027.873173 read 0x02aec6 thru 0x039a02 (0xeb3d bytes) 000126 1787866027.875033 trunc from 0x03f53c to 0x009367 (0x361d4 bytes) 000127 1787866027.895462 fallocate 0x01ff2e thru 0x023992 (0x3a65 bytes) 000128 1787866027.901259 fallocate 0x031363 thru 0x038843 (0x74e1 bytes) 000129 1787866027.911315 punch from 0x0069f3 to 0x009367 (0x2975 bytes) 000130 1787866027.917442 trunc from 0x004658 to 0x009367 (0x4d10 bytes) 000131 1787866027.923462 trunc from 0x008a5e to 0x009367 (0x90a bytes) 000133 1787866027.930873 mapwrite 0x00efda thru 0x016635 (0x765c bytes) 000134 1787866027.985804 mapread 0x00bbce thru 0x00e439 (0x286c bytes) 000135 1787866027.992168 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x16636): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16636): Mirror 4: 0x5879bf70 (stale) CRC-32 checksum value for chunk [0, 0x16636): Mirror 1: 0x60910b49 Mirror 5: 0x60910b49 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16636: skipped 000136 1787866029.445232 write 0x02de02 thru 0x02fe32 (0x2031 bytes) 000137 1787866029.535250 fallocate 0x01dc0e thru 0x028611 (0xaa04 bytes) 000138 1787866029.545493 mapwrite 0x0022d4 thru 0x007b84 (0x58b1 bytes) 000139 1787866029.610542 trunc from 0x02fe33 to 0x01f16a (0x10cc8 bytes) 000140 1787866029.634269 punch from 0x007cf4 to 0x01452d (0xc83a bytes) 000142 1787866029.641065 punch from 0x00f75e to 0x0190f4 (0x9997 bytes) 000144 1787866029.650242 mapwrite 0x00b527 thru 0x00ba8a (0x564 bytes) 000145 1787866029.694039 trunc from 0x01f16b to 0x0150aa (0xa0c0 bytes) 000146 1787866029.721147 mapread 0x006371 thru 0x012e86 (0xcb16 bytes) 000147 1787866029.725064 trunc from 0x0150ab to 0x00762f (0xda7b bytes) 000148 1787866029.774084 MIRROR_RESYNC 3 mirrors 000151 1787866030.752508 read 0x0015b1 thru 0x00762f (0x607f bytes) 000153 1787866030.789688 fallocate 0x01b29d thru 0x027af1 (0xc855 bytes) 000154 1787866030.834985 trunc from 0x01ed2b to 0x027af1 (0x8dc7 bytes) 000155 1787866030.854470 MIRROR_RESYNC 3 mirrors 000156 1787866031.954202 mapread 0x027ae6 thru 0x027af1 (0xc bytes) 000157 1787866031.962231 fallocate 0x020c09 thru 0x028d4b (0x8143 bytes) 000158 1787866032.024097 MIRROR_RESYNC 3 mirrors 000159 1787866032.880292 write 0x012de9 thru 0x01832b (0x5543 bytes) 000161 1787866032.963884 trunc from 0x01538b to 0x021a4c (0xc6c2 bytes) 000162 1787866032.988159 mapwrite 0x030660 thru 0x03de5e (0xd7ff bytes) 000163 1787866033.198063 mapwrite 0x01a5f8 thru 0x02896c (0xe375 bytes) 000164 1787866033.300864 write 0x022647 thru 0x02d40f (0xadc9 bytes) 000165 1787866033.321480 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3de5f): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3de5f): Mirror 4: 0x1b7a96fd (stale) Mirror 5: 0x1b7a96fd (stale) CRC-32 checksum value for chunk [0, 0x3de5f): Mirror 1: 0x9e96107c Mirror 6: 0x9e96107c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3de5f: skipped 000166 1787866034.886383 mapread 0x0034f0 thru 0x00ebae (0xb6bf bytes) 000168 1787866034.926273 write 0x01ae11 thru 0x020f0c (0x60fc bytes) 000169 1787866035.043929 mapwrite 0x029609 thru 0x039304 (0xfcfc bytes) 000170 1787866035.151388 fallocate 0x00fb38 thru 0x0147cc (0x4c95 bytes) 000173 1787866035.173355 trunc from 0x03de5f to 0x021fc1 (0x1be9d bytes) 000175 1787866035.259943 mapwrite 0x000bfd thru 0x007b39 (0x6f3d bytes) 000176 1787866035.327094 read 0x01fdd2 thru 0x021fc1 (0x21f0 bytes) 000177 1787866035.339534 fallocate 0x00b153 thru 0x00f670 (0x451e bytes) 000178 1787866035.357445 trunc from 0x018100 to 0x019b9a (0x1a9b bytes) 000179 1787866035.368617 fallocate 0x01571f thru 0x01a9ea (0x52cc bytes) 000181 1787866035.385013 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21fc2): Mirror 4: 0x5f7bef27 (stale) Mirror 6: 0xc8fbec39 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21fc2): Mirror 4: 0x5f7bef27 (stale) Mirror 6: 0xc8fbec39 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21fc2: skipped 000182 1787866036.019823 trunc from 0x021fc2 to 0x03a68f (0x186ce bytes) 000183 1787866036.059868 write 0x00a1e2 thru 0x00c779 (0x2598 bytes) 000184 1787866036.073484 fallocate 0x03462c thru 0x03f48b (0xae60 bytes) 000185 1787866036.086020 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a690): Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a690): Mirror 4: 0xe620a57d (stale) Mirror 6: 0xde5c9f45 (stale) CRC-32 checksum value for chunk [0, 0x3a690): Mirror 1: 0x3a702467 Mirror 7: 0x3a702467 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a690: skipped 000186 1787866037.450539 mapread 0x038286 thru 0x03a02d (0x1da8 bytes) 000187 1787866037.479333 trunc from 0x023d03 to 0x0243ca (0x6c8 bytes) 000188 1787866037.527561 punch from 0x00877a to 0x017f38 (0xf7bf bytes) 000189 1787866037.534320 read 0x0303f0 thru 0x03a5ae (0xa1bf bytes) 000190 1787866037.546010 mapwrite 0x037390 thru 0x03ffff (0x8c70 bytes) 000191 1787866037.607187 write 0x039b28 thru 0x03ffff (0x64d8 bytes) 000192 1787866037.608450 read 0x01e8bc thru 0x021f43 (0x3688 bytes) 000193 1787866037.647076 trunc from 0x0084b3 to 0x014bfe (0xc74c bytes) 000194 1787866037.653427 mapread 0x018534 thru 0x01d3d7 (0x4ea4 bytes) 000195 1787866037.660814 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xdf8cba8a (stale) Mirror 6: 0x5a728dbc (stale) Mirror 7: 0xa73b6e9c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa2cb68f Mirror 8: 0xa2cb68f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000196 1787866039.125042 read 0x000606 thru 0x000a15 (0x410 bytes) 000197 1787866039.132800 punch from 0x005a97 to 0x011b29 (0xc093 bytes) 000198 1787866039.201107 punch from 0x0108d8 to 0x01a3d3 (0x9afc bytes) 000199 1787866039.224257 trunc from 0x040000 to 0x02bb87 (0x14478 bytes) 000200 1787866039.291269 mapwrite 0x0377f5 thru 0x03ffff (0x880b bytes) 000201 1787866039.407394 read 0x02a877 thru 0x02c9af (0x2139 bytes) 000203 1787866039.465944 MIRROR_RESYNC 5 mirrors 000204 1787866040.806220 trunc from 0x040000 to 0x00c364 (0x33c9b bytes) 000205 1787866040.884276 read 0x002f90 thru 0x004f46 (0x1fb7 bytes) 000206 1787866040.894334 fallocate 0x000074 thru 0x00e984 (0xe911 bytes) 000208 1787866040.903282 mapwrite 0x037969 thru 0x03ac49 (0x32e1 bytes) 000209 1787866041.003352 mapwrite 0x01976b thru 0x01b810 (0x20a6 bytes) 000210 1787866041.051058 fallocate 0x03aeb9 thru 0x03bfee (0x1136 bytes) 000211 1787866041.060022 read 0x007864 thru 0x00fbd3 (0x8370 bytes) 000212 1787866041.084118 read 0x01ad42 thru 0x01d9b1 (0x2c70 bytes) 000213 1787866041.085116 fallocate 0x01c25a thru 0x029c7e (0xda25 bytes) 000215 1787866041.092412 write 0x034d17 thru 0x037bce (0x2eb8 bytes) 000216 1787866041.095065 mapread 0x0357ac thru 0x03ac49 (0x549e bytes) 000217 1787866041.096885 trunc from 0x03ac4a to 0x028a39 (0x12210 bytes) 000218 1787866041.129919 punch from 0x019281 to 0x022e9e (0x9c1e bytes) 000219 1787866041.138073 fallocate 0x026560 thru 0x03613c (0xfbdd bytes) 000220 1787866041.146229 mapwrite 0x010ee2 thru 0x01bac6 (0xabe5 bytes) 000221 1787866041.236325 trunc from 0x000045 to 0x00190f (0x18cb bytes) 000222 1787866041.247788 fallocate 0x02f56e thru 0x03cb79 (0xd60c bytes) 000223 1787866041.271011 trunc from 0x0163e8 to 0x01ad68 (0x4981 bytes) 000224 1787866041.281121 write 0x028b96 thru 0x029ba5 (0x1010 bytes) 000225 1787866041.282159 read 0x02c37c thru 0x039d70 (0xd9f5 bytes) 000226 1787866041.295262 fallocate 0x026ace thru 0x026fad (0x4e0 bytes) 000227 1787866041.306907 trunc from 0x001718 to 0x010f62 (0xf84b bytes) 000228 1787866041.323141 punch from 0x01c3bb to 0x01f986 (0x35cc bytes) 000229 1787866041.335218 punch from 0x01e2c8 to 0x026a06 (0x873f bytes) 000230 1787866041.360626 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3cb7a): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cb7a): Mirror 4: 0x7728eab8 (stale) Mirror 6: 0x7728eab8 (stale) Mirror 7: 0x7728eab8 (stale) Mirror 8: 0x7728eab8 (stale) CRC-32 checksum value for chunk [0, 0x3cb7a): Mirror 1: 0x305801fc Mirror 9: 0x305801fc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cb7a: skipped 000231 1787866042.395212 fallocate 0x032079 thru 0x03dfc3 (0xbf4b bytes) 000233 1787866042.441768 read 0x0112b1 thru 0x01372a (0x247a bytes) 000235 1787866042.452891 punch from 0x00cd18 to 0x0114e8 (0x47d1 bytes) 000236 1787866042.459203 write 0x028249 thru 0x02b73b (0x34f3 bytes) 000237 1787866042.487074 fallocate 0x00017c thru 0x00ff1a (0xfd9f bytes) 000238 1787866042.496112 mapwrite 0x03441a thru 0x0393d2 (0x4fb9 bytes) 000239 1787866042.566364 write 0x01ead8 thru 0x02422c (0x5755 bytes) 000240 1787866042.588019 mapread 0x01e429 thru 0x02d9ca (0xf5a2 bytes) 000241 1787866042.606597 mapread 0x0139d0 thru 0x01c972 (0x8fa3 bytes) 000243 1787866042.645531 mapread 0x017a85 thru 0x01def2 (0x646e bytes) 000244 1787866042.655095 mapwrite 0x00fb51 thru 0x016127 (0x65d7 bytes) 000245 1787866042.701040 fallocate 0x00dd82 thru 0x01015a (0x23d9 bytes) 000246 1787866042.712027 mapwrite 0x0376ce thru 0x03ffff (0x8932 bytes) 000247 1787866042.873004 mapread 0x015f2a thru 0x023857 (0xd92e bytes) 000248 1787866042.874225 trunc from 0x040000 to 0x01059f (0x2fa60 bytes) 000249 1787866042.896133 trunc from 0x0105a0 to 0x013717 (0x3178 bytes) 000250 1787866042.917298 trunc from 0x013718 to 0x02082a (0xd113 bytes) 000251 1787866042.934965 fallocate 0x0328e1 thru 0x03ffff (0xd71f bytes) 000252 1787866042.940391 punch from 0x00772e to 0x01084d (0x9120 bytes) 000253 1787866042.951000 read 0x001659 thru 0x01052c (0xeed4 bytes) 000254 1787866042.971945 MIRROR_RESYNC 6 mirrors 000255 1787866044.551941 fallocate 0x0111a6 thru 0x020823 (0xf67e bytes) 000256 1787866044.605203 trunc from 0x040000 to 0x00c343 (0x33cbc bytes) 000257 1787866044.629628 write 0x036adc thru 0x03ac87 (0x41ac bytes) 000258 1787866044.631933 punch from 0x00f58c to 0x01696f (0x73e4 bytes) 000259 1787866044.636297 punch from 0x022a40 to 0x025578 (0x2b39 bytes) 000260 1787866044.642431 read 0x00fe78 thru 0x019fcb (0xa154 bytes) 000261 1787866044.656191 MIRROR_RESYNC 6 mirrors 000262 1787866046.322578 punch from 0x004c1f to 0x011ba5 (0xcf87 bytes) 000263 1787866046.409136 punch from 0x01246e to 0x01d97a (0xb50d bytes) 000264 1787866046.416777 write 0x029fe8 thru 0x034408 (0xa421 bytes) 000265 1787866046.437035 punch from 0x013837 to 0x020bff (0xd3c9 bytes) 000266 1787866046.453941 trunc from 0x010be7 to 0x01ea30 (0xde4a bytes) 000267 1787866046.471218 write 0x03a9c5 thru 0x03e695 (0x3cd1 bytes) 000268 1787866046.483400 mapwrite 0x014834 thru 0x01bb1e (0x72eb bytes) 000269 1787866046.566474 trunc from 0x006db5 to 0x00a0fc (0x3348 bytes) 000270 1787866046.576703 mapread 0x02059b thru 0x0220fb (0x1b61 bytes) 000271 1787866046.604495 mapread 0x034cd2 thru 0x03dcdb (0x900a bytes) 000272 1787866046.626285 write 0x03c764 thru 0x03ffff (0x389c bytes) 000275 1787866046.634717 write 0x0080b9 thru 0x012805 (0xa74d bytes) 000276 1787866046.643047 write 0x0388dd thru 0x03ffff (0x7723 bytes) 000278 1787866046.646874 mapwrite 0x02c038 thru 0x0331a8 (0x7171 bytes) 000279 1787866046.695116 fallocate 0x02a4de thru 0x038a66 (0xe589 bytes) 000280 1787866046.701968 mapwrite 0x03cf25 thru 0x03ffff (0x30db bytes) 000281 1787866046.724772 read 0x01de9f thru 0x0243c3 (0x6525 bytes) 000282 1787866046.736274 MIRROR_RESYNC 6 mirrors 000283 1787866049.077047 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000285 1787866049.795116 fallocate 0x026941 thru 0x02e814 (0x7ed4 bytes) 000286 1787866049.872949 trunc from 0x040000 to 0x0027ca (0x3d835 bytes) 000287 1787866049.915169 mapwrite 0x011045 thru 0x014534 (0x34f0 bytes) 000288 1787866049.997865 mapwrite 0x02c929 thru 0x039c58 (0xd330 bytes) 000289 1787866050.125605 mapread 0x027967 thru 0x034ffa (0xd694 bytes) 000291 1787866050.128413 read 0x00aa5e thru 0x010b24 (0x60c7 bytes) 000292 1787866050.137196 mapwrite 0x01e102 thru 0x02cfcc (0xeecb bytes) 000293 1787866050.185646 mapread 0x02961d thru 0x032c73 (0x9657 bytes) 000294 1787866050.186568 trunc from 0x01dae9 to 0x02d08e (0xf5a6 bytes) 000295 1787866050.194220 read 0x01eafd thru 0x02b7ab (0xccaf bytes) 000296 1787866050.213858 MIRROR_RESYNC 5 mirrors 000298 1787866051.619735 trunc from 0x039c59 to 0x037697 (0x25c1 bytes) 000299 1787866051.696357 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x37698): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37698): Mirror 4: 0x4e9ba768 (stale) Mirror 6: 0x4e9ba768 (stale) Mirror 7: 0x4e9ba768 (stale) Mirror 8: 0x4e9ba768 (stale) CRC-32 checksum value for chunk [0, 0x37698): Mirror 1: 0x4e9ba768 Mirror 9: 0x4e9ba768 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37698: skipped 000300 1787866053.078762 read 0x004d34 thru 0x005750 (0xa1d bytes) 000301 1787866053.085361 mapread 0x02a8bc thru 0x03078d (0x5ed2 bytes) 000302 1787866053.114080 read 0x00198f thru 0x003ed3 (0x2545 bytes) 000303 1787866053.141174 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37698): Mirror 6: 0x4e9ba768 (stale) Mirror 7: 0x4e9ba768 (stale) Mirror 8: 0x4e9ba768 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37698): Mirror 6: 0x4e9ba768 (stale) Mirror 7: 0x4e9ba768 (stale) Mirror 8: 0x4e9ba768 (stale) CRC-32 checksum value for chunk [0, 0x37698): Mirror 1: 0x4e9ba768 Mirror 9: 0x4e9ba768 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37698: skipped 000304 1787866054.221920 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37698): Mirror 6: 0x4e9ba768 (stale) Mirror 7: 0x4e9ba768 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37698): Mirror 6: 0x4e9ba768 (stale) Mirror 7: 0x4e9ba768 (stale) CRC-32 checksum value for chunk [0, 0x37698): Mirror 1: 0x4e9ba768 Mirror 9: 0x4e9ba768 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37698: skipped 000305 1787866055.196484 punch from 0x028ce3 to 0x0364fc (0xd81a bytes) 000306 1787866055.305350 read 0x020a89 thru 0x02588e (0x4e06 bytes) 000307 1787866055.337139 punch from 0x0069d8 to 0x0111db (0xa804 bytes) 000308 1787866055.353235 trunc from 0x020285 to 0x0251b6 (0x4f32 bytes) 000309 1787866055.375505 trunc from 0x037698 to 0x009875 (0x2de22 bytes) 000310 1787866055.447225 fallocate 0x00b671 thru 0x00bc49 (0x5d9 bytes) 000312 1787866055.466077 MIRROR_RESYNC 4 mirrors 000313 1787866056.594136 mapread 0x0014c6 thru 0x003aeb (0x2626 bytes) 000314 1787866056.618519 fallocate 0x007630 thru 0x012bc0 (0xb591 bytes) 000315 1787866056.687006 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xbc4a): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbc4a): Mirror 6: 0xf5b53b00 (stale) Mirror 7: 0xf5b53b00 (stale) Mirror 9: 0xf5b53b00 (stale) CRC-32 checksum value for chunk [0, 0xbc4a): Mirror 1: 0xf5b53b00 Mirror 10: 0xf5b53b00 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbc4a: skipped 000316 1787866058.258077 fallocate 0x02e9dc thru 0x03c68f (0xdcb4 bytes) 000317 1787866058.343348 mapwrite 0x02301f thru 0x024335 (0x1317 bytes) 000318 1787866058.403581 MIRROR_RESYNC 5 mirrors 000319 1787866059.957854 read 0x00f2aa thru 0x00fa19 (0x770 bytes) 000320 1787866059.964003 punch from 0x03bd36 to 0x03c68f (0x95a bytes) 000321 1787866060.009335 fallocate 0x00cf90 thru 0x01403c (0x70ad bytes) 000322 1787866060.015222 trunc from 0x03c690 to 0x01c66b (0x20024 bytes) 000323 1787866060.050940 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c66c): Mirror 7: 0x46562cc9 (stale) Mirror 9: 0x46562cc9 (stale) Mirror 10: 0x46562cc9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c66c): Mirror 7: 0x46562cc9 (stale) Mirror 9: 0x46562cc9 (stale) Mirror 10: 0x46562cc9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c66c: skipped 000324 1787866060.626921 trunc from 0x01c66c to 0x015d06 (0x6965 bytes) 000325 1787866060.682547 read 0x00a9bc thru 0x011065 (0x66aa bytes) 000326 1787866060.696094 punch from 0x015438 to 0x015d06 (0x8cf bytes) 000327 1787866060.702878 mapwrite 0x03a01c thru 0x03b524 (0x1509 bytes) 000328 1787866060.781891 mapwrite 0x01163f thru 0x013de7 (0x27a9 bytes) 000329 1787866060.812797 mapwrite 0x00f3b3 thru 0x016bbb (0x7809 bytes) 000330 1787866060.841545 trunc from 0x0253ce to 0x02cbb2 (0x77e5 bytes) 000331 1787866060.851596 MIRROR_RESYNC 4 mirrors 000332 1787866062.237127 read 0x00efef thru 0x01388d (0x489f bytes) 000333 1787866062.258948 trunc from 0x03b525 to 0x017cfb (0x23829 bytes) 000334 1787866062.322024 punch from 0x017904 to 0x017cfb (0x3f8 bytes) 000335 1787866062.340998 punch from 0x012c48 to 0x017cfb (0x50b4 bytes) 000336 1787866062.356426 trunc from 0x01667d to 0x017cfb (0x167f bytes) 000337 1787866062.366302 read 0x00222e thru 0x0028ec (0x6bf bytes) 000338 1787866062.385184 fallocate 0x0184fd thru 0x02252e (0xa032 bytes) 000339 1787866062.398408 fallocate 0x03f4de thru 0x03ffff (0xb22 bytes) 000340 1787866062.407584 trunc from 0x017cfc to 0x02198c (0x9c91 bytes) 000341 1787866062.441650 read 0x00b5e5 thru 0x00bcfd (0x719 bytes) 000342 1787866062.443276 write 0x00f47d thru 0x01c80c (0xd390 bytes) 000344 1787866062.459278 mapread 0x00bbc1 thru 0x010b15 (0x4f55 bytes) 000346 1787866062.481403 MIRROR_RESYNC 4 mirrors 000347 1787866063.764178 MIRROR_RESYNC 4 mirrors 000348 1787866064.353732 mapwrite 0x001358 thru 0x0094f0 (0x8199 bytes) 000349 1787866064.534810 write 0x000fc4 thru 0x00e14c (0xd189 bytes) 000350 1787866064.556872 trunc from 0x009402 to 0x01517c (0xbd7b bytes) 000351 1787866064.600832 mapwrite 0x0233f3 thru 0x03155d (0xe16b bytes) 000353 1787866064.758323 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3155e): Mirror 7: 0xcc341c56 (stale) Mirror 9: 0xcc341c56 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3155e): Mirror 7: 0xcc341c56 (stale) Mirror 9: 0xcc341c56 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3155e: skipped 000355 1787866065.198149 fallocate 0x035d56 thru 0x037a5b (0x1d06 bytes) 000356 1787866065.217330 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37a5c): Mirror 7: 0x3c84dd80 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a5c): Mirror 7: 0x3c84dd80 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a5c: skipped 000357 1787866065.610113 mapwrite 0x00824c thru 0x014fd2 (0xcd87 bytes) 000358 1787866065.687802 fallocate 0x02df56 thru 0x02f0a4 (0x114f bytes) 000359 1787866065.693804 trunc from 0x026395 to 0x0273a0 (0x100c bytes) 000360 1787866065.700850 punch from 0x02a5ca to 0x031409 (0x6e40 bytes) 000361 1787866065.707406 mapwrite 0x035c9a thru 0x038098 (0x23ff bytes) 000362 1787866065.754243 trunc from 0x03486a to 0x038098 (0x382f bytes) 000363 1787866065.760287 trunc from 0x038099 to 0x00ec44 (0x29454 bytes) 000364 1787866065.778615 fallocate 0x012513 thru 0x01848e (0x5f7c bytes) 000365 1787866065.803856 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000366 1787866066.014476 mapwrite 0x007710 thru 0x014a13 (0xd304 bytes) 000367 1787866066.200160 mapread 0x004701 thru 0x008841 (0x4141 bytes) 000368 1787866066.205569 fallocate 0x03f518 thru 0x03ffff (0xae8 bytes) 000369 1787866066.216847 fallocate 0x038d24 thru 0x03ff9b (0x7278 bytes) 000370 1787866066.228021 mapwrite 0x003950 thru 0x01037f (0xca30 bytes) 000371 1787866066.295306 trunc from 0x0290e5 to 0x030dc7 (0x7ce3 bytes) 000372 1787866066.304833 read 0x03d5e9 thru 0x03ffff (0x2a17 bytes) 000373 1787866066.312900 trunc from 0x040000 to 0x021fe0 (0x1e01f bytes) 000376 1787866066.345128 read 0x005dd5 thru 0x014630 (0xe85c bytes) 000377 1787866066.346193 read 0x014ff5 thru 0x019cdd (0x4ce9 bytes) 000378 1787866066.357849 trunc from 0x0180ff to 0x01c914 (0x4816 bytes) 000379 1787866066.363926 read 0x00515c thru 0x008162 (0x3007 bytes) 000380 1787866066.364506 mapwrite 0x03ff50 thru 0x03ffff (0xb0 bytes) 000381 1787866066.403146 fallocate 0x00f05d thru 0x01b8f8 (0xc89c bytes) 000382 1787866066.410597 trunc from 0x01f080 to 0x02215a (0x30db bytes) 000383 1787866066.416260 trunc from 0x040000 to 0x021b11 (0x1e4ee bytes) 000384 1787866066.432849 mapwrite 0x01226d thru 0x01aec8 (0x8c5c bytes) 000387 1787866066.466640 trunc from 0x021b12 to 0x02dd8a (0xc279 bytes) 000388 1787866066.487870 mapwrite 0x031e2a thru 0x035b04 (0x3cdb bytes) 000389 1787866066.568860 mapwrite 0x03fc26 thru 0x03ffff (0x3da bytes) 000390 1787866066.613186 write 0x038603 thru 0x03ffff (0x79fd bytes) 000391 1787866066.620575 write 0x015eac thru 0x01a27e (0x43d3 bytes) 000392 1787866066.621975 mapread 0x018b08 thru 0x01ff35 (0x742e bytes) 000393 1787866066.634303 trunc from 0x0037f2 to 0x004b80 (0x138f bytes) 000394 1787866066.641282 write 0x0108c7 thru 0x011fdd (0x1717 bytes) 000395 1787866066.643112 read 0x005958 thru 0x00ca35 (0x70de bytes) 000396 1787866066.643885 punch from 0x038cbf to 0x03ffff (0x7341 bytes) 000397 1787866066.667563 fallocate 0x03f8ab thru 0x03ffff (0x755 bytes) 000399 1787866066.675449 mapwrite 0x00caf8 thru 0x011080 (0x4589 bytes) 000400 1787866066.696739 read 0x00ca08 thru 0x00ec5c (0x2255 bytes) 000401 1787866066.697543 trunc from 0x040000 to 0x01849a (0x27b65 bytes) 000402 1787866066.717589 fallocate 0x003108 thru 0x012f98 (0xfe91 bytes) 000403 1787866066.722836 fallocate 0x029ee2 thru 0x0384cd (0xe5ec bytes) 000404 1787866066.727886 punch from 0x003d05 to 0x007d40 (0x403c bytes) 000405 1787866066.734126 write 0x005939 thru 0x007d70 (0x2438 bytes) 000406 1787866066.748394 trunc from 0x017eb6 to 0x01849a (0x5e5 bytes) 000408 1787866066.755537 read 0x001318 thru 0x011218 (0xff01 bytes) 000409 1787866066.757540 mapread 0x016f72 thru 0x01849a (0x1529 bytes) 000410 1787866066.765188 write 0x03a860 thru 0x03ffff (0x57a0 bytes) 000412 1787866066.768400 trunc from 0x010465 to 0x013fd2 (0x3b6e bytes) 000413 1787866066.774628 read 0x00d41f thru 0x01cb16 (0xf6f8 bytes) 000414 1787866066.789536 mapread 0x01063e thru 0x01dfdb (0xd99e bytes) 000415 1787866066.791576 punch from 0x02de37 to 0x03a239 (0xc403 bytes) 000416 1787866066.811909 mapread 0x01689f thru 0x01717c (0x8de bytes) 000417 1787866066.813173 fallocate 0x030f79 thru 0x033abf (0x2b47 bytes) 000418 1787866066.819095 punch from 0x01fd5f to 0x0247e4 (0x4a86 bytes) 000419 1787866066.829250 MIRROR_EXTEND to 2 mirrors 000420 1787866067.811517 trunc from 0x03bef8 to 0x03ffff (0x4108 bytes) 000421 1787866067.858110 fallocate 0x009908 thru 0x00cf4b (0x3644 bytes) 000422 1787866067.864750 fallocate 0x0234a5 thru 0x0304e5 (0xd041 bytes) 000423 1787866067.876655 trunc from 0x039f00 to 0x03ffff (0x6100 bytes) 000424 1787866067.886731 mapwrite 0x02499c thru 0x02d2ea (0x894f bytes) 000425 1787866067.957652 trunc from 0x040000 to 0x009425 (0x36bda bytes) 000426 1787866068.047435 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x9426): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9426): Mirror 2: 0xda2fcb09 (stale) CRC-32 checksum value for chunk [0, 0x9426): Mirror 1: 0xda2fcb09 Mirror 3: 0xda2fcb09 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9426: skipped 000427 1787866069.006954 fallocate 0x038151 thru 0x03ffff (0x7eaf bytes) 000428 1787866069.061448 mapread 0x009464 thru 0x00fe63 (0x6a00 bytes) 000429 1787866069.086562 read 0x010b2d thru 0x0169d9 (0x5ead bytes) 000430 1787866069.093480 fallocate 0x032e02 thru 0x039dca (0x6fc9 bytes) 000431 1787866069.098736 mapwrite 0x03c6fb thru 0x03ccb1 (0x5b7 bytes) 000432 1787866069.133038 MIRROR_RESYNC 3 mirrors 000433 1787866070.106799 mapwrite 0x03fa8b thru 0x03ffff (0x575 bytes) 000435 1787866070.269016 trunc from 0x040000 to 0x03c2fe (0x3d01 bytes) 000436 1787866070.365785 read 0x02f555 thru 0x03c2fe (0xcdaa bytes) 000437 1787866070.442038 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c2ff): Mirror 3: 0x684db444 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c2ff): Mirror 3: 0x684db444 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c2ff: skipped 000438 1787866070.853170 write 0x01a780 thru 0x01be5d (0x16de bytes) 000439 1787866070.873481 mapread 0x038107 thru 0x03b59b (0x3495 bytes) 000440 1787866070.897795 trunc from 0x03c2ff to 0x02fa16 (0xc8e8 bytes) 000441 1787866070.917057 write 0x0253bc thru 0x027aef (0x2734 bytes) 000442 1787866070.931179 write 0x03fe00 thru 0x03ffff (0x200 bytes) 000443 1787866070.932530 trunc from 0x01da6a to 0x01dfb5 (0x54c bytes) 000444 1787866070.937811 fallocate 0x022ff1 thru 0x024bab (0x1bbb bytes) 000446 1787866070.944425 mapwrite 0x03cc25 thru 0x03ffff (0x33db bytes) 000447 1787866070.979231 mapread 0x01c244 thru 0x026176 (0x9f33 bytes) 000449 1787866070.995146 fallocate 0x019eb3 thru 0x01d1d7 (0x3325 bytes) 000451 1787866071.002087 trunc from 0x0035e2 to 0x011711 (0xe130 bytes) 000452 1787866071.008553 punch from 0x0009ee to 0x005864 (0x4e77 bytes) 000453 1787866071.015518 punch from 0x02ba02 to 0x02e6e1 (0x2ce0 bytes) 000454 1787866071.021490 mapwrite 0x00d5da thru 0x00fb9f (0x25c6 bytes) 000455 1787866071.053129 punch from 0x01a2c3 to 0x0269e8 (0xc726 bytes) 000456 1787866071.059027 mapwrite 0x00b39a thru 0x016050 (0xacb7 bytes) 000457 1787866071.097552 trunc from 0x0147f4 to 0x01cc01 (0x840e bytes) 000458 1787866071.107466 trunc from 0x002a73 to 0x0044aa (0x1a38 bytes) 000459 1787866071.112457 punch from 0x02b38c to 0x03607d (0xacf2 bytes) 000460 1787866071.118937 trunc from 0x040000 to 0x02b672 (0x1498d bytes) 000461 1787866071.143610 mapwrite 0x0349ee thru 0x0396ce (0x4ce1 bytes) 000462 1787866071.193927 read 0x015753 thru 0x018618 (0x2ec6 bytes) 000463 1787866071.201130 read 0x016a96 thru 0x025c04 (0xf16f bytes) 000464 1787866071.219300 mapread 0x02021a thru 0x025815 (0x55fc bytes) 000465 1787866071.220212 punch from 0x016554 to 0x01c2e1 (0x5d8e bytes) 000466 1787866071.225421 write 0x0344b7 thru 0x03f73a (0xb284 bytes) 000467 1787866071.227008 write 0x00522d thru 0x00bc52 (0x6a26 bytes) 000468 1787866071.229331 trunc from 0x03f73b to 0x03954e (0x61ec bytes) 000469 1787866071.265236 punch from 0x036a55 to 0x03954e (0x2afa bytes) 000470 1787866071.272483 fallocate 0x00fd33 thru 0x01dd93 (0xe061 bytes) 000471 1787866071.277756 trunc from 0x03954f to 0x034f3b (0x4613 bytes) 000472 1787866071.292012 read 0x016da0 thru 0x01baaa (0x4d0b bytes) 000473 1787866071.302298 trunc from 0x02e23a to 0x034f3b (0x6d02 bytes) 000474 1787866071.312799 mapread 0x0061bb thru 0x00689a (0x6e0 bytes) 000475 1787866071.314255 mapread 0x02c231 thru 0x0322e7 (0x60b7 bytes) 000476 1787866071.374183 read 0x029b83 thru 0x032bf2 (0x9070 bytes) 000477 1787866071.375054 trunc from 0x02898c to 0x0301e2 (0x7857 bytes) 000479 1787866071.387396 mapwrite 0x01887d thru 0x01df37 (0x56bb bytes) 000480 1787866071.434308 mapwrite 0x03f574 thru 0x03f6a3 (0x130 bytes) 000481 1787866071.492089 write 0x005e69 thru 0x01544f (0xf5e7 bytes) 000482 1787866071.499063 mapread 0x008ab2 thru 0x00b801 (0x2d50 bytes) 000483 1787866071.500154 write 0x01b785 thru 0x028d44 (0xd5c0 bytes) 000484 1787866071.504161 punch from 0x00fda6 to 0x011358 (0x15b3 bytes) 000485 1787866071.528077 trunc from 0x01594c to 0x01719e (0x1853 bytes) 000486 1787866071.544376 fallocate 0x01bd16 thru 0x02935c (0xd647 bytes) 000487 1787866071.562006 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000488 1787866071.763348 trunc from 0x03666f to 0x03f6a3 (0x9035 bytes) 000489 1787866071.772346 fallocate 0x03cef3 thru 0x03ffff (0x310d bytes) 000490 1787866071.778342 trunc from 0x000781 to 0x004824 (0x40a4 bytes) 000491 1787866071.786040 mapwrite 0x0294d7 thru 0x02eeb4 (0x59de bytes) 000492 1787866071.856714 mapread 0x00b2b0 thru 0x00f74e (0x449f bytes) 000493 1787866071.875651 write 0x019269 thru 0x026dc1 (0xdb59 bytes) 000494 1787866071.891183 mapwrite 0x037fa0 thru 0x03a2a7 (0x2308 bytes) 000495 1787866071.941280 fallocate 0x0104fc thru 0x018f0f (0x8a14 bytes) 000496 1787866071.949645 mapread 0x0285aa thru 0x02fae3 (0x753a bytes) 000497 1787866071.951531 punch from 0x0114aa to 0x019522 (0x8079 bytes) 000498 1787866071.970512 MIRROR_EXTEND to 2 mirrors 000499 1787866073.121072 read 0x02dda8 thru 0x036833 (0x8a8c bytes) 000501 1787866073.136960 trunc from 0x030845 to 0x03d4c6 (0xcc82 bytes) 000502 1787866073.228086 mapwrite 0x02240b thru 0x03162c (0xf222 bytes) 000504 1787866073.349337 trunc from 0x040000 to 0x004fa9 (0x3b056 bytes) 000505 1787866073.395810 read 0x004820 thru 0x004fa9 (0x78a bytes) 000506 1787866073.413859 trunc from 0x004faa to 0x00fc46 (0xac9d bytes) 000507 1787866073.444903 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xfc47): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfc47): Mirror 2: 0x9cec0e23 (stale) CRC-32 checksum value for chunk [0, 0xfc47): Mirror 1: 0xe75c5719 Mirror 3: 0xe75c5719 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfc47: skipped 000508 1787866074.446657 punch from 0x007eeb to 0x00c76d (0x4883 bytes) 000509 1787866074.496762 trunc from 0x00b3f2 to 0x00eb3f (0x374e bytes) 000510 1787866074.503269 mapread 0x0074f8 thru 0x00a7ab (0x32b4 bytes) 000511 1787866074.523800 mapwrite 0x015e08 thru 0x01f48a (0x9683 bytes) 000512 1787866074.657480 fallocate 0x02c3f8 thru 0x032ef0 (0x6af9 bytes) 000513 1787866074.665660 fallocate 0x005d28 thru 0x007816 (0x1aef bytes) 000514 1787866074.672772 trunc from 0x010adc to 0x0170bb (0x65e0 bytes) 000515 1787866074.683956 write 0x00825b thru 0x017846 (0xf5ec bytes) 000516 1787866074.692925 trunc from 0x021127 to 0x02dd72 (0xcc4c bytes) 000517 1787866074.702022 read 0x013494 thru 0x01bc9f (0x880c bytes) 000518 1787866074.703888 read 0x0320cd thru 0x032ef0 (0xe24 bytes) 000519 1787866074.719990 write 0x033510 thru 0x033d6d (0x85e bytes) 000520 1787866074.722322 read 0x01c888 thru 0x02b132 (0xe8ab bytes) 000522 1787866074.724531 mapread 0x027540 thru 0x0285ab (0x106c bytes) 000523 1787866074.726311 mapread 0x00df6e thru 0x01578d (0x7820 bytes) 000524 1787866074.727306 read 0x003507 thru 0x00f061 (0xbb5b bytes) 000525 1787866074.728384 mapwrite 0x00e9b1 thru 0x01cde4 (0xe434 bytes) 000526 1787866074.756236 read 0x01d13f thru 0x026116 (0x8fd8 bytes) 000527 1787866074.763257 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33d6e): Mirror 3: 0x6113295b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33d6e): Mirror 3: 0x6113295b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33d6e: skipped 000528 1787866075.069307 trunc from 0x017b00 to 0x01e642 (0x6b43 bytes) 000529 1787866075.078258 punch from 0x031853 to 0x033d6d (0x251b bytes) 000530 1787866075.086213 punch from 0x00be4a to 0x00cf7b (0x1132 bytes) 000531 1787866075.092262 read 0x02e08a thru 0x03241a (0x4391 bytes) 000532 1787866075.105191 mapwrite 0x001395 thru 0x001c95 (0x901 bytes) 000533 1787866075.135625 fallocate 0x03e3f2 thru 0x03ffff (0x1c0e bytes) 000534 1787866075.141394 fallocate 0x01a27a thru 0x0256b9 (0xb440 bytes) 000535 1787866075.146925 write 0x003b7d thru 0x00654c (0x29d0 bytes) 000536 1787866075.161499 trunc from 0x033d6e to 0x0223c5 (0x119a8 bytes) 000537 1787866075.181499 fallocate 0x00a750 thru 0x00daa0 (0x3351 bytes) 000538 1787866075.187371 punch from 0x020326 to 0x0223c5 (0x20a0 bytes) 000539 1787866075.192522 punch from 0x01c166 to 0x0223c5 (0x6260 bytes) 000540 1787866075.198268 mapread 0x001691 thru 0x00b6ca (0xa03a bytes) 000541 1787866075.220705 MIRROR_RESYNC 2 mirrors 000542 1787866076.068223 read 0x017a20 thru 0x01975d (0x1d3e bytes) 000544 1787866076.078604 write 0x015fba thru 0x0251d2 (0xf219 bytes) 000545 1787866076.152181 fallocate 0x037f2e thru 0x03c5ba (0x468d bytes) 000546 1787866076.165123 mapread 0x000ffe thru 0x0062e3 (0x52e6 bytes) 000547 1787866076.204406 mapwrite 0x02d3f5 thru 0x034c8b (0x7897 bytes) 000549 1787866076.373280 read 0x022b9e thru 0x026ef6 (0x4359 bytes) 000550 1787866076.374534 write 0x010215 thru 0x01742e (0x721a bytes) 000551 1787866076.404129 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x34c8c): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34c8c): Mirror 3: 0xd7229a7d (stale) CRC-32 checksum value for chunk [0, 0x34c8c): Mirror 1: 0x89e728e7 Mirror 4: 0x89e728e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34c8c: skipped 000552 1787866077.298130 trunc from 0x01893a to 0x022170 (0x9837 bytes) 000553 1787866077.348558 trunc from 0x01dcc3 to 0x02b67c (0xd9ba bytes) 000554 1787866077.357848 write 0x00f024 thru 0x01a5d6 (0xb5b3 bytes) 000555 1787866077.399939 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x34c8c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34c8c): Mirror 3: 0xd7229a7d (stale) Mirror 4: 0x89e728e7 (stale) CRC-32 checksum value for chunk [0, 0x34c8c): Mirror 1: 0xa36e672 Mirror 5: 0xa36e672 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34c8c: skipped 000556 1787866078.428472 mapread 0x0242cc thru 0x02be00 (0x7b35 bytes) 000557 1787866078.452038 fallocate 0x02baf9 thru 0x036461 (0xa969 bytes) 000559 1787866078.497710 mapwrite 0x02f2c8 thru 0x0368f6 (0x762f bytes) 000560 1787866078.555896 write 0x01f23e thru 0x0297dc (0xa59f bytes) 000561 1787866078.562402 trunc from 0x0335e9 to 0x034305 (0xd1d bytes) 000562 1787866078.568535 write 0x018aa9 thru 0x024a28 (0xbf80 bytes) 000563 1787866078.575453 write 0x003750 thru 0x00e2f7 (0xaba8 bytes) 000564 1787866078.590601 mapread 0x00733b thru 0x0094f0 (0x21b6 bytes) 000565 1787866078.591596 trunc from 0x0368f7 to 0x000c37 (0x35cbf bytes) 000566 1787866078.612288 punch from 0x0009e0 to 0x000c37 (0x258 bytes) 000567 1787866078.618455 mapread 0x000b43 thru 0x000c37 (0xf5 bytes) 000568 1787866078.624253 punch from 0x000a4c to 0x000c37 (0x1ec bytes) 000569 1787866078.634250 trunc from 0x0005b3 to 0x000c37 (0x685 bytes) 000570 1787866078.640450 read 0x00051d thru 0x000c37 (0x71b bytes) 000571 1787866078.646405 trunc from 0x000c38 to 0x03cab9 (0x3be82 bytes) 000572 1787866078.668479 mapwrite 0x024568 thru 0x02b531 (0x6fca bytes) 000574 1787866078.708275 write 0x006033 thru 0x00d36f (0x733d bytes) 000575 1787866078.726349 MIRROR_RESYNC 4 mirrors 000576 1787866080.506355 fallocate 0x03c97c thru 0x03ffff (0x3684 bytes) 000577 1787866080.565041 punch from 0x02f0ae to 0x0352f2 (0x6245 bytes) 000578 1787866080.572372 trunc from 0x040000 to 0x036aa6 (0x9559 bytes) 000579 1787866080.596904 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36aa7): Mirror 3: 0x9d5bb164 (stale) Mirror 5: 0x9d5bb164 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36aa7): Mirror 3: 0x9d5bb164 (stale) Mirror 5: 0x9d5bb164 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36aa7: skipped 000581 1787866080.980654 trunc from 0x036aa7 to 0x00406d (0x32a39 bytes) 000582 1787866081.006695 trunc from 0x00406e to 0x025802 (0x21795 bytes) 000583 1787866081.032407 fallocate 0x0116c5 thru 0x01d519 (0xbe55 bytes) 000584 1787866081.039081 trunc from 0x00cf50 to 0x018af5 (0xbba6 bytes) 000585 1787866081.046222 mapwrite 0x02eeb1 thru 0x036642 (0x7792 bytes) 000586 1787866081.107153 fallocate 0x02e4b2 thru 0x02fce7 (0x1836 bytes) 000587 1787866081.113378 trunc from 0x006653 to 0x0136f7 (0xd0a5 bytes) 000588 1787866081.119892 trunc from 0x01f89a to 0x01fd03 (0x46a bytes) 000589 1787866081.125105 punch from 0x00fd44 to 0x01eac9 (0xed86 bytes) 000590 1787866081.133542 read 0x021ab9 thru 0x02c993 (0xaedb bytes) 000591 1787866081.151572 mapread 0x0357fc thru 0x036642 (0xe47 bytes) 000592 1787866081.152932 mapwrite 0x006e1f thru 0x0146e1 (0xd8c3 bytes) 000593 1787866081.207187 write 0x02489f thru 0x0259b3 (0x1115 bytes) 000594 1787866081.209380 write 0x025419 thru 0x032295 (0xce7d bytes) 000595 1787866081.213452 trunc from 0x00f644 to 0x01ecbe (0xf67b bytes) 000596 1787866081.221161 mapwrite 0x01637e thru 0x01a83f (0x44c2 bytes) 000597 1787866081.254935 write 0x03358b thru 0x038dea (0x5860 bytes) 000598 1787866081.256100 read 0x018b1e thru 0x0192ac (0x78f bytes) 000599 1787866081.256643 write 0x02d9e5 thru 0x0372da (0x98f6 bytes) 000600 1787866081.258888 fallocate 0x036d3e thru 0x03ffff (0x92c2 bytes) 000601 1787866081.265310 trunc from 0x033a4a to 0x0373eb (0x39a2 bytes) 000603 1787866081.289069 write 0x03ef1b thru 0x03ffff (0x10e5 bytes) 000604 1787866081.305302 trunc from 0x01bd16 to 0x01dfd4 (0x22bf bytes) 000605 1787866081.312140 write 0x028f60 thru 0x037ea8 (0xef49 bytes) 000606 1787866081.315525 mapread 0x015f5c thru 0x01b696 (0x573b bytes) 000607 1787866081.328105 MIRROR_RESYNC 3 mirrors 000608 1787866082.632269 trunc from 0x0213cd to 0x02a3e8 (0x901c bytes) 000609 1787866082.675839 write 0x02cdb4 thru 0x038883 (0xbad0 bytes) 000610 1787866082.691277 trunc from 0x040000 to 0x0391e0 (0x6e1f bytes) 000612 1787866082.712582 read 0x02f9d5 thru 0x0312ea (0x1916 bytes) 000613 1787866082.713327 read 0x004218 thru 0x00914c (0x4f35 bytes) 000614 1787866082.734514 fallocate 0x005eab thru 0x010596 (0xa6ec bytes) 000615 1787866082.741976 mapwrite 0x00c69f thru 0x012351 (0x5cb3 bytes) 000616 1787866082.769208 punch from 0x01914c to 0x020439 (0x72ee bytes) 000617 1787866082.779428 mapwrite 0x035a28 thru 0x03ac67 (0x5240 bytes) 000618 1787866082.858387 fallocate 0x0320a7 thru 0x03677f (0x46d9 bytes) 000619 1787866082.872167 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3ac68): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ac68): Mirror 3: 0xcc9be5c2 (stale) Mirror 5: 0xcc9be5c2 (stale) CRC-32 checksum value for chunk [0, 0x3ac68): Mirror 1: 0xe83aea6 Mirror 6: 0xe83aea6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ac68: skipped 000621 1787866084.079896 read 0x01ae4d thru 0x027b92 (0xcd46 bytes) 000622 1787866084.093321 trunc from 0x00bb6a to 0x016a44 (0xaedb bytes) 000623 1787866084.160418 read 0x001663 thru 0x00da86 (0xc424 bytes) 000624 1787866084.194573 mapread 0x02c459 thru 0x02e28f (0x1e37 bytes) 000625 1787866084.196112 punch from 0x03418a to 0x03ac67 (0x6ade bytes) 000626 1787866084.210533 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ac68): Mirror 3: 0xcc9be5c2 (stale) Mirror 6: 0xe83aea6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ac68): Mirror 3: 0xcc9be5c2 (stale) Mirror 6: 0xe83aea6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ac68: skipped 000628 1787866084.743124 trunc from 0x03ac68 to 0x0105fe (0x2a669 bytes) 000629 1787866084.782628 trunc from 0x008a77 to 0x0105fe (0x7b88 bytes) 000630 1787866084.791281 mapread 0x00a06b thru 0x0105fe (0x6594 bytes) 000631 1787866084.814079 read 0x005ed0 thru 0x0105fe (0xa72f bytes) 000633 1787866084.816031 mapwrite 0x03f838 thru 0x03ffff (0x7c8 bytes) 000634 1787866084.862266 fallocate 0x006318 thru 0x0098de (0x35c7 bytes) 000635 1787866084.869058 mapread 0x03b1f0 thru 0x03ffff (0x4e10 bytes) 000636 1787866084.870480 mapwrite 0x00a41c thru 0x00eaf0 (0x46d5 bytes) 000637 1787866084.894627 mapwrite 0x03ccf0 thru 0x03ffff (0x3310 bytes) 000638 1787866084.919224 write 0x033fe8 thru 0x03fcac (0xbcc5 bytes) 000639 1787866084.937554 trunc from 0x040000 to 0x034a41 (0xb5be bytes) 000640 1787866084.972217 trunc from 0x034a42 to 0x02097d (0x140c4 bytes) 000641 1787866085.002978 trunc from 0x00d9b1 to 0x01869c (0xacec bytes) 000642 1787866085.011939 punch from 0x000734 to 0x00f642 (0xef0f bytes) 000643 1787866085.026583 trunc from 0x02097e to 0x03eb61 (0x1e1e4 bytes) 000644 1787866085.083139 fallocate 0x021094 thru 0x026c58 (0x5bc5 bytes) 000645 1787866085.094008 trunc from 0x02b6d5 to 0x038472 (0xcd9e bytes) 000646 1787866085.106219 trunc from 0x024a56 to 0x03068a (0xbc35 bytes) 000647 1787866085.118504 write 0x03ca3e thru 0x03ffff (0x35c2 bytes) 000648 1787866085.127329 read 0x03e8de thru 0x03ffff (0x1722 bytes) 000649 1787866085.128224 read 0x02de42 thru 0x03be7f (0xe03e bytes) 000650 1787866085.157372 trunc from 0x040000 to 0x0337e1 (0xc81e bytes) 000651 1787866085.203022 mapread 0x025fee thru 0x027ecb (0x1ede bytes) 000652 1787866085.204157 read 0x01b35e thru 0x023107 (0x7daa bytes) 000654 1787866085.206265 mapread 0x01dd22 thru 0x028845 (0xab24 bytes) 000655 1787866085.208131 punch from 0x030113 to 0x0337e1 (0x36cf bytes) 000656 1787866085.218123 trunc from 0x008068 to 0x00d4b2 (0x544b bytes) 000657 1787866085.227754 mapread 0x0174d7 thru 0x0267b7 (0xf2e1 bytes) 000658 1787866085.229403 trunc from 0x0337e2 to 0x00b279 (0x28568 bytes) 000659 1787866085.267172 punch from 0x0020ae to 0x0097bb (0x770e bytes) 000660 1787866085.293251 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb27a): Mirror 6: 0x142843f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb27a): Mirror 6: 0x142843f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb27a: skipped 000662 1787866085.749778 write 0x015a16 thru 0x01ac9c (0x5287 bytes) 000663 1787866085.756306 punch from 0x010e9a to 0x017c79 (0x6de0 bytes) 000664 1787866085.778944 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1ac9d): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ac9d): Mirror 6: 0xbeba9687 (stale) CRC-32 checksum value for chunk [0, 0x1ac9d): Mirror 1: 0x11f019cd Mirror 7: 0x11f019cd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ac9d: skipped 000665 1787866086.745536 mapwrite 0x03350f thru 0x0395e8 (0x60da bytes) 000666 1787866086.908481 mapwrite 0x03fa33 thru 0x03ffff (0x5cd bytes) 000667 1787866086.982027 trunc from 0x008b60 to 0x0121aa (0x964b bytes) 000668 1787866086.991488 fallocate 0x025564 thru 0x0258ee (0x38b bytes) 000669 1787866086.996793 write 0x0041a5 thru 0x00bc99 (0x7af5 bytes) 000670 1787866087.007512 read 0x01d8a4 thru 0x02c8f5 (0xf052 bytes) 000671 1787866087.025046 read 0x01b0d6 thru 0x02a56d (0xf498 bytes) 000672 1787866087.041816 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x379e3aac (stale) Mirror 7: 0xc85a7863 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xec05c442 Mirror 8: 0xec05c442 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000674 1787866088.179344 trunc from 0x040000 to 0x02129f (0x1ed60 bytes) 000675 1787866088.316609 write 0x02813b thru 0x0323cd (0xa293 bytes) 000676 1787866088.318166 punch from 0x026b5a to 0x0279c5 (0xe6c bytes) 000677 1787866088.363189 MIRROR_RESYNC 4 mirrors 000678 1787866090.160229 fallocate 0x0004a8 thru 0x00b844 (0xb39d bytes) 000679 1787866090.211200 punch from 0x024b91 to 0x0323cd (0xd83d bytes) 000680 1787866090.219757 trunc from 0x0323ce to 0x01f5b9 (0x12e14 bytes) 000681 1787866090.277105 MIRROR_RESYNC 4 mirrors 000683 1787866091.925786 trunc from 0x01f5ba to 0x03b7a6 (0x1c1ed bytes) 000684 1787866092.043338 punch from 0x023a9c to 0x02dda5 (0xa30a bytes) 000686 1787866092.055242 write 0x00885a thru 0x011622 (0x8dc9 bytes) 000687 1787866092.077053 trunc from 0x03b7a7 to 0x02e4af (0xd2f7 bytes) 000688 1787866092.115765 trunc from 0x02e4b0 to 0x0202b4 (0xe1fb bytes) 000689 1787866092.147157 fallocate 0x008184 thru 0x01144a (0x92c7 bytes) 000690 1787866092.167005 write 0x0111b1 thru 0x01440a (0x325a bytes) 000691 1787866092.187585 fallocate 0x008008 thru 0x011c0e (0x9c07 bytes) 000692 1787866092.203514 trunc from 0x0202b5 to 0x027e76 (0x7bc2 bytes) 000693 1787866092.245715 read 0x012c5e thru 0x01d698 (0xaa3b bytes) 000694 1787866092.271767 read 0x026198 thru 0x027e76 (0x1cdf bytes) 000695 1787866092.299620 fallocate 0x031445 thru 0x03cb74 (0xb730 bytes) 000696 1787866092.307476 write 0x000b5f thru 0x000d2e (0x1d0 bytes) 000697 1787866092.309567 trunc from 0x027e77 to 0x03f8ed (0x17a77 bytes) 000698 1787866092.360212 fallocate 0x031a07 thru 0x03ffff (0xe5f9 bytes) 000699 1787866092.384026 trunc from 0x03f8ee to 0x03ff75 (0x688 bytes) 000700 1787866092.413053 write 0x027a05 thru 0x02b948 (0x3f44 bytes) 000701 1787866092.436076 mapwrite 0x02de55 thru 0x036082 (0x822e bytes) 000702 1787866092.494089 mapwrite 0x025f69 thru 0x032095 (0xc12d bytes) 000703 1787866092.523990 mapread 0x02dd7b thru 0x02e06a (0x2f0 bytes) 000704 1787866092.525933 fallocate 0x02f44e thru 0x0366c1 (0x7274 bytes) 000705 1787866092.533331 write 0x012a32 thru 0x01f438 (0xca07 bytes) 000706 1787866092.536396 mapread 0x024fac thru 0x02f619 (0xa66e bytes) 000707 1787866092.538782 punch from 0x032c29 to 0x03ff75 (0xd34d bytes) 000708 1787866092.545783 mapread 0x01e91f thru 0x022ab5 (0x4197 bytes) 000709 1787866092.547925 mapwrite 0x01e93a thru 0x026d01 (0x83c8 bytes) 000712 1787866092.581085 mapwrite 0x02a8f1 thru 0x02c9c7 (0x20d7 bytes) 000713 1787866092.601471 write 0x026217 thru 0x036134 (0xff1e bytes) 000714 1787866092.617262 punch from 0x039c0c to 0x03ff75 (0x636a bytes) 000715 1787866092.630843 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3ff76): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ff76): Mirror 6: 0x78eb90cc (stale) Mirror 7: 0x78eb90cc (stale) Mirror 8: 0x78eb90cc (stale) CRC-32 checksum value for chunk [0, 0x3ff76): Mirror 1: 0xd934f468 Mirror 9: 0xd934f468 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ff76: skipped 000716 1787866094.230766 punch from 0x002c52 to 0x008f2d (0x62dc bytes) 000717 1787866094.300347 trunc from 0x03ff76 to 0x00c272 (0x33d03 bytes) 000718 1787866094.381091 MIRROR_RESYNC 5 mirrors 000719 1787866095.586629 trunc from 0x00c273 to 0x0185d4 (0xc362 bytes) 000720 1787866095.666346 trunc from 0x0185d5 to 0x002188 (0x1644c bytes) 000721 1787866095.693255 mapread 0x00214f thru 0x002188 (0x3a bytes) 000722 1787866095.701034 punch from 0x00080f to 0x002188 (0x197a bytes) 000723 1787866095.710209 trunc from 0x002189 to 0x011599 (0xf411 bytes) 000724 1787866095.735994 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1159a): Mirror 6: 0xa78c6cc (stale) Mirror 7: 0xa78c6cc (stale) Mirror 9: 0xa78c6cc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1159a): Mirror 6: 0xa78c6cc (stale) Mirror 7: 0xa78c6cc (stale) Mirror 9: 0xa78c6cc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1159a: skipped 000725 1787866096.195002 trunc from 0x01159a to 0x023daa (0x12811 bytes) 000726 1787866096.220312 write 0x019af4 thru 0x024112 (0xa61f bytes) 000728 1787866096.230133 fallocate 0x03011d thru 0x034ff5 (0x4ed9 bytes) 000729 1787866096.239252 trunc from 0x034ff6 to 0x01139b (0x23c5a bytes) 000730 1787866096.266011 fallocate 0x03c2e2 thru 0x03ffff (0x3d1e bytes) 000731 1787866096.271935 write 0x03cab9 thru 0x03ffff (0x3547 bytes) 000732 1787866096.277996 mapwrite 0x02d1c3 thru 0x03a2bd (0xd0fb bytes) 000734 1787866096.342995 mapread 0x019fa6 thru 0x023410 (0x946b bytes) 000735 1787866096.367760 mapwrite 0x03348e thru 0x03ffff (0xcb72 bytes) 000736 1787866096.401218 mapread 0x034edf thru 0x03ffff (0xb121 bytes) 000737 1787866096.409335 MIRROR_RESYNC 4 mirrors 000738 1787866097.497053 trunc from 0x040000 to 0x02d46b (0x12b94 bytes) 000739 1787866097.613321 punch from 0x0157fe to 0x01a857 (0x505a bytes) 000740 1787866097.624563 trunc from 0x0107b2 to 0x017f04 (0x7753 bytes) 000742 1787866097.637017 mapread 0x029c95 thru 0x02d46b (0x37d7 bytes) 000744 1787866097.694120 write 0x03f7f2 thru 0x03ffff (0x80e bytes) 000745 1787866097.698115 punch from 0x02cb8f to 0x0308ad (0x3d1f bytes) 000746 1787866097.711506 read 0x000521 thru 0x0083a7 (0x7e87 bytes) 000747 1787866097.750889 trunc from 0x040000 to 0x0114eb (0x2eb14 bytes) 000748 1787866097.792126 trunc from 0x0114ec to 0x02eb43 (0x1d658 bytes) 000749 1787866097.825263 fallocate 0x023824 thru 0x031d12 (0xe4ef bytes) 000750 1787866097.837577 mapwrite 0x03c22c thru 0x03ffff (0x3dd4 bytes) 000751 1787866097.887755 read 0x035a50 thru 0x03a932 (0x4ee3 bytes) 000752 1787866097.894877 MIRROR_RESYNC 4 mirrors 000753 1787866099.062227 trunc from 0x03969a to 0x03ffff (0x6966 bytes) 000754 1787866099.139823 mapread 0x038d28 thru 0x03ffff (0x72d8 bytes) 000755 1787866099.179291 trunc from 0x040000 to 0x008ed2 (0x3712d bytes) 000756 1787866099.239901 punch from 0x0003f5 to 0x0066eb (0x62f7 bytes) 000757 1787866099.249138 mapread 0x0049bc thru 0x008ed2 (0x4517 bytes) 000758 1787866099.271053 write 0x0289d4 thru 0x0296f0 (0xd1d bytes) 000759 1787866099.272516 mapread 0x0280d1 thru 0x0296f0 (0x1620 bytes) 000760 1787866099.277022 mapread 0x014526 thru 0x023454 (0xef2f bytes) 000761 1787866099.295497 trunc from 0x016821 to 0x01e765 (0x7f45 bytes) 000762 1787866099.302473 mapwrite 0x01f8ca thru 0x0286a6 (0x8ddd bytes) 000763 1787866099.335655 mapwrite 0x00e15f thru 0x0186ae (0xa550 bytes) 000764 1787866099.369352 punch from 0x0034b5 to 0x005916 (0x2462 bytes) 000765 1787866099.376298 read 0x0048cc thru 0x013f95 (0xf6ca bytes) 000766 1787866099.391128 trunc from 0x0296f1 to 0x02166e (0x8082 bytes) 000767 1787866099.433094 read 0x017d2e thru 0x01a0de (0x23b1 bytes) 000768 1787866099.440259 mapwrite 0x02f3bd thru 0x02f928 (0x56c bytes) 000769 1787866099.493208 trunc from 0x010b8d to 0x0135d3 (0x2a47 bytes) 000770 1787866099.500927 trunc from 0x00a20d to 0x019010 (0xee04 bytes) 000771 1787866099.508092 trunc from 0x02f929 to 0x000e94 (0x2ea94 bytes) 000773 1787866099.537188 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xe95): Mirror 7: 0x565f7c5 (stale) Mirror 9: 0x565f7c5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe95): Mirror 7: 0x565f7c5 (stale) Mirror 9: 0x565f7c5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe95: skipped 000775 1787866099.946996 punch from 0x000afa to 0x000e94 (0x39b bytes) 000776 1787866099.963263 mapwrite 0x028db6 thru 0x02a279 (0x14c4 bytes) 000777 1787866100.087366 mapread 0x00b700 thru 0x013fed (0x88ee bytes) 000778 1787866100.120853 read 0x022ed0 thru 0x02a279 (0x73aa bytes) 000779 1787866100.132258 mapwrite 0x02b000 thru 0x0399fe (0xe9ff bytes) 000781 1787866100.240542 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x399ff): Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x399ff): Mirror 7: 0x4989248a (stale) Mirror 9: 0x4989248a (stale) CRC-32 checksum value for chunk [0, 0x399ff): Mirror 1: 0x5b4c34e2 Mirror 10: 0x5b4c34e2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x399ff: skipped 000782 1787866101.513559 punch from 0x01ea26 to 0x0206da (0x1cb5 bytes) 000783 1787866101.584254 mapwrite 0x015f5a thru 0x01d119 (0x71c0 bytes) 000784 1787866101.672179 mapwrite 0x024e08 thru 0x02597b (0xb74 bytes) 000785 1787866101.702101 trunc from 0x014dda to 0x016c0a (0x1e31 bytes) 000786 1787866101.710966 write 0x035cc8 thru 0x03ffff (0xa338 bytes) 000787 1787866101.729229 mapread 0x025cf9 thru 0x028fe2 (0x32ea bytes) 000789 1787866101.730827 mapwrite 0x011a58 thru 0x01edc2 (0xd36b bytes) 000790 1787866101.774095 MIRROR_RESYNC 4 mirrors 000792 1787866103.679729 mapwrite 0x01d720 thru 0x0227f7 (0x50d8 bytes) 000793 1787866103.795655 mapread 0x03aa82 thru 0x03ffff (0x557e bytes) 000794 1787866103.813785 mapread 0x019332 thru 0x0282f8 (0xefc7 bytes) 000795 1787866103.850952 read 0x01a3c1 thru 0x01c1c0 (0x1e00 bytes) 000796 1787866103.862031 trunc from 0x0107ac to 0x019c1a (0x946f bytes) 000797 1787866103.881279 fallocate 0x036568 thru 0x03d317 (0x6db0 bytes) 000798 1787866103.887980 read 0x03b5b7 thru 0x03cb2c (0x1576 bytes) 000799 1787866103.890024 fallocate 0x033cb3 thru 0x0397a3 (0x5af1 bytes) 000800 1787866103.898728 mapread 0x032c4f thru 0x03f7f4 (0xcba6 bytes) 000801 1787866103.965376 write 0x00274b thru 0x00d6ab (0xaf61 bytes) 000802 1787866103.974882 mapread 0x020b6f thru 0x02282c (0x1cbe bytes) 000803 1787866103.978195 trunc from 0x025bd9 to 0x02b021 (0x5449 bytes) 000804 1787866103.993760 fallocate 0x01e2b7 thru 0x02d9e9 (0xf733 bytes) 000805 1787866104.010297 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x3d713d6a (stale) Mirror 10: 0x3d713d6a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x3d713d6a (stale) Mirror 10: 0x3d713d6a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000806 1787866104.412001 trunc from 0x040000 to 0x0147da (0x2b825 bytes) 000807 1787866104.439129 trunc from 0x0147db to 0x00df5e (0x687c bytes) 000808 1787866104.460194 fallocate 0x02fddb thru 0x03438d (0x45b3 bytes) 000809 1787866104.466713 punch from 0x017691 to 0x025c6d (0xe5dd bytes) 000810 1787866104.471010 mapwrite 0x02a0d5 thru 0x032efd (0x8e29 bytes) 000811 1787866104.575313 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3438e): Mirror 10: 0x898545a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3438e): Mirror 10: 0x898545a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3438e: skipped 000812 1787866104.941169 trunc from 0x03438e to 0x03d772 (0x93e5 bytes) 000813 1787866104.991161 write 0x037988 thru 0x03ffff (0x8678 bytes) 000814 1787866105.006388 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000816 1787866105.173070 fallocate 0x01eee7 thru 0x024bf6 (0x5d10 bytes) 000817 1787866105.181442 punch from 0x02dda0 to 0x031786 (0x39e7 bytes) 000818 1787866105.186866 trunc from 0x040000 to 0x018aae (0x27551 bytes) 000819 1787866105.211102 trunc from 0x018aaf to 0x023825 (0xad77 bytes) 000820 1787866105.233114 punch from 0x021c13 to 0x023825 (0x1c13 bytes) 000821 1787866105.238915 trunc from 0x023826 to 0x000d08 (0x22b1d bytes) 000824 1787866105.260319 fallocate 0x00e3ca thru 0x0160a6 (0x7cdd bytes) 000826 1787866105.270777 MIRROR_EXTEND to 2 mirrors 000827 1787866105.905752 fallocate 0x0187c2 thru 0x01a7f8 (0x2037 bytes) 000828 1787866105.975896 punch from 0x0004b9 to 0x000d08 (0x850 bytes) 000829 1787866105.988908 read 0x0000fb thru 0x000d08 (0xc0e bytes) 000830 1787866106.000175 trunc from 0x000d09 to 0x03bd68 (0x3b060 bytes) 000832 1787866106.059021 MIRROR_RESYNC 2 mirrors 000833 1787866106.775447 MIRROR_EXTEND to 3 mirrors 000835 1787866107.763981 trunc from 0x03bd69 to 0x03558f (0x67d9 bytes) 000836 1787866107.858033 trunc from 0x035590 to 0x02ba72 (0x9b1d bytes) 000837 1787866107.888730 trunc from 0x02ba73 to 0x00f600 (0x1c472 bytes) 000838 1787866107.921445 mapread 0x003021 thru 0x00e0e3 (0xb0c3 bytes) 000839 1787866107.961274 write 0x027cfa thru 0x0294b6 (0x17bd bytes) 000840 1787866107.965600 fallocate 0x02d517 thru 0x034aeb (0x75d5 bytes) 000842 1787866107.972594 trunc from 0x034aec to 0x01832d (0x1c7be bytes) 000843 1787866108.011776 trunc from 0x003ada to 0x013a45 (0xff6c bytes) 000844 1787866108.020476 write 0x0116ee thru 0x014960 (0x3273 bytes) 000845 1787866108.045278 fallocate 0x0041ae thru 0x01063c (0xc48f bytes) 000846 1787866108.051766 punch from 0x004e28 to 0x0089af (0x3b88 bytes) 000847 1787866108.066547 trunc from 0x01832e to 0x0158af (0x2a7e bytes) 000848 1787866108.089124 trunc from 0x0158b0 to 0x0092cb (0xc5e4 bytes) 000849 1787866108.107384 write 0x02798e thru 0x0350ee (0xd761 bytes) 000850 1787866108.111129 trunc from 0x01745e to 0x02671b (0xf2be bytes) 000851 1787866108.117421 punch from 0x0180bb to 0x02341e (0xb364 bytes) 000852 1787866108.123354 trunc from 0x0155ba to 0x016837 (0x127e bytes) 000853 1787866108.129146 write 0x017b4f thru 0x0187f7 (0xca9 bytes) 000854 1787866108.141584 write 0x0035a2 thru 0x00b2bc (0x7d1b bytes) 000855 1787866108.153908 mapread 0x00dce9 thru 0x0198b3 (0xbbcb bytes) 000856 1787866108.179092 trunc from 0x01334a to 0x01c65b (0x9312 bytes) 000857 1787866108.200442 trunc from 0x0350ef to 0x02769b (0xda53 bytes) 000858 1787866108.230432 fallocate 0x01c7ec thru 0x0276f9 (0xaf0e bytes) 000859 1787866108.235569 mapwrite 0x00af1f thru 0x010886 (0x5968 bytes) 000860 1787866108.257383 punch from 0x01da3d to 0x01f2da (0x189e bytes) 000861 1787866108.265085 write 0x028805 thru 0x031dff (0x95fb bytes) 000862 1787866108.267894 mapwrite 0x02b02d thru 0x0355e0 (0xa5b4 bytes) 000863 1787866108.387978 trunc from 0x0355e1 to 0x03ea2c (0x944c bytes) 000864 1787866108.426771 write 0x0014a7 thru 0x01023b (0xed95 bytes) 000865 1787866108.441384 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ea2d): Mirror 2: 0x526bb41c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ea2d): Mirror 2: 0x526bb41c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ea2d: skipped 000868 1787866108.953425 mapread 0x03e3c6 thru 0x03ea2c (0x667 bytes) 000869 1787866108.966072 mapread 0x02fd2c thru 0x039180 (0x9455 bytes) 000870 1787866109.041919 trunc from 0x01ad59 to 0x01b8e5 (0xb8d bytes) 000871 1787866109.048473 fallocate 0x02483f thru 0x033ee5 (0xf6a7 bytes) 000872 1787866109.060132 fallocate 0x029051 thru 0x02e485 (0x5435 bytes) 000873 1787866109.068172 mapwrite 0x02eb14 thru 0x03cd33 (0xe220 bytes) 000874 1787866109.091993 mapwrite 0x02f324 thru 0x03ab62 (0xb83f bytes) 000875 1787866109.143857 read 0x02771c thru 0x03470e (0xcff3 bytes) 000876 1787866109.161597 read 0x032ad7 thru 0x03ea2c (0xbf56 bytes) 000877 1787866109.162372 read 0x03e87c thru 0x03ea2c (0x1b1 bytes) 000878 1787866109.163080 mapwrite 0x017f5e thru 0x02276f (0xa812 bytes) 000880 1787866109.262840 trunc from 0x03ea2d to 0x024a8b (0x19fa1 bytes) 000881 1787866109.304995 mapwrite 0x00c1d4 thru 0x016c51 (0xaa7e bytes) 000882 1787866109.332781 fallocate 0x003f00 thru 0x0044bc (0x5bd bytes) 000883 1787866109.338929 fallocate 0x03c3ea thru 0x03ffff (0x3c16 bytes) 000884 1787866109.344965 read 0x0014ee thru 0x009ba4 (0x86b7 bytes) 000885 1787866109.346065 punch from 0x02d917 to 0x030d08 (0x33f2 bytes) 000886 1787866109.352122 mapread 0x00efae thru 0x0132b8 (0x430b bytes) 000887 1787866109.354090 read 0x03bde8 thru 0x03e83f (0x2a58 bytes) 000889 1787866109.383054 mapwrite 0x027afd thru 0x02aa00 (0x2f04 bytes) 000890 1787866109.409854 write 0x01b454 thru 0x02a5bf (0xf16c bytes) 000891 1787866109.411698 write 0x0253df thru 0x0297e9 (0x440b bytes) 000892 1787866109.416248 trunc from 0x040000 to 0x0012d2 (0x3ed2d bytes) 000893 1787866109.453008 fallocate 0x03f45b thru 0x03ffff (0xba5 bytes) 000894 1787866109.465104 fallocate 0x003b1e thru 0x010130 (0xc613 bytes) 000895 1787866109.476907 trunc from 0x00982c to 0x014d93 (0xb568 bytes) 000896 1787866109.509813 mapread 0x00c307 thru 0x013f37 (0x7c31 bytes) 000897 1787866109.587652 trunc from 0x03ffa7 to 0x03ffff (0x59 bytes) 000898 1787866109.593543 mapwrite 0x03580b thru 0x03ebc0 (0x93b6 bytes) 000899 1787866109.690418 fallocate 0x03bc4d thru 0x03fe2a (0x41de bytes) 000900 1787866109.696574 trunc from 0x040000 to 0x028716 (0x178e9 bytes) 000901 1787866109.713440 punch from 0x014869 to 0x01ef1b (0xa6b3 bytes) 000902 1787866109.720105 fallocate 0x0102ea thru 0x01f2b3 (0xefca bytes) 000903 1787866109.727002 read 0x00ea16 thru 0x012148 (0x3733 bytes) 000904 1787866109.733030 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x28717): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28717): Mirror 2: 0xea854303 (stale) CRC-32 checksum value for chunk [0, 0x28717): Mirror 1: 0xea854303 Mirror 3: 0xea854303 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28717: skipped 000905 1787866110.636815 read 0x01b09b thru 0x025975 (0xa8db bytes) 000907 1787866110.657288 trunc from 0x0109a5 to 0x0158b0 (0x4f0c bytes) 000908 1787866110.720112 write 0x036cc4 thru 0x03ffff (0x933c bytes) 000909 1787866110.741867 MIRROR_RESYNC 3 mirrors 000910 1787866111.856769 mapread 0x02ff0f thru 0x03ce8c (0xcf7e bytes) 000911 1787866111.944415 punch from 0x01dd83 to 0x023bf1 (0x5e6f bytes) 000912 1787866112.032764 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x6b5d55db (stale) Mirror 3: 0x6b5d55db (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6b5d55db Mirror 4: 0x6b5d55db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000913 1787866113.141024 trunc from 0x040000 to 0x02ff9e (0x10061 bytes) 000914 1787866113.249310 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ff9f): Mirror 2: 0x40cf2950 (stale) Mirror 3: 0x40cf2950 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ff9f): Mirror 2: 0x40cf2950 (stale) Mirror 3: 0x40cf2950 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ff9f: skipped 000916 1787866113.631472 punch from 0x028ad1 to 0x02bb41 (0x3071 bytes) 000917 1787866113.638309 mapread 0x01bdd0 thru 0x01d98d (0x1bbe bytes) 000918 1787866113.661504 fallocate 0x023f91 thru 0x02657b (0x25eb bytes) 000919 1787866113.666912 mapread 0x01ebd8 thru 0x02d2d6 (0xe6ff bytes) 000920 1787866113.678010 mapwrite 0x0012a7 thru 0x0016d2 (0x42c bytes) 000921 1787866113.729864 write 0x00b729 thru 0x00ff2c (0x4804 bytes) 000922 1787866113.742715 trunc from 0x01f68d to 0x0271ef (0x7b63 bytes) 000923 1787866113.768097 read 0x01274a thru 0x01da9d (0xb354 bytes) 000924 1787866113.783841 trunc from 0x02ff9f to 0x017057 (0x18f47 bytes) 000925 1787866113.841952 fallocate 0x0158d4 thru 0x016ef5 (0x1622 bytes) 000927 1787866113.852296 read 0x000f0e thru 0x00f599 (0xe68c bytes) 000929 1787866113.872922 punch from 0x002669 to 0x00f5bd (0xcf55 bytes) 000930 1787866113.911711 write 0x034cec thru 0x03ffff (0xb314 bytes) 000931 1787866113.920540 trunc from 0x00f3f2 to 0x0171ae (0x7dbd bytes) 000932 1787866113.928847 write 0x024ab7 thru 0x02fb23 (0xb06d bytes) 000933 1787866113.941378 mapwrite 0x02b875 thru 0x038ff8 (0xd784 bytes) 000934 1787866113.992626 mapwrite 0x0336ed thru 0x03f3b8 (0xbccc bytes) 000935 1787866114.024186 write 0x03d443 thru 0x03ffff (0x2bbd bytes) 000936 1787866114.025788 punch from 0x0302de to 0x03c2d2 (0xbff5 bytes) 000937 1787866114.035572 read 0x00c5bd thru 0x018d98 (0xc7dc bytes) 000938 1787866114.063176 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x6b5d55db (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x6b5d55db (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000939 1787866114.496958 mapread 0x02f127 thru 0x0380b0 (0x8f8a bytes) 000940 1787866114.531311 punch from 0x00d256 to 0x00eaca (0x1875 bytes) 000941 1787866114.537287 trunc from 0x0015de to 0x009346 (0x7d69 bytes) 000942 1787866114.543283 trunc from 0x040000 to 0x03c523 (0x3adc bytes) 000944 1787866114.574744 mapwrite 0x0386f9 thru 0x03d584 (0x4e8c bytes) 000945 1787866114.641782 trunc from 0x03d585 to 0x022f61 (0x1a623 bytes) 000947 1787866114.670527 fallocate 0x03177b thru 0x036878 (0x50fe bytes) 000948 1787866114.684402 read 0x019eac thru 0x022f61 (0x90b6 bytes) 000949 1787866114.736414 trunc from 0x022f62 to 0x012119 (0x10e48 bytes) 000950 1787866114.768084 fallocate 0x02f7ee thru 0x0330e1 (0x38f4 bytes) 000951 1787866114.779636 trunc from 0x00b70f to 0x012119 (0x6a0b bytes) 000952 1787866114.792342 punch from 0x00f58c to 0x012119 (0x2b8e bytes) 000953 1787866114.808418 fallocate 0x027dbc thru 0x02b2e5 (0x352a bytes) 000954 1787866114.836755 MIRROR_RESYNC 2 mirrors 000955 1787866115.831315 trunc from 0x01211a to 0x024555 (0x1243c bytes) 000956 1787866115.926853 write 0x01a9c1 thru 0x022dd1 (0x8411 bytes) 000957 1787866115.957617 trunc from 0x00b310 to 0x00cbae (0x189f bytes) 000958 1787866115.963980 read 0x000b78 thru 0x0026ad (0x1b36 bytes) 000959 1787866115.975626 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000961 1787866116.153597 fallocate 0x038526 thru 0x03ffff (0x7ada bytes) 000962 1787866116.160331 mapwrite 0x033499 thru 0x033f56 (0xabe bytes) 000963 1787866116.188333 punch from 0x00a1af to 0x018f91 (0xede3 bytes) 000964 1787866116.195781 punch from 0x02c645 to 0x03aa91 (0xe44d bytes) 000965 1787866116.203708 read 0x020b7c thru 0x02ea79 (0xdefe bytes) 000966 1787866116.231577 trunc from 0x03b493 to 0x03ffff (0x4b6d bytes) 000967 1787866116.237596 fallocate 0x013442 thru 0x01437d (0xf3c bytes) 000968 1787866116.248240 trunc from 0x039c35 to 0x03ffff (0x63cb bytes) 000969 1787866116.260391 fallocate 0x019808 thru 0x0204ee (0x6ce7 bytes) 000970 1787866116.266857 punch from 0x01e93c to 0x01fcf8 (0x13bd bytes) 000971 1787866116.275525 punch from 0x03bb51 to 0x03ffff (0x44af bytes) 000972 1787866116.291361 read 0x028eea thru 0x037342 (0xe459 bytes) 000973 1787866116.347992 write 0x02afef thru 0x0356c7 (0xa6d9 bytes) 000974 1787866116.355248 punch from 0x007a06 to 0x0167c8 (0xedc3 bytes) 000976 1787866116.362146 mapwrite 0x03acce thru 0x03ffff (0x5332 bytes) 000977 1787866116.389261 punch from 0x01bdf7 to 0x01de2a (0x2034 bytes) 000979 1787866116.396481 write 0x01bbca thru 0x0211e8 (0x561f bytes) 000980 1787866116.405623 fallocate 0x0352f8 thru 0x03d670 (0x8379 bytes) 000982 1787866116.414459 trunc from 0x03edca to 0x03ffff (0x1236 bytes) 000983 1787866116.423473 mapread 0x028b09 thru 0x02e37f (0x5877 bytes) 000985 1787866116.427412 trunc from 0x040000 to 0x00a5bd (0x35a42 bytes) 000986 1787866116.460650 trunc from 0x00417f to 0x0063e5 (0x2267 bytes) 000988 1787866116.468722 write 0x017a03 thru 0x024daf (0xd3ad bytes) 000989 1787866116.472738 punch from 0x01a1f6 to 0x024daf (0xabba bytes) 000990 1787866116.493267 mapwrite 0x004891 thru 0x010e32 (0xc5a2 bytes) 000991 1787866116.595803 write 0x03b6e4 thru 0x03ffff (0x491c bytes) 000992 1787866116.597660 read 0x013b87 thru 0x015c80 (0x20fa bytes) 000993 1787866116.614261 write 0x017b17 thru 0x01d01e (0x5508 bytes) 000994 1787866116.622363 write 0x00ec8f thru 0x010fb2 (0x2324 bytes) 000995 1787866116.628355 trunc from 0x040000 to 0x026eef (0x19110 bytes) 000997 1787866116.679755 punch from 0x0202ff to 0x0263e2 (0x60e4 bytes) 000998 1787866116.698430 punch from 0x009a99 to 0x010174 (0x66dc bytes) 000999 1787866116.722125 mapwrite 0x0184e4 thru 0x01b778 (0x3295 bytes) 001000 1787866116.740150 read 0x008334 thru 0x00a49d (0x216a bytes) All operations completed A-OK! PASS 70a (121s) == sanity-flr test 71: check mirror extend parent fid ==== 17:28:42 (1787866122) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x1bf:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (13s) == sanity-flr test 72: check append on FLR file ========== 17:28:55 (1787866135) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.30718 s, 6.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164774 s, 6.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d6:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d6:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f7:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 17:29:05 (1787866145) Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre2 Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1511072 .. done Extending file size to 2078976 .. Extending file size to 3197184 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7385632 .. Extending file size to 8238144 .. Extending file size to 7841792 .. Extending file size to 8309856 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8363136 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9009216 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8458656 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9071584 .. Waiting 62643 62644 62645 62648 62651 done 192.168.204.120@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg420-client.virtnet /mnt/lustre2 (opts:) 192.168.204.120@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg420-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (92s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 17:30:36 (1787866236) Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre2 Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6432256 .. Extending file size to 787104 .. Extending file size to 1489856 .. Extending file size to 8709120 .. Extending file size to 8326784 .. Extending file size to 9199648 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8988992 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 64274 64275 64276 64278 64279 done 192.168.204.120@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg420-client.virtnet /mnt/lustre2 (opts:) 192.168.204.120@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg420-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (82s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 17:31:58 (1787866318) Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2516 1268788 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1988 1269316 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17908 3589112 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25800 3581220 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43708 7170332 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2516 1268788 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1988 1269316 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17908 3589112 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25800 3581220 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43708 7170332 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.144487 s, 14.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.188149 s, 11.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x2ea5bea9 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.204.120@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg420-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 17:32:18 (1787866338) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100913 s, 20.8 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 17:32:24 (1787866344) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ce:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f2:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f2:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f3:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x312:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 17:32:32 (1787866352) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.153818 s, 13.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x313:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x314:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 17:32:40 (1787866360) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.133216 s, 15.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x49:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x49:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x49:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x49:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4a:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 17:32:50 (1787866370) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.117901 s, 17.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x315:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x315:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x316:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 17:33:24 (1787866404) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg420-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925028 s, 11.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x318:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x317:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x318:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x317:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f7:0x0] } PASS 204d (57s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 17:34:21 (1787866461) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg420-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0980778 s, 21.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x319:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x319:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (55s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 17:35:16 (1787866516) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg420-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg420-server: Pool lustre.test_204f created Adding targets to pool oleg420-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg420-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.122168 s, 17.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg420-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg420-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg420-server: Pool lustre.test_204f destroyed PASS 204f (71s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 17:36:28 (1787866588) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 17:36:38 (1787866598) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 17:36:48 (1787866608) oleg420-server: Pool lustre.test_206 created Adding targets to pool oleg420-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg420-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg420-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg420-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg420-server: Pool lustre.test_206 destroyed PASS 206 (31s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 17:37:19 (1787866639) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778818 s, 13.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432147 s, 24.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xf3b16f30 Mirror 3: 0xf3b16f30 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 17:37:32 (1787866652) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 17:37:35 (1787866655) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 17:37:39 (1787866659) Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0725278 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2100000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1fa0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1fa0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1fa0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x14a0000 Read this (4 bytes): BBBB 192.168.204.120@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg420-client.virtnet /mnt/lustre2 (opts:) PASS 209a (38s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 17:38:17 (1787866697) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.021714 s, 377 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 17:38:26 (1787866706) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058801 s, 17.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1ef lmm_fid: [0x200000405:0x1ef:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1ef lmm_fid: [0x200000405:0x1ef:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x304:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611631 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 17:38:27.000000000 -0400 Modify: 2026-08-27 17:38:27.000000000 -0400 Change: 2026-08-27 17:38:29.000000000 -0400 Birth: 2026-08-27 17:38:28.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 17:38:34 (1787866714) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480984 s, 21.8 MB/s oleg420-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg420-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg420-server: O/280000401/d5/305: File not found by ext2_lookup PASS 210b (21s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 17:38:55 (1787866735) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.281389 s, 146 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0281113 s, 146 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 17:38 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f4:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 17:38 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 17:39:04 (1787866744) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 17:39:12 (1787866752) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1830 sec =========== 17:39:21 (1787866761) === sanity-flr: start cleanup 17:39:23 (1787866763) === === sanity-flr: finish cleanup 17:39:32 (1787866772) ===