-----============= acceptance-small: sanity-flr ============----- Wed Aug 26 20:32:48 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg431-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 20:33:04 (1787790784) === oleg431-client.virtnet: executing check_config_client /mnt/lustre oleg431-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg431-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff99318a2d0800.idle_timeout=debug osc.lustre-OST0001-osc-ffff99318a2d0800.idle_timeout=debug disable quota as required oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 20:33:39 (1787790819) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8211] == sanity-flr test 0a: lfs mirror create with -N option == 20:33:41 (1787790821) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:33:56 (1787790836) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:33:59 (1787790839) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 20:34:02 (1787790842) PASS 0d (21s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:34:23 (1787790863) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:34:27 (1787790867) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 20:34:30 (1787790870) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff99318a2d0800.stats=clear osc.lustre-OST0001-osc-ffff99318a2d0800.stats=clear PASS 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:34:37 (1787790877) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:34:47 (1787790887) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:34:54 (1787790894) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 20:34:57 (1787790897) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 20:35:04 (1787790904) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 20:35:12 (1787790912) PASS 2 (5s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:35:17 (1787790917) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:35:24 (1787790924) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:35:31 (1787790931) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 20:35:40 (1787790940) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:35:46 (1787790946) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 1.17967 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106799 s, 9.8 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:35:55 (1787790955) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 0.317897 s, 46.2 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 20:36:04 (1787790964) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (8s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:36:13 (1787790973) Stopping /mnt/lustre-ost1 (opts:) on oleg431-server oleg431-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99318a2d0800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99318a2d0800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg431-server: oleg431-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg431-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg431-server oleg431-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99318a2d0800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99318a2d0800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg431-server: oleg431-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg431-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (62s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:37:14 (1787791034) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.136527 s, 15.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg431-server oleg431-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99318a2d0800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99318a2d0800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg431-server: oleg431-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg431-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (35s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:37:49 (1787791069) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg431-server oleg431-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99318a2d0800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99318a2d0800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg431-server: oleg431-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg431-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg431-server oleg431-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99318a2d0800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99318a2d0800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg431-server: oleg431-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg431-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (71s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:39:00 (1787791140) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.86791 s, 18.3 MB/s -rw-r--r-- 1 root root 52428800 Aug 26 20:39 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.707s user 0m0.002s sys 0m0.187s Stopping /mnt/lustre-ost1 (opts:) on oleg431-server oleg431-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99318a2d0800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99318a2d0800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.933s user 0m0.001s sys 0m0.208s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg431-server: oleg431-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg431-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg431-server oleg431-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99318a2d0800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99318a2d0800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.099s user 0m0.001s sys 0m0.191s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg431-server: oleg431-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg431-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg431-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99318a2d0800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99318a2d0800.ost_server_uuid in FULL state after 0 sec PASS 33b (79s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:40:20 (1787791220) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:40:22 (1787791222) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:40:24 (1787791224) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 20:40:27 (1787791227) PASS 35 (5s) == sanity-flr test 36a: write to mirrored files ========== 20:40:32 (1787791232) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.486456 s, 25.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (9s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:40:41 (1787791241) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386826 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03959 s, 26.5 MB/s PASS 36b (5s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:40:46 (1787791246) PASS 36c (6s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:40:52 (1787791252) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00206678 s, 495 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00153113 s, 669 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000985367 s, 1.0 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00115066 s, 890 kB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 20:41:00 (1787791260) create test files with size 15730157 .. 84bfb6c1eea3e74ce1c95699cae45bff - 842cbe7510857c3c78a05c6dcaff44ce - 94e5296d8016eaa3c45f6bd568123704 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre PASS 37 (18s) == sanity-flr test 38: resync ============================ 20:41:18 (1787791278) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:41:30 (1787791290) PASS 39 (5s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:41:35 (1787791295) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465627 s, 22.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xab:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294628 s, 35.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xac:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xad:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xce:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (7s) == sanity-flr test 41: lfs mirror resync check =========== 20:41:42 (1787791302) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0874571 s, 24.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.293354 s, 14.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123715 s, 17.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (13s) == sanity-flr test 42: lfs mirror verify ================= 20:41:56 (1787791316) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 20:41:59 (1787791319) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:42:02 (1787791322) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497668 s, 21.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 43c: read prefer uncompressed mirror == 20:42:09 (1787791329) llite.lustre-ffff993183a3d800.enable_compression=1 striped dir -i1 -c2 -H crush2 /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (6s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 20:42:15 (1787791335) llite.lustre-ffff993183a3d800.enable_compression=1 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590483 s, 17.8 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257613 s, 40.7 MB/s **verify components PASS 43d (7s) == sanity-flr test 44a: lfs mirror split check =========== 20:42:22 (1787791342) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.275925 s, 11.4 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:42:32 (1787791352) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xda:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xba:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xba:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:42:41 (1787791361) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.327573 s, 32.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x9:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:42:51 (1787791371) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff993183fbe800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.247099 s, 42.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff993186608800.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:43:04 (1787791384) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (33s) == sanity-flr test 46: Verify setstripe --copy option ==== 20:43:37 (1787791417) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (30s) == sanity-flr test 47: Verify mirror obj alloc =========== 20:44:07 (1787791447) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 20:44:10 (1787791450) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.18907 s, 16.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.13141 s, 23.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x71f97a1c (nosync,stale) Mirror 2: 0x71f97a1c (nosync,stale) ** verify mirror 2 doesn't change original checksum: 9ac8903f6b6565f905282ab765f6a414 - mirror 1 checksum: 04b3b62a03fd5f12f1931ddb63155e40 - mirror 2 checksum: 9ac8903f6b6565f905282ab765f6a414 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-26T20:44:12-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-26T20:44:12-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 9ac8903f6b6565f905282ab765f6a414 - mirror 1 checksum: 04b3b62a03fd5f12f1931ddb63155e40 - mirror 2 checksum: 04b3b62a03fd5f12f1931ddb63155e40 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-26T20:44:12-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-26T20:44:12-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } PASS 48 (8s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 20:44:18 (1787791458) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311479 s, 33.7 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (6s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:44:24 (1787791464) Failing mds1 on oleg431-server Stopping /mnt/lustre-mds1 (opts:) on oleg431-server 20:44:35 (1787791475) shut down facet: mds1 facet_host: oleg431-server facet_failover_host: oleg431-server Failover mds1 to oleg431-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-MDT0000 20:44:55 (1787791495) targets are mounted 20:44:55 (1787791495) facet_failover done oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317971 s, 33.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } PASS 50A (45s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:45:09 (1787791509) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0424338 s, 483 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0200852 s, 1.0 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (8s) == sanity-flr test 50b: mirror rsync handles sparseness == 20:45:17 (1787791517) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0462621 s, 443 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00793624 s, 2.6 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (7s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:45:25 (1787791525) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0064662 s, 2.5 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 20:45:30 (1787791530) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0445268 s, 460 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (5s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:45:35 (1787791535) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00516512 s, 793 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0403982 s, 101 kB/s PASS 60a (6s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:45:41 (1787791541) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) PASS 60b (19s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:46:00 (1787791560) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (24s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:46:25 (1787791585) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (21s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:46:45 (1787791605) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (21s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:47:06 (1787791626) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (5s) == sanity-flr test 70: mirror create and split race ====== 20:47:12 (1787791632) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 58790 mirror split pid 58791 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 58790 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 58791 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 20:48:17 (1787791697) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 1699 truncating to largest ever: 0x2927a 000001 1787791699.298191 trunc from 00000000 to 0x029279 (0x2927a bytes) 000002 1787791699.317409 mapread 0x01c532 thru 0x0240ec (0x7bbb bytes) 000003 1787791699.341434 fallocate 0x011dba thru 0x016749 (0x4990 bytes) 000005 1787791699.349405 mapwrite 0x01b438 thru 0x0246ae (0x9277 bytes) truncating to largest ever: 0x3fbea 000007 1787791699.369196 trunc from 0x02927a to 0x03fbe9 (0x16970 bytes) 000009 1787791699.386255 write 0x01a5c2 thru 0x02602b (0xba6a bytes) 000011 1787791699.388568 read 0x0282b4 thru 0x031bc5 (0x9912 bytes) 000012 1787791699.416232 trunc from 0x03fbea to 0x038edb (0x6d0e bytes) 000014 1787791699.431559 trunc from 0x019693 to 0x0206de (0x704c bytes) 000015 1787791699.453423 punch from 0x014c5f to 0x01bd07 (0x70a9 bytes) 000016 1787791699.458461 trunc from 0x02fa30 to 0x03438a (0x495b bytes) 000018 1787791699.463570 mapread 0x00ab9f thru 0x00d57e (0x29e0 bytes) 000019 1787791699.483347 write 0x00aaab thru 0x016ff4 (0xc54a bytes) 000021 1787791699.494492 mapread 0x01c510 thru 0x0209a9 (0x449a bytes) 000022 1787791699.514527 trunc from 0x038edc to 0x0081dc (0x30cff bytes) 000023 1787791699.543456 read 0x0035c6 thru 0x003f23 (0x95e bytes) 000024 1787791699.544060 trunc from 0x004cdf to 0x0081dc (0x34fe bytes) 000025 1787791699.564602 MIRROR_EXTEND to 2 mirrors 000026 1787791700.163246 punch from 0x000442 to 0x0081dc (0x7d9b bytes) 000027 1787791700.215918 mapread 0x002627 thru 0x0081dc (0x5bb6 bytes) 000028 1787791700.258565 punch from 0x002016 to 0x0081dc (0x61c7 bytes) 000029 1787791700.266097 mapread 0x00385f thru 0x0081dc (0x497e bytes) 000030 1787791700.277753 punch from 0x000590 to 0x0081dc (0x7c4d bytes) 000031 1787791700.283584 trunc from 0x0011b2 to 0x0081dc (0x702b bytes) fallocating to largest ever: 0x40000 000032 1787791700.294184 fallocate 0x03fa8b thru 0x03ffff (0x575 bytes) 000033 1787791700.299493 punch from 0x0033e2 to 0x0081dc (0x4dfb bytes) 000034 1787791700.308617 fallocate 0x003dd6 thru 0x010730 (0xc95b bytes) 000035 1787791700.320511 trunc from 0x008caf to 0x010730 (0x7a82 bytes) 000036 1787791700.327910 mapwrite 0x001d2f thru 0x01039c (0xe66e bytes) 000037 1787791700.391917 trunc from 0x00c026 to 0x010730 (0x470b bytes) 000039 1787791700.404357 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x10731): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10731): Mirror 2: 0xcf9b8786 (stale) CRC-32 checksum value for chunk [0, 0x10731): Mirror 1: 0xba2a3cdf Mirror 3: 0xba2a3cdf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10731: skipped 000040 1787791701.149225 mapwrite 0x01ae9e thru 0x0235de (0x8741 bytes) 000041 1787791701.283095 mapread 0x003efe thru 0x0125e6 (0xe6e9 bytes) 000042 1787791701.307890 mapwrite 0x03c1fa thru 0x03f310 (0x3117 bytes) 000043 1787791701.384381 fallocate 0x0102ee thru 0x018854 (0x8567 bytes) 000045 1787791701.390091 trunc from 0x03f311 to 0x0122d1 (0x2d03f bytes) 000046 1787791701.409365 trunc from 0x0122d2 to 0x02eb4d (0x1c87c bytes) 000047 1787791701.434250 fallocate 0x0108a1 thru 0x01cb52 (0xc2b2 bytes) 000048 1787791701.440314 read 0x020ef6 thru 0x02eb4d (0xdc58 bytes) 000049 1787791701.451849 trunc from 0x0236e1 to 0x02eb4d (0xb46d bytes) 000050 1787791701.458362 mapread 0x008f4e thru 0x00d616 (0x46c9 bytes) 000051 1787791701.459293 mapwrite 0x00f256 thru 0x016609 (0x73b4 bytes) 000052 1787791701.500121 fallocate 0x0369ad thru 0x03ffff (0x9653 bytes) 000053 1787791701.505470 mapread 0x013839 thru 0x0148e0 (0x10a8 bytes) 000054 1787791701.507273 mapwrite 0x0064ee thru 0x0087de (0x22f1 bytes) 000055 1787791701.522111 fallocate 0x029c90 thru 0x02badb (0x1e4c bytes) 000056 1787791701.527380 fallocate 0x035823 thru 0x037b9f (0x237d bytes) 000058 1787791701.533908 mapread 0x002995 thru 0x00fb1a (0xd186 bytes) 000060 1787791701.536330 mapwrite 0x036ad5 thru 0x03ffff (0x952b bytes) 000061 1787791701.592389 mapwrite 0x013042 thru 0x01f87d (0xc83c bytes) 000062 1787791701.617188 trunc from 0x040000 to 0x01f33b (0x20cc4 bytes) 000063 1787791701.637696 mapwrite 0x0286e5 thru 0x0317e4 (0x9100 bytes) 000064 1787791701.716677 mapwrite 0x0262d1 thru 0x027fc9 (0x1cf9 bytes) 000065 1787791701.745454 trunc from 0x02e75d to 0x0317e4 (0x3088 bytes) 000066 1787791701.750517 trunc from 0x0209c9 to 0x0235bb (0x2bf3 bytes) 000067 1787791701.755815 trunc from 0x01230a to 0x019fa1 (0x7c98 bytes) 000068 1787791701.767140 fallocate 0x006164 thru 0x00b838 (0x56d5 bytes) 000069 1787791701.785543 fallocate 0x01b61a thru 0x024cb8 (0x969f bytes) 000070 1787791701.798158 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x317e5): Mirror 3: 0x798d054b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x317e5): Mirror 3: 0x798d054b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x317e5: skipped 000071 1787791702.144051 fallocate 0x02d0ac thru 0x030be8 (0x3b3d bytes) 000072 1787791702.150427 fallocate 0x039a86 thru 0x03ffff (0x657a bytes) 000074 1787791702.165167 write 0x007e97 thru 0x017bad (0xfd17 bytes) 000075 1787791702.199457 mapwrite 0x0055b1 thru 0x0130b9 (0xdb09 bytes) 000076 1787791702.266729 trunc from 0x0372be to 0x03ffff (0x8d42 bytes) 000077 1787791702.275338 mapread 0x0243bd thru 0x02e5cd (0xa211 bytes) 000078 1787791702.310021 punch from 0x03ad1f to 0x03ffff (0x52e1 bytes) 000079 1787791702.320128 write 0x023a6b thru 0x02ae9c (0x7432 bytes) 000080 1787791702.321483 trunc from 0x03a310 to 0x03ffff (0x5cf0 bytes) 000082 1787791702.335322 fallocate 0x00d4d3 thru 0x01a324 (0xce52 bytes) 000083 1787791702.341407 punch from 0x03e291 to 0x03ffff (0x1d6f bytes) 000084 1787791702.346561 trunc from 0x00e7b0 to 0x015307 (0x6b58 bytes) 000085 1787791702.379871 punch from 0x00534c to 0x00a70a (0x53bf bytes) 000086 1787791702.389255 mapread 0x017205 thru 0x01bd3b (0x4b37 bytes) 000087 1787791702.423212 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e3:0x0]: Invalid argument (22) 000088 1787791702.631560 trunc from 0x00c537 to 0x017dd3 (0xb89d bytes) 000089 1787791702.639728 trunc from 0x040000 to 0x0161b1 (0x29e4e bytes) 000090 1787791702.674123 fallocate 0x002d90 thru 0x004523 (0x1794 bytes) 000092 1787791702.687344 trunc from 0x00e273 to 0x0161b1 (0x7f3f bytes) 000093 1787791702.700720 mapwrite 0x010d80 thru 0x016cfd (0x5f7e bytes) 000094 1787791702.785788 fallocate 0x02b741 thru 0x039063 (0xd923 bytes) 000095 1787791702.798320 write 0x000ae3 thru 0x0066f1 (0x5c0f bytes) 000096 1787791702.810889 trunc from 0x039064 to 0x03aa71 (0x1a0e bytes) 000097 1787791702.839157 mapread 0x01922c thru 0x0211e8 (0x7fbd bytes) 000098 1787791702.855464 MIRROR_EXTEND to 2 mirrors 000099 1787791703.552231 trunc from 0x026e66 to 0x02e14b (0x72e6 bytes) 000101 1787791703.613518 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3aa72): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aa72): Mirror 2: 0x15bd1f98 (stale) CRC-32 checksum value for chunk [0, 0x3aa72): Mirror 1: 0x15bd1f98 Mirror 3: 0x15bd1f98 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aa72: skipped 000102 1787791704.562618 trunc from 0x03aa72 to 0x02a36a (0x10707 bytes) 000103 1787791704.613375 read 0x01a48d thru 0x029aa6 (0xf61a bytes) 000104 1787791704.631521 trunc from 0x02a36b to 0x02a225 (0x145 bytes) 000105 1787791704.664567 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a226): Mirror 3: 0x104bb0bf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a226): Mirror 3: 0x104bb0bf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a226: skipped 000106 1787791705.018886 read 0x01372c thru 0x02020b (0xcae0 bytes) 000107 1787791705.029304 trunc from 0x0299a9 to 0x02a225 (0x87d bytes) 000108 1787791705.034819 mapwrite 0x01cbb4 thru 0x024356 (0x77a3 bytes) 000109 1787791705.073290 mapwrite 0x02f599 thru 0x0333a4 (0x3e0c bytes) 000111 1787791705.142588 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e3:0x0]: Invalid argument (22) 000112 1787791705.304511 fallocate 0x0158a6 thru 0x01976d (0x3ec8 bytes) 000113 1787791705.314482 mapread 0x02663b thru 0x0289de (0x23a4 bytes) 000114 1787791705.341318 mapwrite 0x003c03 thru 0x0059ee (0x1dec bytes) 000115 1787791705.369218 mapread 0x000d33 thru 0x0034c9 (0x2797 bytes) 000116 1787791705.378061 mapwrite 0x0376d5 thru 0x03ffff (0x892b bytes) 000118 1787791705.452064 fallocate 0x01061b thru 0x01eeba (0xe8a0 bytes) 000119 1787791705.458984 mapwrite 0x02b1a8 thru 0x02e630 (0x3489 bytes) 000121 1787791705.482652 trunc from 0x040000 to 0x005a10 (0x3a5ef bytes) 000122 1787791705.501850 punch from 0x001fe5 to 0x0031e7 (0x1203 bytes) 000123 1787791705.507598 trunc from 0x0050ad to 0x005a10 (0x964 bytes) 000124 1787791705.514892 write 0x03cc9d thru 0x03ffff (0x3363 bytes) 000125 1787791705.517187 write 0x03e0a4 thru 0x03ffff (0x1f5c bytes) 000129 1787791705.529908 read 0x032549 thru 0x03446b (0x1f23 bytes) 000131 1787791705.540065 fallocate 0x033967 thru 0x03d53e (0x9bd8 bytes) 000132 1787791705.544106 mapwrite 0x005a48 thru 0x00e857 (0x8e10 bytes) 000133 1787791705.588163 write 0x03d3b4 thru 0x03ffff (0x2c4c bytes) 000134 1787791705.589333 mapread 0x026719 thru 0x03009e (0x9986 bytes) 000135 1787791705.612884 read 0x01f361 thru 0x021447 (0x20e7 bytes) 000136 1787791705.616913 mapwrite 0x023813 thru 0x03033f (0xcb2d bytes) 000138 1787791705.650303 mapwrite 0x03c1e4 thru 0x03cd44 (0xb61 bytes) 000139 1787791705.661955 write 0x0101dd thru 0x01d847 (0xd66b bytes) 000140 1787791705.669827 trunc from 0x040000 to 0x03678e (0x9871 bytes) 000141 1787791705.693164 trunc from 0x03678f to 0x03ff0a (0x977c bytes) 000142 1787791705.720594 read 0x017475 thru 0x019712 (0x229e bytes) 000143 1787791705.722681 read 0x0190f4 thru 0x021170 (0x807d bytes) 000145 1787791705.737773 punch from 0x005c45 to 0x010444 (0xa800 bytes) 000146 1787791705.756859 mapread 0x001397 thru 0x00e89b (0xd505 bytes) 000147 1787791705.814253 mapwrite 0x031ee6 thru 0x034484 (0x259f bytes) 000148 1787791705.828693 trunc from 0x03ff0b to 0x0232c4 (0x1cc46 bytes) 000150 1787791705.849034 mapread 0x013a85 thru 0x0232c4 (0xf840 bytes) 000151 1787791705.850509 mapwrite 0x009a73 thru 0x01963f (0xfbcd bytes) 000152 1787791705.893501 punch from 0x01a285 to 0x0232c4 (0x9040 bytes) 000153 1787791705.900673 read 0x016d52 thru 0x0232c4 (0xc573 bytes) 000155 1787791705.913180 trunc from 0x019f07 to 0x01af70 (0x106a bytes) 000156 1787791705.926352 read 0x00cb3c thru 0x0167d6 (0x9c9b bytes) 000157 1787791705.927144 write 0x02740b thru 0x02b1f5 (0x3deb bytes) 000158 1787791705.930223 write 0x0274ae thru 0x0357bd (0xe310 bytes) 000159 1787791705.932712 trunc from 0x0357be to 0x02ca44 (0x8d79 bytes) 000160 1787791705.948301 write 0x028760 thru 0x02c5e5 (0x3e86 bytes) 000161 1787791705.949588 punch from 0x00c61d to 0x017c1c (0xb600 bytes) 000162 1787791705.956430 trunc from 0x02ca45 to 0x02037d (0xc6c7 bytes) 000163 1787791705.974634 trunc from 0x01b2fb to 0x02037d (0x5083 bytes) 000164 1787791705.982369 read 0x00ba49 thru 0x017d75 (0xc32d bytes) 000166 1787791706.001959 mapwrite 0x01a0cf thru 0x01a9e6 (0x918 bytes) 000167 1787791706.029261 trunc from 0x01e19e to 0x02037d (0x21e0 bytes) 000168 1787791706.036204 mapread 0x00ef3d thru 0x01b609 (0xc6cd bytes) 000169 1787791706.037332 fallocate 0x018cfe thru 0x0266d4 (0xd9d7 bytes) 000171 1787791706.046678 trunc from 0x0266d5 to 0x0284df (0x1e0b bytes) 000172 1787791706.071376 mapread 0x01ec7e thru 0x0284df (0x9862 bytes) 000173 1787791706.098536 write 0x027324 thru 0x029ad2 (0x27af bytes) 000174 1787791706.101836 mapread 0x01faae thru 0x029ad2 (0xa025 bytes) 000175 1787791706.103370 write 0x03e229 thru 0x03fc46 (0x1a1e bytes) 000176 1787791706.104414 read 0x03fb87 thru 0x03fc46 (0xc0 bytes) 000178 1787791706.105563 punch from 0x00da68 to 0x01c18c (0xe725 bytes) 000179 1787791706.110058 trunc from 0x02e7f7 to 0x030748 (0x1f52 bytes) 000180 1787791706.119124 mapread 0x01b05f thru 0x01f373 (0x4315 bytes) 000181 1787791706.132142 mapread 0x018f50 thru 0x022434 (0x94e5 bytes) 000183 1787791706.137320 trunc from 0x035c9a to 0x03d833 (0x7b9a bytes) 000185 1787791706.144830 trunc from 0x03fc47 to 0x01d854 (0x223f2 bytes) 000186 1787791706.177047 fallocate 0x01ed8b thru 0x02b5c7 (0xc83d bytes) 000189 1787791706.191374 mapread 0x0121af thru 0x0161db (0x402d bytes) 000191 1787791706.193106 write 0x005007 thru 0x00abbd (0x5bb7 bytes) 000192 1787791706.195905 trunc from 0x0059b5 to 0x01597a (0xffc6 bytes) 000194 1787791706.230733 trunc from 0x01cb9d to 0x01d854 (0xcb8 bytes) 000195 1787791706.237885 read 0x0173d8 thru 0x01d854 (0x647d bytes) 000196 1787791706.254105 punch from 0x012931 to 0x01d854 (0xaf24 bytes) 000197 1787791706.262584 read 0x0001df thru 0x00c10b (0xbf2d bytes) 000198 1787791706.263852 trunc from 0x003c4e to 0x00c11a (0x84cd bytes) 000199 1787791706.271257 mapread 0x008d87 thru 0x01338a (0xa604 bytes) 000202 1787791706.297897 punch from 0x0155b9 to 0x01d854 (0x829c bytes) 000203 1787791706.311522 MIRROR_EXTEND to 2 mirrors 000206 1787791707.002623 MIRROR_RESYNC 2 mirrors 000207 1787791707.366661 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e3:0x0]: Invalid argument (22) 000208 1787791707.499906 trunc from 0x01d855 to 0x017019 (0x683b bytes) 000209 1787791707.552515 trunc from 0x01701a to 0x035ab4 (0x1ea9b bytes) 000210 1787791707.571515 write 0x012dcc thru 0x01c7e5 (0x9a1a bytes) 000211 1787791707.582504 trunc from 0x035ab5 to 0x0091b9 (0x2c8fb bytes) 000212 1787791707.605852 trunc from 0x0026ca to 0x0091b9 (0x6af0 bytes) 000213 1787791707.614436 write 0x029fd4 thru 0x02dc90 (0x3cbd bytes) 000214 1787791707.615813 read 0x01918c thru 0x021323 (0x8198 bytes) 000215 1787791707.635148 trunc from 0x0239e3 to 0x027ad5 (0x40f3 bytes) 000216 1787791707.640827 trunc from 0x014a85 to 0x0205d6 (0xbb52 bytes) 000217 1787791707.652222 fallocate 0x034a0c thru 0x03f590 (0xab85 bytes) 000218 1787791707.659355 fallocate 0x034b34 thru 0x03ffff (0xb4cc bytes) 000219 1787791707.675327 trunc from 0x040000 to 0x0075d7 (0x38a28 bytes) 000220 1787791707.698308 write 0x01478e thru 0x0196c2 (0x4f35 bytes) 000221 1787791707.699519 fallocate 0x007aaa thru 0x0162fa (0xe851 bytes) 000222 1787791707.706743 fallocate 0x02fa9e thru 0x034f87 (0x54ea bytes) 000223 1787791707.714287 trunc from 0x01198d to 0x01666e (0x4ce2 bytes) 000224 1787791707.736259 punch from 0x018798 to 0x0196c2 (0xf2b bytes) 000225 1787791707.749500 fallocate 0x017e3c thru 0x0203a8 (0x856d bytes) 000226 1787791707.766579 MIRROR_EXTEND to 2 mirrors 000227 1787791708.516620 read 0x019629 thru 0x0196c2 (0x9a bytes) 000228 1787791708.524796 trunc from 0x0196c3 to 0x00145c (0x18266 bytes) 000229 1787791708.595693 read 0x0005fd thru 0x00145c (0xe60 bytes) 000230 1787791708.606170 punch from 0x00123e to 0x00141b (0x1de bytes) 000231 1787791708.614977 trunc from 0x00145d to 0x02bea8 (0x2aa4c bytes) 000232 1787791708.652796 read 0x02557b thru 0x02bea8 (0x692e bytes) 000233 1787791708.671432 read 0x023664 thru 0x02bea8 (0x8845 bytes) 000234 1787791708.673338 fallocate 0x002b89 thru 0x00ace4 (0x815c bytes) 000235 1787791708.679614 fallocate 0x00b7eb thru 0x00c2a7 (0xabd bytes) 000236 1787791708.698055 MIRROR_RESYNC 2 mirrors 000237 1787791709.378843 mapread 0x021219 thru 0x02bea8 (0xac90 bytes) 000238 1787791709.405347 trunc from 0x02bea9 to 0x02310d (0x8d9b bytes) 000239 1787791709.465342 trunc from 0x02310e to 0x0363ff (0x132f2 bytes) 000240 1787791709.515446 mapread 0x007dc5 thru 0x00e506 (0x6742 bytes) 000241 1787791709.533784 read 0x01fd28 thru 0x020c7f (0xf58 bytes) 000242 1787791709.544743 write 0x038125 thru 0x038d6b (0xc47 bytes) 000243 1787791709.548433 fallocate 0x02640f thru 0x0359b8 (0xf5aa bytes) 000244 1787791709.563992 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e3:0x0]: Invalid argument (22) 000246 1787791709.703248 mapwrite 0x0342cc thru 0x03ffff (0xbd34 bytes) 000247 1787791709.783280 write 0x01c044 thru 0x021977 (0x5934 bytes) 000248 1787791709.814195 mapread 0x01fe39 thru 0x025bd7 (0x5d9f bytes) 000250 1787791709.815618 write 0x019034 thru 0x02751b (0xe4e8 bytes) 000251 1787791709.832899 punch from 0x00f7b0 to 0x01ba26 (0xc277 bytes) 000252 1787791709.856816 mapread 0x005720 thru 0x0147f7 (0xf0d8 bytes) 000253 1787791709.902623 write 0x0364a8 thru 0x036f32 (0xa8b bytes) 000256 1787791709.907130 fallocate 0x02d0c6 thru 0x02e98e (0x18c9 bytes) 000257 1787791709.921996 write 0x0291cc thru 0x032d71 (0x9ba6 bytes) 000258 1787791709.924130 trunc from 0x01a163 to 0x022e9a (0x8d38 bytes) 000259 1787791709.935144 write 0x00df50 thru 0x01817a (0xa22b bytes) 000260 1787791709.940641 mapread 0x0018ab thru 0x009327 (0x7a7d bytes) 000262 1787791709.947147 read 0x03e6ec thru 0x03ff6a (0x187f bytes) 000263 1787791709.948665 mapwrite 0x0055a3 thru 0x007e46 (0x28a4 bytes) 000264 1787791709.978595 fallocate 0x01bda8 thru 0x029c45 (0xde9e bytes) 000265 1787791709.995718 write 0x036ffe thru 0x03f04e (0x8051 bytes) 000266 1787791709.998162 mapread 0x02b1e1 thru 0x02cf9a (0x1dba bytes) 000269 1787791710.003787 trunc from 0x00f6f4 to 0x0152f1 (0x5bfe bytes) 000270 1787791710.013249 write 0x003097 thru 0x01237c (0xf2e6 bytes) 000272 1787791710.022682 trunc from 0x040000 to 0x015c9f (0x2a360 bytes) 000273 1787791710.041634 trunc from 0x01051e to 0x015c9f (0x5782 bytes) 000274 1787791710.058523 mapread 0x00b203 thru 0x00c4d1 (0x12cf bytes) 000275 1787791710.059389 read 0x004045 thru 0x006ca2 (0x2c5e bytes) 000278 1787791710.065451 write 0x0051cf thru 0x0071a8 (0x1fda bytes) 000281 1787791710.070153 trunc from 0x015ca0 to 0x00cdec (0x8eb3 bytes) 000282 1787791710.099913 mapwrite 0x03eb34 thru 0x03ffff (0x14cc bytes) 000283 1787791710.180820 read 0x00f98d thru 0x00fae9 (0x15d bytes) 000284 1787791710.181607 write 0x01204e thru 0x0153fd (0x33b0 bytes) 000285 1787791710.183580 mapwrite 0x0207fb thru 0x02740f (0x6c15 bytes) 000287 1787791710.211893 punch from 0x00bab2 to 0x01b474 (0xf9c3 bytes) 000288 1787791710.221392 write 0x01bda2 thru 0x028f88 (0xd1e7 bytes) 000290 1787791710.235456 fallocate 0x03b890 thru 0x03ffff (0x4770 bytes) 000291 1787791710.240917 punch from 0x0309f5 to 0x031c09 (0x1215 bytes) 000292 1787791710.250560 mapread 0x028a7a thru 0x0365cc (0xdb53 bytes) 000294 1787791710.265368 mapread 0x01035e thru 0x0153c6 (0x5069 bytes) 000295 1787791710.266577 mapread 0x03ddde thru 0x03ffff (0x2222 bytes) 000296 1787791710.269421 read 0x011b08 thru 0x013c73 (0x216c bytes) 000297 1787791710.270608 mapwrite 0x00c031 thru 0x015c20 (0x9bf0 bytes) 000298 1787791710.302391 write 0x019506 thru 0x01f794 (0x628f bytes) 000299 1787791710.303588 read 0x029680 thru 0x0318f9 (0x827a bytes) 000300 1787791710.306199 punch from 0x0316aa to 0x03e8b1 (0xd208 bytes) 000301 1787791710.312783 trunc from 0x040000 to 0x007228 (0x38dd7 bytes) 000302 1787791710.331200 read 0x0062b6 thru 0x007228 (0xf73 bytes) 000303 1787791710.331873 read 0x005cdd thru 0x007228 (0x154c bytes) 000304 1787791710.332601 read 0x006ccb thru 0x007228 (0x55e bytes) 000305 1787791710.333914 mapread 0x001347 thru 0x007228 (0x5ee2 bytes) 000306 1787791710.335688 fallocate 0x02a1bf thru 0x02c773 (0x25b5 bytes) 000309 1787791710.344163 mapwrite 0x012bba thru 0x01ce19 (0xa260 bytes) 000312 1787791710.400889 read 0x01249b thru 0x01ce19 (0xa97f bytes) 000313 1787791710.405117 trunc from 0x01ce1a to 0x0110ea (0xbd2f bytes) 000314 1787791710.431763 mapwrite 0x00cead thru 0x017264 (0xa3b8 bytes) 000315 1787791710.487089 fallocate 0x02fa13 thru 0x03773b (0x7d29 bytes) 000316 1787791710.493324 mapread 0x0277c7 thru 0x0350f6 (0xd930 bytes) 000317 1787791710.507913 mapread 0x001e33 thru 0x00cce6 (0xaeb4 bytes) 000318 1787791710.509495 trunc from 0x01cf9f to 0x01e506 (0x1568 bytes) 000319 1787791710.514488 read 0x029476 thru 0x032295 (0x8e20 bytes) 000320 1787791710.515074 mapread 0x023581 thru 0x02f57c (0xbffc bytes) 000322 1787791710.516799 mapread 0x00cf32 thru 0x00e433 (0x1502 bytes) 000323 1787791710.517738 fallocate 0x00dac2 thru 0x011f52 (0x4491 bytes) 000324 1787791710.530907 MIRROR_EXTEND to 2 mirrors 000325 1787791711.158109 punch from 0x02cd5b to 0x035c6c (0x8f12 bytes) 000326 1787791711.204310 read 0x001cc0 thru 0x002645 (0x986 bytes) 000328 1787791711.215792 trunc from 0x03773c to 0x01d2e0 (0x1a45b bytes) 000329 1787791711.260879 fallocate 0x012f29 thru 0x01ae58 (0x7f30 bytes) 000331 1787791711.271228 trunc from 0x017ba4 to 0x01d2e0 (0x573d bytes) 000332 1787791711.278522 punch from 0x010a20 to 0x012b64 (0x2145 bytes) 000333 1787791711.285410 fallocate 0x00ed06 thru 0x017e43 (0x913e bytes) 000334 1787791711.290808 trunc from 0x0136ba to 0x016edf (0x3826 bytes) 000335 1787791711.295687 fallocate 0x026375 thru 0x02c982 (0x660e bytes) 000336 1787791711.300569 mapread 0x0142fd thru 0x017242 (0x2f46 bytes) 000337 1787791711.321083 punch from 0x00017b to 0x007fbd (0x7e43 bytes) 000338 1787791711.327402 mapread 0x01aa1b thru 0x01d2e0 (0x28c6 bytes) 000339 1787791711.328746 write 0x037cb5 thru 0x03ffff (0x834b bytes) 000340 1787791711.336411 MIRROR_RESYNC 2 mirrors 000341 1787791712.127193 MIRROR_RESYNC 2 mirrors 000342 1787791712.438661 punch from 0x01f3b7 to 0x027389 (0x7fd3 bytes) 000343 1787791712.481851 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e3:0x0]: Invalid argument (22) 000344 1787791712.645415 trunc from 0x040000 to 0x01e74f (0x218b0 bytes) 000345 1787791712.691969 read 0x014232 thru 0x01e74f (0xa51e bytes) 000346 1787791712.706631 read 0x007c6f thru 0x00c507 (0x4899 bytes) 000347 1787791712.715637 read 0x01bea3 thru 0x01e74f (0x28ad bytes) 000348 1787791712.716424 fallocate 0x00f0bb thru 0x01ec97 (0xfbdd bytes) 000349 1787791712.722182 write 0x02df0d thru 0x03020a (0x22fe bytes) 000351 1787791712.728177 MIRROR_EXTEND to 2 mirrors 000352 1787791713.462847 MIRROR_EXTEND to 3 mirrors 000353 1787791714.411052 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e3:0x0]: Invalid argument (22) 000354 1787791714.762008 trunc from 0x02731f to 0x02f663 (0x8345 bytes) 000355 1787791714.807592 mapwrite 0x024e71 thru 0x033215 (0xe3a5 bytes) 000356 1787791714.887492 fallocate 0x022e73 thru 0x025ea1 (0x302f bytes) 000357 1787791714.896760 write 0x03dd91 thru 0x03ffff (0x226f bytes) 000358 1787791714.905497 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e3:0x0]: Invalid argument (22) 000359 1787791715.065992 fallocate 0x006ea0 thru 0x00c0f2 (0x5253 bytes) 000360 1787791715.073653 fallocate 0x015391 thru 0x01563f (0x2af bytes) 000361 1787791715.084118 write 0x00bd43 thru 0x00d130 (0x13ee bytes) 000362 1787791715.108866 punch from 0x006586 to 0x00a2a2 (0x3d1d bytes) 000363 1787791715.117664 trunc from 0x040000 to 0x00ea4b (0x315b4 bytes) 000365 1787791715.135913 trunc from 0x00280f to 0x00ea4b (0xc23d bytes) 000366 1787791715.146576 fallocate 0x00ce5e thru 0x019539 (0xc6dc bytes) 000367 1787791715.152450 write 0x027bb7 thru 0x02d446 (0x5890 bytes) 000368 1787791715.154672 trunc from 0x011011 to 0x017219 (0x6209 bytes) 000370 1787791715.162128 read 0x014644 thru 0x021913 (0xd2d0 bytes) 000371 1787791715.176658 mapread 0x02cb51 thru 0x02d446 (0x8f6 bytes) 000372 1787791715.180483 fallocate 0x003604 thru 0x0068e4 (0x32e1 bytes) 000373 1787791715.189619 mapread 0x02a4e0 thru 0x02d446 (0x2f67 bytes) 000374 1787791715.191897 mapwrite 0x03ca80 thru 0x03ffff (0x3580 bytes) 000375 1787791715.251030 mapread 0x019fe1 thru 0x029355 (0xf375 bytes) 000376 1787791715.264861 mapread 0x03bdc0 thru 0x03ffff (0x4240 bytes) 000377 1787791715.265883 trunc from 0x040000 to 0x01bfd9 (0x24026 bytes) 000378 1787791715.280572 read 0x014a7d thru 0x01bfd9 (0x755d bytes) 000380 1787791715.281673 write 0x037d8f thru 0x03ffff (0x8271 bytes) 000383 1787791715.284501 mapwrite 0x03203e thru 0x03328e (0x1251 bytes) 000384 1787791715.315801 mapread 0x01871f thru 0x0246c0 (0xbfa2 bytes) 000386 1787791715.319102 read 0x0150f2 thru 0x0212ab (0xc1ba bytes) 000387 1787791715.319708 read 0x018761 thru 0x01bc9d (0x353d bytes) 000388 1787791715.320484 mapwrite 0x036133 thru 0x03ffff (0x9ecd bytes) 000389 1787791715.353376 mapread 0x02f588 thru 0x030fb1 (0x1a2a bytes) 000391 1787791715.354718 trunc from 0x031224 to 0x03298a (0x1767 bytes) 000392 1787791715.360237 write 0x02de80 thru 0x031b65 (0x3ce6 bytes) 000394 1787791715.366438 trunc from 0x040000 to 0x01369d (0x2c962 bytes) 000395 1787791715.382645 punch from 0x00cb1d to 0x012a81 (0x5f65 bytes) 000396 1787791715.388854 mapwrite 0x015e46 thru 0x01dd46 (0x7f01 bytes) 000398 1787791715.498097 write 0x015c38 thru 0x01d679 (0x7a42 bytes) 000399 1787791715.501044 trunc from 0x0026ae to 0x0117ec (0xf13f bytes) 000400 1787791715.507036 trunc from 0x01dd47 to 0x0042ec (0x19a5a bytes) 000401 1787791715.538561 MIRROR_EXTEND to 2 mirrors 000402 1787791716.184087 write 0x011afc thru 0x019889 (0x7d8e bytes) 000403 1787791716.226604 read 0x001816 thru 0x007c72 (0x645d bytes) 000404 1787791716.237442 mapread 0x003dbc thru 0x008ce2 (0x4f27 bytes) 000405 1787791716.245520 write 0x00f249 thru 0x01557b (0x6333 bytes) 000406 1787791716.258499 mapwrite 0x01d621 thru 0x021938 (0x4318 bytes) 000407 1787791716.358641 mapread 0x01333e thru 0x020d6c (0xda2f bytes) 000408 1787791716.360705 read 0x017b25 thru 0x01da78 (0x5f54 bytes) 000409 1787791716.361639 trunc from 0x01383b to 0x020827 (0xcfed bytes) 000412 1787791716.369571 read 0x000893 thru 0x0068d8 (0x6046 bytes) 000413 1787791716.383259 trunc from 0x01fe15 to 0x021938 (0x1b24 bytes) 000414 1787791716.390566 trunc from 0x021939 to 0x018d67 (0x8bd1 bytes) 000415 1787791716.416833 read 0x00fafb thru 0x018bcd (0x90d3 bytes) 000416 1787791716.424433 MIRROR_RESYNC 2 mirrors 000417 1787791717.079557 mapwrite 0x00e278 thru 0x017b78 (0x9901 bytes) 000418 1787791717.228638 write 0x01e9c1 thru 0x01ea69 (0xa9 bytes) 000419 1787791717.229717 trunc from 0x01ddd5 to 0x01ea69 (0xc95 bytes) 000420 1787791717.251717 trunc from 0x01ea6a to 0x007326 (0x17743 bytes) 000421 1787791717.308323 mapread 0x004e19 thru 0x007326 (0x250e bytes) 000422 1787791717.327826 mapread 0x004717 thru 0x007326 (0x2c10 bytes) 000423 1787791717.329015 read 0x003999 thru 0x007326 (0x398e bytes) 000424 1787791717.330259 mapwrite 0x01a7a6 thru 0x02a635 (0xfe90 bytes) 000425 1787791717.384568 fallocate 0x008b7a thru 0x015877 (0xccfe bytes) 000426 1787791717.390562 punch from 0x016d6a to 0x01d9c2 (0x6c59 bytes) 000427 1787791717.396076 punch from 0x01e0b3 to 0x0227f9 (0x4747 bytes) 000428 1787791717.404367 read 0x016ace thru 0x01fd3a (0x926d bytes) 000429 1787791717.424479 read 0x00d37f thru 0x0144d2 (0x7154 bytes) 000430 1787791717.425286 write 0x0386d2 thru 0x03ffff (0x792e bytes) 000431 1787791717.436067 mapwrite 0x011af9 thru 0x018977 (0x6e7f bytes) 000432 1787791717.473786 punch from 0x00b5f8 to 0x015aa5 (0xa4ae bytes) 000433 1787791717.498443 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x23dd0bb3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x854a5c4 Mirror 3: 0x854a5c4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000434 1787791718.622079 MIRROR_RESYNC 3 mirrors 000435 1787791719.435098 MIRROR_EXTEND to 4 mirrors 000436 1787791720.286284 read 0x01b152 thru 0x01b59a (0x449 bytes) 000437 1787791720.293319 trunc from 0x013b86 to 0x018227 (0x46a2 bytes) 000438 1787791720.338047 read 0x02c390 thru 0x030621 (0x4292 bytes) 000439 1787791720.358199 punch from 0x015f30 to 0x01abdd (0x4cae bytes) 000440 1787791720.373351 fallocate 0x01d8e3 thru 0x029109 (0xb827 bytes) 000441 1787791720.379574 fallocate 0x00c1f1 thru 0x013840 (0x7650 bytes) 000442 1787791720.385069 punch from 0x0242eb to 0x0334b4 (0xf1ca bytes) 000443 1787791720.390200 write 0x030688 thru 0x038e47 (0x87c0 bytes) 000444 1787791720.399823 trunc from 0x024378 to 0x02c973 (0x85fc bytes) 000445 1787791720.407056 mapread 0x01f7d3 thru 0x021cb2 (0x24e0 bytes) 000446 1787791720.408238 mapread 0x039a00 thru 0x03ffff (0x6600 bytes) 000447 1787791720.415281 MIRROR_RESYNC 4 mirrors 000448 1787791721.636310 read 0x006e3f thru 0x00f82c (0x89ee bytes) 000449 1787791721.652328 mapread 0x028d57 thru 0x02fb07 (0x6db1 bytes) 000450 1787791721.684156 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e3:0x0]: Invalid argument (22) 000451 1787791722.008733 MIRROR_EXTEND to 4 mirrors 000452 1787791722.754075 write 0x0120ec thru 0x01c4c9 (0xa3de bytes) 000453 1787791722.805850 mapwrite 0x026c77 thru 0x028150 (0x14da bytes) 000454 1787791722.848050 write 0x02f6ea thru 0x03741d (0x7d34 bytes) 000455 1787791722.857708 read 0x006f37 thru 0x00e32e (0x73f8 bytes) 000456 1787791722.867153 punch from 0x01f7f7 to 0x01f9da (0x1e4 bytes) 000457 1787791722.875750 mapread 0x028596 thru 0x033421 (0xae8c bytes) 000458 1787791722.876915 mapread 0x02695e thru 0x0340ca (0xd76d bytes) 000459 1787791722.878292 mapread 0x010f05 thru 0x018c71 (0x7d6d bytes) 000461 1787791722.886560 mapwrite 0x03e68f thru 0x03ffff (0x1971 bytes) 000462 1787791722.908268 write 0x01c1b2 thru 0x0219e7 (0x5836 bytes) 000463 1787791722.909566 trunc from 0x040000 to 0x00f3e8 (0x30c17 bytes) 000464 1787791722.941024 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf3e9): Mirror 4: 0xed12c08f (stale) Mirror 5: 0xed12c08f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf3e9): Mirror 4: 0xed12c08f (stale) Mirror 5: 0xed12c08f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf3e9: skipped 000465 1787791723.273475 read 0x0079bc thru 0x00a862 (0x2ea7 bytes) 000466 1787791723.283624 read 0x000b28 thru 0x0025f7 (0x1ad0 bytes) 000467 1787791723.299489 trunc from 0x00f3e9 to 0x00ffa8 (0xbc0 bytes) 000468 1787791723.323757 punch from 0x00abb4 to 0x00ffa8 (0x53f5 bytes) 000469 1787791723.330080 trunc from 0x00ffa9 to 0x035f0c (0x25f64 bytes) 000470 1787791723.357852 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35f0d): Mirror 4: 0x74a7c30 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35f0d): Mirror 4: 0x74a7c30 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35f0d: skipped 000471 1787791723.739218 trunc from 0x0021f0 to 0x0043a1 (0x21b2 bytes) 000472 1787791723.750131 write 0x0340f4 thru 0x03ffff (0xbf0c bytes) 000473 1787791723.771341 fallocate 0x03fd8d thru 0x03ffff (0x273 bytes) 000474 1787791723.783422 fallocate 0x036a3d thru 0x03d2ba (0x687e bytes) 000475 1787791723.806087 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xb77651d8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x10eacd81 Mirror 5: 0x10eacd81 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000477 1787791724.716907 MIRROR_RESYNC 3 mirrors 000478 1787791725.395242 mapread 0x0370e7 thru 0x03ffff (0x8f19 bytes) 000479 1787791725.423537 fallocate 0x007dd7 thru 0x0171d3 (0xf3fd bytes) 000480 1787791725.474472 mapwrite 0x03a495 thru 0x03ffff (0x5b6b bytes) 000481 1787791725.523631 punch from 0x02edb6 to 0x036a05 (0x7c50 bytes) 000482 1787791725.532768 punch from 0x028396 to 0x02d4fd (0x5168 bytes) 000484 1787791725.544675 punch from 0x026e19 to 0x029fa4 (0x318c bytes) 000485 1787791725.554720 mapwrite 0x020248 thru 0x02da62 (0xd81b bytes) 000486 1787791725.614434 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x10eacd81 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x10eacd81 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000487 1787791725.967164 fallocate 0x00685d thru 0x014b24 (0xe2c8 bytes) 000488 1787791725.974181 fallocate 0x036636 thru 0x03a0c7 (0x3a92 bytes) 000489 1787791725.982869 mapread 0x00a552 thru 0x00ecc1 (0x4770 bytes) 000490 1787791726.022064 punch from 0x03aa15 to 0x03ffff (0x55eb bytes) 000491 1787791726.027255 punch from 0x00bf9b to 0x00e214 (0x227a bytes) 000492 1787791726.039764 punch from 0x00b626 to 0x00c368 (0xd43 bytes) 000493 1787791726.046844 punch from 0x02a3bd to 0x0344e2 (0xa126 bytes) 000494 1787791726.055892 punch from 0x0212d0 to 0x026950 (0x5681 bytes) 000495 1787791726.067908 punch from 0x010959 to 0x016ca1 (0x6349 bytes) 000496 1787791726.075916 punch from 0x01f15e to 0x0264e1 (0x7384 bytes) 000497 1787791726.083738 punch from 0x012db8 to 0x01763b (0x4884 bytes) 000498 1787791726.099305 punch from 0x00f527 to 0x0138a3 (0x437d bytes) 000499 1787791726.104619 write 0x02833d thru 0x035434 (0xd0f8 bytes) 000500 1787791726.128172 write 0x0355f5 thru 0x03c45a (0x6e66 bytes) 000501 1787791726.151520 MIRROR_RESYNC 2 mirrors 000502 1787791726.876623 fallocate 0x0010ea thru 0x004b10 (0x3a27 bytes) 000503 1787791726.936921 mapread 0x01e7c2 thru 0x022219 (0x3a58 bytes) 000504 1787791726.974315 trunc from 0x040000 to 0x0187fb (0x27804 bytes) 000505 1787791726.994385 read 0x014aef thru 0x0187fb (0x3d0d bytes) 000506 1787791727.005866 write 0x0306fa thru 0x032be9 (0x24f0 bytes) 000507 1787791727.008088 mapwrite 0x0096d3 thru 0x00b8b1 (0x21df bytes) 000508 1787791727.036777 trunc from 0x032bea to 0x02fbf9 (0x2ff0 bytes) 000509 1787791727.060904 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e3:0x0]: Invalid argument (22) 000510 1787791727.231572 trunc from 0x02fbfa to 0x02d89a (0x235f bytes) 000511 1787791727.257289 trunc from 0x024ee2 to 0x0298fa (0x4a19 bytes) 000512 1787791727.263346 trunc from 0x00c5f5 to 0x00cf91 (0x99d bytes) 000513 1787791727.268728 mapwrite 0x021558 thru 0x02cd66 (0xb80f bytes) 000515 1787791727.347654 write 0x00a835 thru 0x0159c5 (0xb191 bytes) 000516 1787791727.372795 punch from 0x0184d3 to 0x02590e (0xd43c bytes) 000517 1787791727.382012 punch from 0x021445 to 0x022ba1 (0x175d bytes) 000518 1787791727.390049 fallocate 0x012166 thru 0x013a9c (0x1937 bytes) 000519 1787791727.396154 mapread 0x01f13d thru 0x02511a (0x5fde bytes) 000520 1787791727.417388 MIRROR_EXTEND to 2 mirrors 000521 1787791728.089983 read 0x0236a1 thru 0x02d89a (0xa1fa bytes) 000522 1787791728.101161 fallocate 0x00e55e thru 0x010386 (0x1e29 bytes) 000524 1787791728.162915 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e3:0x0]: Invalid argument (22) 000525 1787791728.324365 fallocate 0x00fe25 thru 0x013534 (0x3710 bytes) 000526 1787791728.333030 write 0x01220f thru 0x0135e2 (0x13d4 bytes) 000527 1787791728.355701 punch from 0x023e28 to 0x02d89a (0x9a73 bytes) 000529 1787791728.370828 punch from 0x016eda to 0x023760 (0xc887 bytes) 000530 1787791728.382682 write 0x0211d9 thru 0x023c28 (0x2a50 bytes) 000531 1787791728.402765 MIRROR_EXTEND to 2 mirrors 000532 1787791729.240762 write 0x03ec0b thru 0x03ffff (0x13f5 bytes) 000533 1787791729.310512 mapread 0x0009f9 thru 0x009932 (0x8f3a bytes) 000534 1787791729.350390 fallocate 0x0369c3 thru 0x03a095 (0x36d3 bytes) 000535 1787791729.361262 trunc from 0x00d9bd to 0x010c6a (0x32ae bytes) 000537 1787791729.368482 trunc from 0x040000 to 0x01e18a (0x21e75 bytes) 000538 1787791729.413612 read 0x007f5f thru 0x015071 (0xd113 bytes) 000539 1787791729.431396 write 0x01e757 thru 0x02c3c3 (0xdc6d bytes) 000540 1787791729.450045 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e3:0x0]: Invalid argument (22) 000541 1787791729.639194 fallocate 0x016f5c thru 0x01d67f (0x6724 bytes) 000542 1787791729.651920 mapwrite 0x022794 thru 0x026eea (0x4757 bytes) 000543 1787791729.693387 read 0x00e62a thru 0x016792 (0x8169 bytes) 000544 1787791729.712730 trunc from 0x00abce to 0x00d915 (0x2d48 bytes) 000545 1787791729.720077 trunc from 0x013839 to 0x01a071 (0x6839 bytes) 000546 1787791729.725391 write 0x005361 thru 0x00c8da (0x757a bytes) 000548 1787791729.741562 fallocate 0x027d0a thru 0x02dbb9 (0x5eb0 bytes) 000549 1787791729.749027 trunc from 0x02dbba to 0x007954 (0x26265 bytes) 000550 1787791729.763856 read 0x00746d thru 0x007954 (0x4e8 bytes) 000551 1787791729.764774 write 0x024673 thru 0x02f2ad (0xac3b bytes) 000552 1787791729.767772 mapwrite 0x01f6ba thru 0x027ee3 (0x882a bytes) 000553 1787791729.808870 MIRROR_EXTEND to 2 mirrors 000554 1787791730.607689 punch from 0x014a5f to 0x0190a3 (0x4645 bytes) 000555 1787791730.643810 punch from 0x00708e to 0x00cd1c (0x5c8f bytes) 000556 1787791730.649497 fallocate 0x02757d thru 0x0314dc (0x9f60 bytes) 000557 1787791730.655833 trunc from 0x02f2ae to 0x0161dc (0x190d1 bytes) 000558 1787791730.686830 mapread 0x003c24 thru 0x012c28 (0xf005 bytes) 000559 1787791730.723875 read 0x013156 thru 0x014d4e (0x1bf9 bytes) 000560 1787791730.725638 mapwrite 0x01afae thru 0x029a53 (0xeaa6 bytes) 000562 1787791730.801633 mapread 0x020894 thru 0x023fa0 (0x370d bytes) 000563 1787791730.804252 punch from 0x023b94 to 0x029a53 (0x5ec0 bytes) 000564 1787791730.809694 mapwrite 0x023ad8 thru 0x032f72 (0xf49b bytes) 000567 1787791730.873160 write 0x036d00 thru 0x03ffff (0x9300 bytes) 000568 1787791730.879153 MIRROR_RESYNC 2 mirrors 000569 1787791731.613111 trunc from 0x017b8e to 0x01fae7 (0x7f5a bytes) 000570 1787791731.662031 trunc from 0x028968 to 0x02f72a (0x6dc3 bytes) 000571 1787791731.671840 read 0x01e408 thru 0x024993 (0x658c bytes) 000572 1787791731.686515 punch from 0x0138f9 to 0x01606c (0x2774 bytes) 000573 1787791731.701491 write 0x00f65d thru 0x012ff9 (0x399d bytes) 000574 1787791731.722795 trunc from 0x033e93 to 0x034af8 (0xc66 bytes) 000575 1787791731.728725 mapwrite 0x00165e thru 0x0087bb (0x715e bytes) 000576 1787791731.764981 fallocate 0x029fc2 thru 0x02dca7 (0x3ce6 bytes) 000577 1787791731.769212 write 0x036721 thru 0x038710 (0x1ff0 bytes) 000578 1787791731.784516 write 0x01e8ef thru 0x022a56 (0x4168 bytes) 000579 1787791731.786540 fallocate 0x0372a4 thru 0x03ffff (0x8d5c bytes) 000580 1787791731.801316 MIRROR_RESYNC 2 mirrors 000581 1787791732.659756 mapread 0x00e362 thru 0x015415 (0x70b4 bytes) 000582 1787791732.698736 read 0x038c72 thru 0x03ffff (0x738e bytes) 000583 1787791732.723793 read 0x004de4 thru 0x00a249 (0x5466 bytes) 000584 1787791732.761998 trunc from 0x00f845 to 0x01eb9a (0xf356 bytes) 000586 1787791732.824094 write 0x0182a6 thru 0x022f3d (0xac98 bytes) 000587 1787791732.836358 mapread 0x00153f thru 0x00fd76 (0xe838 bytes) 000589 1787791732.909290 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x30e91b0b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6ace384 Mirror 3: 0x6ace384 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000590 1787791734.091094 MIRROR_RESYNC 3 mirrors 000591 1787791734.899751 mapwrite 0x021932 thru 0x02d96b (0xc03a bytes) 000592 1787791735.011232 punch from 0x0083f3 to 0x00d5c3 (0x51d1 bytes) 000593 1787791735.018449 trunc from 0x01328b to 0x018308 (0x507e bytes) 000594 1787791735.027664 trunc from 0x03e4d9 to 0x03ffff (0x1b27 bytes) 000595 1787791735.032677 mapwrite 0x03a09d thru 0x03ffff (0x5f63 bytes) 000596 1787791735.064715 trunc from 0x02f19e to 0x034e11 (0x5c74 bytes) 000597 1787791735.070233 read 0x03d7d1 thru 0x03df33 (0x763 bytes) 000598 1787791735.070848 trunc from 0x0289ea to 0x0352ff (0xc916 bytes) 000599 1787791735.076060 fallocate 0x02c19f thru 0x030233 (0x4095 bytes) 000600 1787791735.082512 mapread 0x014aa8 thru 0x01b6c9 (0x6c22 bytes) 000601 1787791735.102283 trunc from 0x040000 to 0x02123c (0x1edc3 bytes) 000602 1787791735.138256 punch from 0x01e939 to 0x02123c (0x2904 bytes) 000603 1787791735.146479 write 0x003e2d thru 0x00d67c (0x9850 bytes) 000604 1787791735.159644 read 0x008818 thru 0x016978 (0xe161 bytes) 000605 1787791735.162752 mapread 0x00c5d6 thru 0x0198cc (0xd2f7 bytes) 000606 1787791735.169785 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2123d): Mirror 3: 0xd645a45c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2123d): Mirror 3: 0xd645a45c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2123d: skipped 000607 1787791735.474179 write 0x01740d thru 0x024dc4 (0xd9b8 bytes) 000608 1787791735.485794 punch from 0x01d8e6 to 0x024dc4 (0x74df bytes) 000609 1787791735.507767 read 0x007a3e thru 0x00a543 (0x2b06 bytes) 000610 1787791735.517147 read 0x01fc50 thru 0x024dc4 (0x5175 bytes) 000611 1787791735.531163 mapwrite 0x0397bf thru 0x03ffff (0x6841 bytes) 000612 1787791735.599483 mapread 0x010d42 thru 0x01ed5f (0xe01e bytes) 000613 1787791735.608625 MIRROR_RESYNC 2 mirrors 000614 1787791736.354223 fallocate 0x003cef thru 0x0076b0 (0x39c2 bytes) 000615 1787791736.390182 write 0x028114 thru 0x02b56a (0x3457 bytes) 000616 1787791736.399981 write 0x02b2b9 thru 0x036d50 (0xba98 bytes) 000617 1787791736.406922 trunc from 0x040000 to 0x039ecf (0x6130 bytes) 000618 1787791736.426365 mapwrite 0x03b125 thru 0x03ffff (0x4edb bytes) 000619 1787791736.492163 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xbbc028a3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc7bd9c76 Mirror 4: 0xc7bd9c76 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000620 1787791737.588090 trunc from 0x00c546 to 0x0180a6 (0xbb61 bytes) 000621 1787791737.666596 read 0x0129f9 thru 0x01f878 (0xce80 bytes) 000622 1787791737.681623 trunc from 0x00e651 to 0x01655c (0x7f0c bytes) 000623 1787791737.688125 trunc from 0x01487f to 0x02185c (0xcfde bytes) 000624 1787791737.695698 read 0x025684 thru 0x02b014 (0x5991 bytes) 000625 1787791737.712213 mapread 0x028c63 thru 0x031d91 (0x912f bytes) 000626 1787791737.713447 mapread 0x01923f thru 0x0210d7 (0x7e99 bytes) 000627 1787791737.714838 mapwrite 0x0148b1 thru 0x022d46 (0xe496 bytes) 000629 1787791737.741914 read 0x0134b1 thru 0x021fd0 (0xeb20 bytes) 000630 1787791737.742537 read 0x007998 thru 0x010ba3 (0x920c bytes) 000631 1787791737.743125 mapwrite 0x03b7be thru 0x03ffff (0x4842 bytes) 000632 1787791737.764747 mapwrite 0x005550 thru 0x00fb34 (0xa5e5 bytes) 000633 1787791737.790353 trunc from 0x037c40 to 0x03ffff (0x83c0 bytes) 000634 1787791737.797135 mapwrite 0x03390d thru 0x03f439 (0xbb2d bytes) 000635 1787791737.855359 read 0x00e47c thru 0x01981a (0xb39f bytes) 000636 1787791737.856073 fallocate 0x032045 thru 0x03ffff (0xdfbb bytes) 000637 1787791737.870903 write 0x010455 thru 0x019890 (0x943c bytes) 000638 1787791737.881255 read 0x016ead thru 0x025d66 (0xeeba bytes) 000639 1787791737.884358 trunc from 0x040000 to 0x01d34a (0x22cb5 bytes) 000640 1787791737.930627 write 0x02dc33 thru 0x02e1de (0x5ac bytes) 000641 1787791737.933423 trunc from 0x02e1df to 0x029015 (0x51c9 bytes) 000643 1787791737.953793 trunc from 0x00e45e to 0x0170ba (0x8c5d bytes) 000644 1787791737.978140 write 0x0124c7 thru 0x01bf7e (0x9ab8 bytes) 000645 1787791738.005630 write 0x00cbb5 thru 0x01c437 (0xf883 bytes) 000646 1787791738.008087 read 0x02647f thru 0x029015 (0x2b97 bytes) 000647 1787791738.009791 read 0x00ca41 thru 0x00db6b (0x112b bytes) 000648 1787791738.010522 trunc from 0x029016 to 0x010513 (0x18b02 bytes) 000651 1787791738.038010 read 0x003f10 thru 0x0087b5 (0x48a6 bytes) 000652 1787791738.038679 punch from 0x00c6a9 to 0x010513 (0x3e6b bytes) 000653 1787791738.055203 write 0x02ab22 thru 0x036867 (0xbd46 bytes) 000654 1787791738.056907 read 0x033c69 thru 0x036867 (0x2bff bytes) 000655 1787791738.058519 mapwrite 0x017556 thru 0x01ca88 (0x5533 bytes) 000656 1787791738.105755 mapwrite 0x02dc4b thru 0x02e681 (0xa37 bytes) 000659 1787791738.136110 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36868): Mirror 3: 0xc4cba4ea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36868): Mirror 3: 0xc4cba4ea (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36868: skipped 000660 1787791738.466598 mapwrite 0x003e70 thru 0x00781f (0x39b0 bytes) 000661 1787791738.517699 mapwrite 0x03caeb thru 0x03ffff (0x3515 bytes) 000662 1787791738.574543 mapread 0x02fd24 thru 0x033794 (0x3a71 bytes) 000663 1787791738.588338 punch from 0x004c1d to 0x0068c3 (0x1ca7 bytes) 000665 1787791738.597233 mapread 0x02e8b1 thru 0x03c536 (0xdc86 bytes) 000666 1787791738.601224 read 0x039e21 thru 0x03ffff (0x61df bytes) 000667 1787791738.602701 trunc from 0x00c4cc to 0x012136 (0x5c6b bytes) 000668 1787791738.610346 mapread 0x026d04 thru 0x0276dc (0x9d9 bytes) 000669 1787791738.628567 write 0x015476 thru 0x0233c0 (0xdf4b bytes) 000670 1787791738.637919 trunc from 0x040000 to 0x019364 (0x26c9b bytes) 000671 1787791738.671104 read 0x0056a3 thru 0x013be1 (0xe53f bytes) 000673 1787791738.685433 read 0x01012e thru 0x019364 (0x9237 bytes) 000674 1787791738.686605 fallocate 0x039f54 thru 0x03ec88 (0x4d35 bytes) 000675 1787791738.692611 mapread 0x02a063 thru 0x0369c0 (0xc95e bytes) 000676 1787791738.708202 trunc from 0x03ec89 to 0x015ce4 (0x28fa4 bytes) 000677 1787791738.753491 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e3:0x0]: Invalid argument (22) 000678 1787791738.941214 trunc from 0x015ce5 to 0x0110a9 (0x4c3b bytes) 000679 1787791738.964745 punch from 0x00fb96 to 0x0110a9 (0x1514 bytes) 000680 1787791738.969566 mapread 0x00628d thru 0x0110a9 (0xae1d bytes) 000681 1787791738.995463 write 0x017123 thru 0x018978 (0x1856 bytes) 000682 1787791738.996635 trunc from 0x018979 to 0x015605 (0x3373 bytes) 000683 1787791739.010186 mapread 0x014c0a thru 0x015605 (0x9fc bytes) 000684 1787791739.015851 trunc from 0x015606 to 0x025e01 (0x107fc bytes) 000685 1787791739.034248 fallocate 0x01eb7d thru 0x02528a (0x670e bytes) 000686 1787791739.038717 trunc from 0x025e02 to 0x010d35 (0x150cc bytes) 000687 1787791739.057948 mapread 0x00291f thru 0x010a74 (0xe156 bytes) 000688 1787791739.058854 mapread 0x00894c thru 0x009707 (0xdbc bytes) 000689 1787791739.059920 mapwrite 0x032c8a thru 0x0337db (0xb52 bytes) 000690 1787791739.107139 punch from 0x01507b to 0x017250 (0x21d6 bytes) 000691 1787791739.117132 fallocate 0x023579 thru 0x031a12 (0xe49a bytes) 000692 1787791739.123237 read 0x009b68 thru 0x0172db (0xd774 bytes) 000693 1787791739.132650 read 0x017c2a thru 0x0220ef (0xa4c6 bytes) 000695 1787791739.150795 MIRROR_EXTEND to 2 mirrors 000696 1787791739.720552 mapwrite 0x03a2bf thru 0x03a8bf (0x601 bytes) 000698 1787791739.801520 trunc from 0x03a8c0 to 0x0163b8 (0x24507 bytes) 000699 1787791739.821117 write 0x000198 thru 0x000225 (0x8e bytes) 000700 1787791739.828410 trunc from 0x0163b9 to 0x0317c6 (0x1b40e bytes) 000701 1787791739.845063 trunc from 0x0066b2 to 0x00a301 (0x3c50 bytes) 000702 1787791739.850885 trunc from 0x0317c7 to 0x009527 (0x2829f bytes) 000703 1787791739.868459 write 0x03a756 thru 0x03d095 (0x2940 bytes) 000704 1787791739.870582 write 0x03f88a thru 0x03ffff (0x776 bytes) 000705 1787791739.871788 mapwrite 0x03611d thru 0x03c414 (0x62f8 bytes) 000706 1787791739.935513 MIRROR_RESYNC 2 mirrors 000707 1787791740.741427 mapread 0x02e046 thru 0x039674 (0xb62f bytes) 000708 1787791740.769402 punch from 0x02739b to 0x0366bb (0xf321 bytes) 000709 1787791740.819086 MIRROR_RESYNC 2 mirrors 000710 1787791741.807804 fallocate 0x02b5f5 thru 0x033c1d (0x8629 bytes) 000711 1787791741.845123 mapread 0x01d98a thru 0x026e17 (0x948e bytes) 000712 1787791741.881327 punch from 0x0225d7 to 0x028bf0 (0x661a bytes) 000713 1787791741.893175 write 0x0063ff thru 0x00d4de (0x70e0 bytes) 000714 1787791741.908635 fallocate 0x011284 thru 0x0168f4 (0x5671 bytes) 000715 1787791741.915281 mapread 0x0038cc thru 0x010170 (0xc8a5 bytes) 000716 1787791741.935632 mapwrite 0x010411 thru 0x0181db (0x7dcb bytes) 000718 1787791741.975235 fallocate 0x00dd0d thru 0x00f9c0 (0x1cb4 bytes) 000719 1787791741.980517 trunc from 0x012d0c to 0x01715c (0x4451 bytes) 000721 1787791741.994186 punch from 0x03a1e6 to 0x03d63f (0x345a bytes) 000722 1787791741.999638 trunc from 0x019a3d to 0x027198 (0xd75c bytes) 000723 1787791742.011800 read 0x0359aa thru 0x03bb72 (0x61c9 bytes) 000724 1787791742.031074 trunc from 0x02d179 to 0x03bb73 (0xe9fb bytes) 000725 1787791742.042205 trunc from 0x00b638 to 0x01ab4a (0xf513 bytes) 000726 1787791742.053370 trunc from 0x006680 to 0x00c378 (0x5cf9 bytes) 000727 1787791742.061682 trunc from 0x040000 to 0x01192b (0x2e6d4 bytes) 000728 1787791742.087623 punch from 0x00b56b to 0x01192b (0x63c1 bytes) 000729 1787791742.093522 trunc from 0x00538d to 0x01192b (0xc59f bytes) 000730 1787791742.099158 trunc from 0x01192c to 0x013065 (0x173a bytes) 000731 1787791742.115145 fallocate 0x0326da thru 0x037df7 (0x571e bytes) 000732 1787791742.120309 read 0x0026d4 thru 0x0102b2 (0xdbdf bytes) 000733 1787791742.132796 mapwrite 0x0174f0 thru 0x0223e4 (0xaef5 bytes) 000734 1787791742.175099 mapread 0x00dbac thru 0x019869 (0xbcbe bytes) 000735 1787791742.176135 trunc from 0x00be37 to 0x0186c6 (0xc890 bytes) 000736 1787791742.186071 mapread 0x01cd2a thru 0x0223e4 (0x56bb bytes) 000737 1787791742.187054 trunc from 0x0223e5 to 0x0218e1 (0xb03 bytes) 000738 1787791742.220897 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e3:0x0]: Invalid argument (22) 000739 1787791742.379867 punch from 0x001d43 to 0x00d207 (0xb4c5 bytes) 000740 1787791742.386325 fallocate 0x00db57 thru 0x0102bc (0x2766 bytes) 000741 1787791742.396287 mapwrite 0x01c2bc thru 0x022a48 (0x678d bytes) 000742 1787791742.484764 write 0x010d8e thru 0x01a703 (0x9976 bytes) 000744 1787791742.495723 read 0x001b51 thru 0x00944c (0x78fc bytes) 000745 1787791742.518567 mapread 0x008aa4 thru 0x01108d (0x85ea bytes) 000746 1787791742.546227 punch from 0x008b31 to 0x016e72 (0xe342 bytes) 000747 1787791742.593837 punch from 0x00d857 to 0x01255a (0x4d04 bytes) 000752 1787791742.627709 punch from 0x014d28 to 0x01c339 (0x7612 bytes) 000753 1787791742.638276 fallocate 0x020b30 thru 0x024d35 (0x4206 bytes) 000754 1787791742.651617 mapread 0x0061b5 thru 0x00e708 (0x8554 bytes) 000755 1787791742.671528 mapread 0x00a44c thru 0x01815d (0xdd12 bytes) 000756 1787791742.693050 punch from 0x022f48 to 0x024d35 (0x1dee bytes) 000757 1787791742.705643 trunc from 0x024d36 to 0x02359a (0x179b bytes) 000758 1787791742.731704 punch from 0x013850 to 0x01d338 (0x9ae9 bytes) 000759 1787791742.742141 trunc from 0x02359b to 0x01efe8 (0x45b2 bytes) 000761 1787791742.765103 mapwrite 0x0272c9 thru 0x0287c8 (0x1500 bytes) 000763 1787791742.821233 mapread 0x001037 thru 0x01044e (0xf418 bytes) 000764 1787791742.822904 fallocate 0x02b2d2 thru 0x0349fe (0x972d bytes) 000765 1787791742.833780 trunc from 0x00986c to 0x00f67a (0x5e0f bytes) 000768 1787791742.842839 trunc from 0x0237b0 to 0x0287c8 (0x5019 bytes) 000769 1787791742.851776 punch from 0x01a2df to 0x01bb4a (0x186c bytes) 000771 1787791742.861112 fallocate 0x01d0a3 thru 0x0276e1 (0xa63f bytes) 000774 1787791742.868544 fallocate 0x01d43f thru 0x01ddd3 (0x995 bytes) 000776 1787791742.877997 read 0x000428 thru 0x00e04e (0xdc27 bytes) 000777 1787791742.887822 write 0x031a99 thru 0x03a5c6 (0x8b2e bytes) 000778 1787791742.894442 MIRROR_EXTEND to 2 mirrors 000779 1787791743.579862 punch from 0x027dfd to 0x02df2b (0x612f bytes) 000780 1787791743.618256 trunc from 0x025071 to 0x0265b2 (0x1542 bytes) 000781 1787791743.622172 fallocate 0x008102 thru 0x01211f (0xa01e bytes) 000782 1787791743.631883 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e3:0x0]: Invalid argument (22) 000783 1787791743.764384 fallocate 0x007622 thru 0x01425d (0xcc3c bytes) 000784 1787791743.771475 mapread 0x00d955 thru 0x00f12f (0x17db bytes) 000785 1787791743.790732 trunc from 0x03a5c7 to 0x000843 (0x39d83 bytes) 000786 1787791743.826434 punch from 0x0000a2 to 0x000843 (0x7a2 bytes) 000788 1787791743.833853 write 0x024734 thru 0x02977d (0x504a bytes) 000791 1787791743.838257 read 0x00c3cf thru 0x0176b6 (0xb2e8 bytes) 000792 1787791743.852607 mapread 0x01ac32 thru 0x029584 (0xe953 bytes) 000793 1787791743.871398 fallocate 0x027e4f thru 0x0353b2 (0xd564 bytes) 000795 1787791743.881242 trunc from 0x02977e to 0x02d568 (0x3deb bytes) 000796 1787791743.903767 mapwrite 0x00881a thru 0x00bdca (0x35b1 bytes) 000797 1787791743.939158 mapwrite 0x02df88 thru 0x0377af (0x9828 bytes) 000798 1787791744.031326 write 0x030fef thru 0x037b90 (0x6ba2 bytes) 000799 1787791744.032515 mapread 0x02954b thru 0x0318d5 (0x838b bytes) 000800 1787791744.035889 read 0x0105ca thru 0x017b73 (0x75aa bytes) 000802 1787791744.037496 read 0x033622 thru 0x036918 (0x32f7 bytes) 000804 1787791744.038839 punch from 0x01c74a to 0x02388b (0x7142 bytes) 000806 1787791744.044177 trunc from 0x01fd2a to 0x021322 (0x15f9 bytes) 000807 1787791744.051476 write 0x02eb89 thru 0x02fed1 (0x1349 bytes) 000808 1787791744.053060 mapwrite 0x00d72a thru 0x011911 (0x41e8 bytes) 000809 1787791744.077902 trunc from 0x037b91 to 0x02b753 (0xc43d bytes) 000810 1787791744.108879 mapread 0x008a7f thru 0x00dd39 (0x52bb bytes) 000811 1787791744.110044 mapwrite 0x0339df thru 0x03ffff (0xc621 bytes) 000812 1787791744.180597 read 0x0033b7 thru 0x00dd9a (0xa9e4 bytes) 000813 1787791744.181478 mapwrite 0x021f8b thru 0x02da24 (0xba9a bytes) 000814 1787791744.205425 punch from 0x02cb82 to 0x03addc (0xe25b bytes) 000815 1787791744.210560 read 0x00f882 thru 0x01811d (0x889c bytes) 000818 1787791744.213092 mapwrite 0x0104a8 thru 0x014984 (0x44dd bytes) 000819 1787791744.232504 mapread 0x02d6f4 thru 0x034133 (0x6a40 bytes) 000820 1787791744.246214 write 0x02f527 thru 0x036ae0 (0x75ba bytes) 000821 1787791744.247421 fallocate 0x02371a thru 0x02ffa4 (0xc88b bytes) 000822 1787791744.252122 write 0x033597 thru 0x035f42 (0x29ac bytes) 000823 1787791744.252905 mapwrite 0x03991d thru 0x03ffff (0x66e3 bytes) 000824 1787791744.271846 fallocate 0x03b1f9 thru 0x03f847 (0x464f bytes) 000826 1787791744.278416 read 0x0347e0 thru 0x03d191 (0x89b2 bytes) 000827 1787791744.279022 write 0x0149f2 thru 0x024182 (0xf791 bytes) 000828 1787791744.283274 write 0x0190fe thru 0x02657f (0xd482 bytes) 000829 1787791744.284709 mapread 0x00d97b thru 0x014fd6 (0x765c bytes) 000831 1787791744.287022 write 0x016973 thru 0x02643f (0xfacd bytes) 000832 1787791744.291642 mapwrite 0x0262d9 thru 0x02a0f6 (0x3e1e bytes) 000833 1787791744.325084 mapwrite 0x03f798 thru 0x03ffff (0x868 bytes) 000834 1787791744.355777 mapread 0x0374b9 thru 0x03ffff (0x8b47 bytes) 000835 1787791744.361827 MIRROR_EXTEND to 2 mirrors 000836 1787791745.415368 mapwrite 0x02fc6b thru 0x03e514 (0xe8aa bytes) 000837 1787791745.676194 write 0x031d8c thru 0x03c9bc (0xac31 bytes) 000838 1787791745.713466 MIRROR_RESYNC 2 mirrors 000839 1787791746.952821 trunc from 0x039a70 to 0x03d782 (0x3d13 bytes) 000840 1787791747.016320 mapread 0x013635 thru 0x0184db (0x4ea7 bytes) 000841 1787791747.041316 write 0x007d55 thru 0x011ce1 (0x9f8d bytes) 000842 1787791747.051653 trunc from 0x010a58 to 0x01def7 (0xd4a0 bytes) 000843 1787791747.072520 mapwrite 0x00aeb5 thru 0x00ea01 (0x3b4d bytes) 000844 1787791747.089276 trunc from 0x040000 to 0x0007c0 (0x3f83f bytes) 000845 1787791747.123445 trunc from 0x000267 to 0x0007c0 (0x55a bytes) 000846 1787791747.138181 mapread 0x0002a0 thru 0x0007c0 (0x521 bytes) 000847 1787791747.172418 mapread 0x000742 thru 0x0007c0 (0x7f bytes) 000848 1787791747.180317 read 0x000063 thru 0x0007c0 (0x75e bytes) 000849 1787791747.181962 trunc from 0x0007c1 to 0x01f5ab (0x1edeb bytes) 000850 1787791747.220253 read 0x017f7f thru 0x01d01b (0x509d bytes) 000851 1787791747.251909 fallocate 0x02e344 thru 0x0319bb (0x3678 bytes) 000852 1787791747.273416 fallocate 0x032ec6 thru 0x038b76 (0x5cb1 bytes) 000853 1787791747.290519 fallocate 0x0018d6 thru 0x00f7d4 (0xdeff bytes) 000854 1787791747.312470 mapwrite 0x02a1dc thru 0x031fb6 (0x7ddb bytes) 000855 1787791747.421829 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x38b77): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38b77): Mirror 2: 0x3f6fa369 (stale) CRC-32 checksum value for chunk [0, 0x38b77): Mirror 1: 0x121a0b7f Mirror 3: 0x121a0b7f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38b77: skipped 000857 1787791748.325555 mapread 0x0231b7 thru 0x029903 (0x674d bytes) 000858 1787791748.346110 punch from 0x02e76a to 0x0333e6 (0x4c7d bytes) 000859 1787791748.387203 write 0x0169e0 thru 0x0243ab (0xd9cc bytes) 000860 1787791748.405903 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38b77): Mirror 3: 0x121a0b7f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38b77): Mirror 3: 0x121a0b7f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38b77: skipped 000861 1787791748.721771 MIRROR_RESYNC 2 mirrors 000862 1787791749.557585 mapread 0x02cf5e thru 0x02dfe4 (0x1087 bytes) 000863 1787791749.578674 mapread 0x013399 thru 0x01ae66 (0x7ace bytes) 000864 1787791749.593326 read 0x0375e1 thru 0x038b76 (0x1596 bytes) 000865 1787791749.598901 mapread 0x02f90c thru 0x038b76 (0x926b bytes) 000866 1787791749.610707 punch from 0x00cad9 to 0x01a0ca (0xd5f2 bytes) 000867 1787791749.654359 trunc from 0x001938 to 0x00d24c (0xb915 bytes) 000868 1787791749.659813 trunc from 0x008057 to 0x00d771 (0x571b bytes) 000869 1787791749.665643 mapread 0x035487 thru 0x03846a (0x2fe4 bytes) 000870 1787791749.677723 trunc from 0x0000f5 to 0x00ec2b (0xeb37 bytes) 000871 1787791749.685883 mapread 0x00195c thru 0x00f98a (0xe02f bytes) 000872 1787791749.707803 read 0x005ca2 thru 0x00b0fc (0x545b bytes) 000873 1787791749.708482 mapwrite 0x00242e thru 0x01224d (0xfe20 bytes) 000874 1787791749.767223 MIRROR_RESYNC 2 mirrors 000875 1787791750.741703 write 0x0104d4 thru 0x015973 (0x54a0 bytes) 000876 1787791750.797562 trunc from 0x01cf9a to 0x01d445 (0x4ac bytes) 000877 1787791750.804847 mapread 0x0289f3 thru 0x02eb66 (0x6174 bytes) 000878 1787791750.832710 trunc from 0x00ebd2 to 0x01de09 (0xf238 bytes) 000879 1787791750.851841 trunc from 0x001a1d to 0x01005e (0xe642 bytes) 000880 1787791750.857907 trunc from 0x038b77 to 0x0337ee (0x5388 bytes) 000881 1787791750.884528 fallocate 0x00d8c7 thru 0x014ab3 (0x71ed bytes) 000882 1787791750.896788 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e3:0x0]: Invalid argument (22) 000883 1787791751.063844 trunc from 0x01bf29 to 0x029872 (0xd94a bytes) 000884 1787791751.069815 mapwrite 0x004e92 thru 0x00876b (0x38da bytes) 000885 1787791751.136572 trunc from 0x0337ef to 0x002901 (0x30eed bytes) 000886 1787791751.186621 mapwrite 0x0368ec thru 0x036b1f (0x234 bytes) 000887 1787791751.252377 read 0x03252c thru 0x035ffa (0x3acf bytes) 000888 1787791751.253023 punch from 0x00b574 to 0x014441 (0x8ece bytes) 000889 1787791751.262443 mapwrite 0x02afd1 thru 0x030413 (0x5443 bytes) 000890 1787791751.298234 mapwrite 0x008e5a thru 0x013047 (0xa1ee bytes) 000891 1787791751.377344 read 0x014419 thru 0x02335a (0xef42 bytes) 000893 1787791751.388634 fallocate 0x0329c1 thru 0x033680 (0xcc0 bytes) 000894 1787791751.394811 trunc from 0x036b20 to 0x01b558 (0x1b5c7 bytes) 000895 1787791751.418142 trunc from 0x00bf88 to 0x017e38 (0xbeb1 bytes) 000896 1787791751.430710 mapwrite 0x0129b7 thru 0x01f092 (0xc6dc bytes) 000897 1787791751.515779 mapread 0x01d98e thru 0x01f092 (0x1705 bytes) 000898 1787791751.517230 mapwrite 0x03aaef thru 0x03ffff (0x5511 bytes) 000899 1787791751.567192 write 0x007e81 thru 0x00a809 (0x2989 bytes) 000901 1787791751.571320 mapread 0x02f2da thru 0x03e651 (0xf378 bytes) 000902 1787791751.605420 trunc from 0x040000 to 0x01242b (0x2dbd4 bytes) 000903 1787791751.624497 punch from 0x00c19e to 0x01242b (0x628e bytes) 000906 1787791751.637505 mapwrite 0x01bfc3 thru 0x029377 (0xd3b5 bytes) 000908 1787791751.796710 mapwrite 0x00c4a5 thru 0x0103bf (0x3f1b bytes) 000909 1787791751.906796 write 0x02d874 thru 0x039797 (0xbf24 bytes) 000910 1787791751.908427 read 0x00151e thru 0x002415 (0xef8 bytes) 000911 1787791751.916705 mapwrite 0x0244dc thru 0x026287 (0x1dac bytes) 000912 1787791751.939534 punch from 0x019267 to 0x02274a (0x94e4 bytes) 000913 1787791751.947685 mapread 0x02b974 thru 0x033919 (0x7fa6 bytes) 000914 1787791751.959254 mapwrite 0x00c382 thru 0x015acb (0x974a bytes) 000916 1787791751.983373 mapread 0x012740 thru 0x0175aa (0x4e6b bytes) 000917 1787791751.986324 write 0x008c98 thru 0x00ae76 (0x21df bytes) 000918 1787791751.991248 mapwrite 0x02c01f thru 0x02eccb (0x2cad bytes) 000919 1787791752.010867 mapread 0x012276 thru 0x014ba7 (0x2932 bytes) 000920 1787791752.013635 mapread 0x0029f6 thru 0x011039 (0xe644 bytes) 000922 1787791752.017039 punch from 0x00fec5 to 0x019371 (0x94ad bytes) 000923 1787791752.026624 read 0x01391e thru 0x01db22 (0xa205 bytes) 000925 1787791752.036863 fallocate 0x01dc49 thru 0x01fdaa (0x2162 bytes) 000927 1787791752.044499 mapwrite 0x037d1f thru 0x03ffff (0x82e1 bytes) 000928 1787791752.141507 write 0x01a674 thru 0x01ef32 (0x48bf bytes) 000929 1787791752.142586 trunc from 0x040000 to 0x00cb9c (0x33463 bytes) 000930 1787791752.178066 fallocate 0x03a0db thru 0x03ffff (0x5f25 bytes) 000931 1787791752.188877 punch from 0x039d1c to 0x03ffff (0x62e4 bytes) 000932 1787791752.194843 trunc from 0x03e364 to 0x03ffff (0x1c9c bytes) 000934 1787791752.213225 punch from 0x03d011 to 0x03ffff (0x2fef bytes) 000937 1787791752.227275 mapread 0x031355 thru 0x03ff9c (0xec48 bytes) 000938 1787791752.361451 fallocate 0x00f8e7 thru 0x01510f (0x5829 bytes) 000939 1787791752.366954 trunc from 0x02d8d1 to 0x03812a (0xa85a bytes) 000941 1787791752.372668 mapread 0x01ab84 thru 0x02982b (0xeca8 bytes) 000942 1787791752.497440 fallocate 0x01cfd1 thru 0x02c55f (0xf58f bytes) 000943 1787791752.508664 mapwrite 0x00f4a6 thru 0x01f419 (0xff74 bytes) 000944 1787791752.614678 mapread 0x018eb8 thru 0x026684 (0xd7cd bytes) 000945 1787791752.615885 fallocate 0x0214ca thru 0x0227cd (0x1304 bytes) 000947 1787791752.622621 mapread 0x01063b thru 0x0192a6 (0x8c6c bytes) 000950 1787791752.626762 punch from 0x00b98e to 0x00fc57 (0x42ca bytes) 000951 1787791752.634077 fallocate 0x01565d thru 0x016dd4 (0x1778 bytes) 000952 1787791752.639316 trunc from 0x032967 to 0x03ae64 (0x84fe bytes) 000954 1787791752.648255 write 0x03e458 thru 0x03ffff (0x1ba8 bytes) 000955 1787791752.650316 fallocate 0x0271a7 thru 0x02f3f1 (0x824b bytes) 000956 1787791752.656584 fallocate 0x01a082 thru 0x0258af (0xb82e bytes) 000957 1787791752.662233 punch from 0x008085 to 0x009c9a (0x1c16 bytes) 000958 1787791752.668075 mapwrite 0x03e256 thru 0x03ffff (0x1daa bytes) 000959 1787791752.682026 mapread 0x0020e9 thru 0x00979b (0x76b3 bytes) 000960 1787791752.696216 trunc from 0x040000 to 0x03d930 (0x26cf bytes) 000962 1787791752.717421 mapwrite 0x021d94 thru 0x02eb80 (0xcded bytes) 000963 1787791752.796546 trunc from 0x03d931 to 0x021b7d (0x1bdb3 bytes) 000964 1787791752.818676 mapwrite 0x0051cc thru 0x00802b (0x2e60 bytes) 000966 1787791752.837856 trunc from 0x0091df to 0x012023 (0x8e45 bytes) 000967 1787791752.846780 write 0x034b9e thru 0x03ffff (0xb462 bytes) 000968 1787791752.850735 fallocate 0x016a79 thru 0x0239dd (0xcf65 bytes) 000969 1787791752.860851 punch from 0x02ee62 to 0x02f33c (0x4db bytes) 000970 1787791752.881099 fallocate 0x00cc75 thru 0x01c209 (0xf595 bytes) 000972 1787791752.889133 read 0x032718 thru 0x036420 (0x3d09 bytes) 000974 1787791752.911641 mapread 0x014cbf thru 0x020d87 (0xc0c9 bytes) 000975 1787791752.913119 fallocate 0x02a839 thru 0x02f686 (0x4e4e bytes) 000976 1787791752.917887 mapread 0x0080cb thru 0x00ce39 (0x4d6f bytes) 000977 1787791752.919099 read 0x0091a2 thru 0x013593 (0xa3f2 bytes) 000978 1787791752.919741 write 0x03553d thru 0x039af8 (0x45bc bytes) 000979 1787791752.923416 trunc from 0x00fc27 to 0x017547 (0x7921 bytes) 000980 1787791752.929279 mapwrite 0x024108 thru 0x02cf59 (0x8e52 bytes) 000982 1787791752.964759 trunc from 0x040000 to 0x023f39 (0x1c0c6 bytes) 000983 1787791752.998913 trunc from 0x00de4b to 0x01b3c1 (0xd577 bytes) 000984 1787791753.013377 mapwrite 0x014dd0 thru 0x021417 (0xc648 bytes) 000985 1787791753.058422 mapwrite 0x0035e8 thru 0x00a442 (0x6e5b bytes) 000986 1787791753.076251 write 0x026c03 thru 0x02cf07 (0x6305 bytes) 000987 1787791753.080341 trunc from 0x02cf08 to 0x016c30 (0x162d7 bytes) 000988 1787791753.099108 mapread 0x0150bc thru 0x016c30 (0x1b75 bytes) 000990 1787791753.100533 punch from 0x0060a4 to 0x01207f (0xbfdc bytes) 000991 1787791753.109657 fallocate 0x03db18 thru 0x03fe2d (0x2316 bytes) 000992 1787791753.127240 mapread 0x03e5ba thru 0x03fe2d (0x1874 bytes) 000993 1787791753.146154 fallocate 0x03ddcc thru 0x03f269 (0x149e bytes) 000994 1787791753.163261 MIRROR_EXTEND to 2 mirrors 000995 1787791754.189913 mapwrite 0x03af6a thru 0x03e221 (0x32b8 bytes) 000996 1787791754.332109 punch from 0x031980 to 0x03fe2d (0xe4ae bytes) 000998 1787791754.352762 mapread 0x020c01 thru 0x02aa29 (0x9e29 bytes) 000999 1787791754.380386 mapwrite 0x030fcb thru 0x03d85a (0xc890 bytes) All operations completed A-OK! PASS 70a (62s) == sanity-flr test 71: check mirror extend parent fid ==== 20:49:20 (1787791760) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x216:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (8s) == sanity-flr test 72: check append on FLR file ========== 20:49:28 (1787791768) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.194363 s, 10.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132573 s, 7.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x117:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x117:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf9:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 20:49:36 (1787791776) Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 6173760 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7166080 .. Extending file size to 8135168 .. Extending file size to 7309184 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7843040 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8184992 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8588064 .. Extending file size to 8892576 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9109888 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8398272 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8651808 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 8913472 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 62722 62724 62726 62728 62729 failed 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) 192.168.204.131@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (74s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:50:51 (1787791851) Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8092960 .. Extending file size to 7642720 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8475552 .. Extending file size to 8425792 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9199936 .. Extending file size to 8631296 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9228128 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 65063 65064 65066 65067 65070 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) 192.168.204.131@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (73s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:52:03 (1787791923) Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2452 1268852 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2180 1269124 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 16888 3590132 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24820 3582200 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 41708 7172332 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2452 1268852 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2180 1269124 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 16888 3590132 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24820 3582200 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 41708 7172332 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0901854 s, 23.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.156522 s, 13.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x255349d (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) PASS 200c (13s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:52:17 (1787791937) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0833391 s, 25.2 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 20:52:23 (1787791943) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5f:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } PASS 203 (5s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:52:28 (1787791948) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.107658 s, 19.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:52:35 (1787791955) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.105817 s, 19.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:52:42 (1787791962) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.115196 s, 18.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x29e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x29c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x29e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x29c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x29d:0x0] } PASS 204c (31s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:53:13 (1787791993) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg431-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536166 s, 19.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x29f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x29f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x29e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x29f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x29f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x29e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2a0:0x0] } PASS 204d (46s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:53:59 (1787792039) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg431-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x138:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x139:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0966976 s, 21.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x138:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x139:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x138:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x139:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (44s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:54:43 (1787792083) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg431-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg431-server: Pool lustre.test_204f created Adding targets to pool oleg431-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg431-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.115942 s, 18.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg431-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg431-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg431-server: Pool lustre.test_204f destroyed PASS 204f (73s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 20:55:56 (1787792156) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 20:56:02 (1787792162) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:56:09 (1787792169) oleg431-server: Pool lustre.test_206 created Adding targets to pool oleg431-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg431-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg431-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg431-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg431-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:56:35 (1787792195) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489857 s, 21.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313526 s, 33.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x17fac9a2 Mirror 3: 0x17fac9a2 PASS 207 (7s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:56:42 (1787792202) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:56:45 (1787792205) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:56:48 (1787792208) Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0152014 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x24f0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x850000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x850000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x850000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xc80000 Read this (4 bytes): BBBB 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) PASS 209a (33s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:57:21 (1787792241) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0215794 s, 380 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:57:28 (1787792248) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498482 s, 21.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x80 lmm_fid: [0x200000405:0x80:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x80 lmm_fid: [0x200000405:0x80:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x124:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611264 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 20:57:29.000000000 -0400 Modify: 2026-08-26 20:57:29.000000000 -0400 Change: 2026-08-26 20:57:31.000000000 -0400 Birth: 2026-08-26 20:57:29.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:57:36 (1787792256) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200091 s, 52.4 MB/s oleg431-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg431-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg431-server: O/280000401/d5/145: File not found by ext2_lookup PASS 210b (24s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:58:00 (1787792280) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.329375 s, 124 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.047723 s, 85.8 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 26 20:58 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x85:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 26 20:58 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 20:58:10 (1787792290) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 20:58:17 (1787792297) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1535 sec =========== 20:58:25 (1787792305) === sanity-flr: start cleanup 20:58:26 (1787792306) === === sanity-flr: finish cleanup 20:58:34 (1787792314) ===