-----============= acceptance-small: sanity-flr ============----- Wed Sep 2 10:28:27 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg427-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 10:28:42 (1788359322) === oleg427-client.virtnet: executing check_config_client /mnt/lustre oleg427-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg427-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff96b5ca977000.idle_timeout=debug osc.lustre-OST0001-osc-ffff96b5ca977000.idle_timeout=debug disable quota as required oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 10:29:15 (1788359355) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8209] == sanity-flr test 0a: lfs mirror create with -N option == 10:29:16 (1788359356) PASS 0a (10s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:29:26 (1788359366) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:29:28 (1788359368) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 10:29:30 (1788359370) PASS 0d (13s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:29:43 (1788359383) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:29:44 (1788359384) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 10:29:46 (1788359386) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff96b5ca977000.stats=clear osc.lustre-OST0001-osc-ffff96b5ca977000.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:29:50 (1788359390) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (6s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:29:56 (1788359396) PASS 0j (5s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:30:01 (1788359401) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 0m: mirror extend with -N= option ===== 10:30:03 (1788359403) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 10:30:07 (1788359407) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (5s) == sanity-flr test 2: create components from existing files ========================================================== 10:30:12 (1788359412) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:30:15 (1788359415) PASS 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:30:19 (1788359419) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:30:22 (1788359422) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 10:30:27 (1788359427) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:30:30 (1788359430) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 0.719636 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489667 s, 21.4 MB/s PASS 21 (4s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:30:34 (1788359434) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.233972 s, 71.7 MB/s PASS 22 (4s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 10:30:38 (1788359438) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (5s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:30:43 (1788359443) Stopping /mnt/lustre-ost1 (opts:) on oleg427-server oleg427-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff96b5ca977000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff96b5ca977000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg427-server: oleg427-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg427-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg427-server oleg427-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff96b5ca977000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff96b5ca977000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg427-server: oleg427-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg427-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (39s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:31:22 (1788359482) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.263301 s, 39.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg427-server oleg427-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff96b5ca977000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff96b5ca977000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg427-server: oleg427-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg427-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (24s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:31:46 (1788359506) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg427-server oleg427-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff96b5ca977000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff96b5ca977000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg427-server: oleg427-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg427-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg427-server oleg427-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff96b5ca977000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff96b5ca977000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg427-server: oleg427-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg427-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (38s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:32:25 (1788359545) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.60884 s, 32.6 MB/s -rw-r--r-- 1 root root 52428800 Sep 2 10:32 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.514s user 0m0.001s sys 0m0.169s Stopping /mnt/lustre-ost1 (opts:) on oleg427-server oleg427-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff96b5ca977000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff96b5ca977000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.362s user 0m0.000s sys 0m0.163s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg427-server: oleg427-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg427-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg427-server oleg427-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff96b5ca977000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff96b5ca977000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.555s user 0m0.000s sys 0m0.171s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg427-server: oleg427-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg427-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg427-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff96b5ca977000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff96b5ca977000.ost_server_uuid in FULL state after 0 sec PASS 33b (43s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:33:09 (1788359589) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:33:10 (1788359590) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:33:12 (1788359592) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 10:33:13 (1788359593) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 10:33:16 (1788359596) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.346171 s, 36.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (7s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:33:23 (1788359603) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176673 s, 59.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209217 s, 50.1 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:33:26 (1788359606) PASS 36c (5s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:33:31 (1788359611) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000732544 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000903345 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00176661 s, 580 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00105817 s, 968 kB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 10:33:36 (1788359616) create test files with size 15756726 .. 968ed1c67282fa020979c2cb04912dc3 - 4c06a1a3dbad913d340ca531c706dd10 - 28c84bbfb31b301e03f189dd4d83403c - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg427-client.virtnet /mnt/lustre (opts:) Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre PASS 37 (9s) == sanity-flr test 38: resync ============================ 10:33:45 (1788359625) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (6s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:33:51 (1788359631) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:33:54 (1788359634) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293807 s, 35.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb3:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb4:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc4:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022694 s, 46.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc5:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc6:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb5:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (5s) == sanity-flr test 41: lfs mirror resync check =========== 10:33:59 (1788359639) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0404441 s, 51.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.154254 s, 27.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.082455 s, 25.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (6s) == sanity-flr test 42: lfs mirror verify ================= 10:34:05 (1788359645) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 10:34:07 (1788359647) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:34:08 (1788359648) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205764 s, 51.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 10:34:12 (1788359652) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.162255 s, 19.4 MB/s PASS 44a (5s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:34:17 (1788359657) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbe:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd0:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd0:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:34:22 (1788359662) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.141251 s, 74.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg427-client.virtnet /mnt/lustre (opts:) Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:34:34 (1788359674) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff96b5c8d41000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.144185 s, 72.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg427-client.virtnet /mnt/lustre (opts:) Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff96b5c2c39000.xattr_cache=1 PASS 44e (7s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:34:40 (1788359680) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (21s) == sanity-flr test 46: Verify setstripe --copy option ==== 10:35:01 (1788359701) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (19s) == sanity-flr test 47: Verify mirror obj alloc =========== 10:35:20 (1788359720) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 10:35:22 (1788359722) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.121902 s, 25.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0730579 s, 43.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xc37f4de6 (nosync,stale) Mirror 2: 0xc37f4de6 (nosync,stale) ** verify mirror 2 doesn't change original checksum: e360b8ce7c8fdb34e30a0535bf3d7130 - mirror 1 checksum: 177a127172fb2fa34ab8b36f788d318d - mirror 2 checksum: e360b8ce7c8fdb34e30a0535bf3d7130 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-02T10:35:25-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-02T10:35:25-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: e360b8ce7c8fdb34e30a0535bf3d7130 - mirror 1 checksum: 177a127172fb2fa34ab8b36f788d318d - mirror 2 checksum: 177a127172fb2fa34ab8b36f788d318d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-02T10:35:25-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-02T10:35:25-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } PASS 48 (5s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 10:35:27 (1788359727) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189171 s, 55.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (3s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 10:35:30 (1788359730) Failing mds1 on oleg427-server Stopping /mnt/lustre-mds1 (opts:) on oleg427-server 10:35:39 (1788359739) shut down facet: mds1 facet_host: oleg427-server facet_failover_host: oleg427-server Failover mds1 to oleg427-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0000 10:35:56 (1788359756) targets are mounted 10:35:56 (1788359756) facet_failover done oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134894 s, 77.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } PASS 50A (36s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 10:36:06 (1788359766) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.017408 s, 1.2 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0145467 s, 1.4 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 10:36:10 (1788359770) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0284519 s, 720 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00765806 s, 2.7 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (5s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 10:36:15 (1788359775) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00542835 s, 3.0 MB/s PASS 50c (4s) == sanity-flr test 50d: mirror rsync keep holes ========== 10:36:19 (1788359779) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0309377 s, 662 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 10:36:23 (1788359783) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00238749 s, 1.7 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0155489 s, 263 kB/s PASS 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 10:36:26 (1788359786) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) PASS 60b (11s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 10:36:37 (1788359797) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (21s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 10:36:58 (1788359818) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x48:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (19s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 10:37:17 (1788359837) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 10:37:36 (1788359856) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 10:37:39 (1788359859) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 57473 mirror split pid 57474 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5070: 57473 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5070: 57474 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 10:38:42 (1788359922) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 9923 skipping zero size read skipping zero length zero range skipping zero size read 000004 1788359923.709356 write 0x0395ad thru 0x03a5b8 (0x100c bytes) truncating to largest ever: 0xf772 000005 1788359923.711215 trunc from 0x03a5b9 to 0x00f771 (0x2ae47 bytes) 000006 1788359923.722606 read 0x003a50 thru 0x00f771 (0xbd22 bytes) 000008 1788359923.734747 mapwrite 0x032505 thru 0x034802 (0x22fe bytes) zero_range to largest ever: 0x10e89 000009 1788359923.770210 trunc from 0x0027f1 to 0x010e88 (0xe698 bytes) 000010 1788359923.772765 punch from 0x0304ea to 0x031a18 (0x152f bytes) zero_range to largest ever: 0x318ae 000011 1788359923.775777 trunc from 0x030e20 to 0x0318ad (0xa8e bytes) 000012 1788359923.778141 read 0x00768f thru 0x00d792 (0x6104 bytes) 000014 1788359923.785265 trunc from 0x012b7a to 0x0204d1 (0xd958 bytes) 000015 1788359923.788707 mapwrite 0x0133e5 thru 0x0172b4 (0x3ed0 bytes) 000016 1788359923.822940 trunc from 0x034803 to 0x008a07 (0x2bdfb bytes) 000017 1788359923.834346 MIRROR_EXTEND to 2 mirrors 000018 1788359924.100006 write 0x00c099 thru 0x013bdc (0x7b44 bytes) 000019 1788359924.118967 read 0x00ba1c thru 0x011b65 (0x614a bytes) 000020 1788359924.122328 read 0x006ea7 thru 0x00a6b7 (0x3811 bytes) 000021 1788359924.128515 mapread 0x005df2 thru 0x0115d6 (0xb7e5 bytes) 000023 1788359924.133116 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x13bdd): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13bdd): Mirror 2: 0x7deed0dd (stale) CRC-32 checksum value for chunk [0, 0x13bdd): Mirror 1: 0xc050691b Mirror 3: 0xc050691b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13bdd: skipped fallocating to largest ever: 0x376b5 000024 1788359924.472005 fallocate 0x030fc3 thru 0x0376b4 (0x66f2 bytes) 000025 1788359924.499039 trunc from 0x005947 to 0x012a5f (0xd119 bytes) 000026 1788359924.504393 trunc from 0x0376b5 to 0x013ed3 (0x237e1 bytes) 000027 1788359924.529628 trunc from 0x013ed4 to 0x0366ce (0x227fb bytes) truncating to largest ever: 0x3fbaa 000028 1788359924.540735 trunc from 0x0366cf to 0x03fba9 (0x94db bytes) 000029 1788359924.548838 trunc from 0x016047 to 0x019826 (0x37e0 bytes) truncating to largest ever: 0x3fe4a 000030 1788359924.551609 trunc from 0x03fbaa to 0x03fe49 (0x2a0 bytes) 000031 1788359924.560684 mapwrite 0x03cee1 thru 0x03ffff (0x311f bytes) 000032 1788359924.592394 punch from 0x001230 to 0x00b595 (0xa366 bytes) 000033 1788359924.595811 trunc from 0x040000 to 0x007937 (0x386c8 bytes) 000034 1788359924.606036 mapread 0x007302 thru 0x007937 (0x636 bytes) 000035 1788359924.612343 punch from 0x0068b9 to 0x007937 (0x107f bytes) 000036 1788359924.615738 write 0x00eda6 thru 0x013137 (0x4392 bytes) 000037 1788359924.616854 read 0x00b92c thru 0x013137 (0x780c bytes) 000038 1788359924.620251 read 0x00b0de thru 0x013137 (0x805a bytes) 000039 1788359924.621009 write 0x019950 thru 0x01f70a (0x5dbb bytes) 000040 1788359924.622363 trunc from 0x008ef4 to 0x00f51a (0x6627 bytes) 000041 1788359924.629419 punch from 0x01d02c to 0x01f70a (0x26df bytes) 000042 1788359924.631779 punch from 0x01cca0 to 0x01f70a (0x2a6b bytes) 000043 1788359924.634069 read 0x01e230 thru 0x01f70a (0x14db bytes) 000044 1788359924.638934 fallocate 0x018b8c thru 0x0239ce (0xae43 bytes) 000046 1788359924.642285 write 0x02ed76 thru 0x03e6d1 (0xf95c bytes) 000047 1788359924.644073 mapwrite 0x0205e6 thru 0x021c3f (0x165a bytes) 000048 1788359924.654104 mapwrite 0x00babe thru 0x01751f (0xba62 bytes) 000049 1788359924.664507 mapread 0x03b4dd thru 0x03e6d1 (0x31f5 bytes) 000052 1788359924.666253 write 0x02ae6f thru 0x034314 (0x94a6 bytes) 000053 1788359924.667543 trunc from 0x03e6d2 to 0x02dc6c (0x10a65 bytes) 000054 1788359924.674642 mapwrite 0x035539 thru 0x03ffff (0xaac7 bytes) 000055 1788359924.717494 mapwrite 0x003986 thru 0x003e1c (0x497 bytes) 000056 1788359924.725792 mapread 0x0089b2 thru 0x01157d (0x8bcc bytes) 000057 1788359924.726966 mapread 0x01f034 thru 0x027896 (0x8863 bytes) 000058 1788359924.728386 write 0x000c51 thru 0x00c323 (0xb6d3 bytes) 000059 1788359924.729886 write 0x02bf41 thru 0x02f0c9 (0x3189 bytes) 000060 1788359924.733709 trunc from 0x040000 to 0x007c75 (0x3838a bytes) 000061 1788359924.743887 read 0x006b3f thru 0x007c75 (0x1137 bytes) 000062 1788359924.744610 punch from 0x0037dd to 0x007c75 (0x4499 bytes) 000063 1788359924.753571 write 0x03c2ba thru 0x03ded0 (0x1c17 bytes) 000064 1788359924.754661 read 0x028306 thru 0x03368b (0xb386 bytes) 000065 1788359924.760528 fallocate 0x03abc3 thru 0x03db4c (0x2f8a bytes) 000066 1788359924.766457 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ded1): Mirror 3: 0x8bc617e9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ded1): Mirror 3: 0x8bc617e9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ded1: skipped 000067 1788359924.895521 MIRROR_RESYNC 2 mirrors 000068 1788359925.191540 write 0x005d28 thru 0x006887 (0xb60 bytes) 000069 1788359925.219011 trunc from 0x03ded1 to 0x02983d (0x14693 bytes) 000070 1788359925.228391 trunc from 0x025a7a to 0x02983d (0x3dc4 bytes) 000071 1788359925.230875 read 0x02457a thru 0x02983d (0x52c4 bytes) 000072 1788359925.238258 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3ee:0x0]: Invalid argument (22) 000074 1788359925.313820 read 0x021e69 thru 0x02983d (0x79d5 bytes) 000077 1788359925.323442 punch from 0x009191 to 0x018967 (0xf7d7 bytes) 000078 1788359925.328088 fallocate 0x00b925 thru 0x00bdf5 (0x4d1 bytes) 000079 1788359925.333945 fallocate 0x034770 thru 0x03b463 (0x6cf4 bytes) 000081 1788359925.338917 trunc from 0x03b464 to 0x00542a (0x36039 bytes) 000082 1788359925.349620 mapwrite 0x03c668 thru 0x03ffff (0x3998 bytes) 000083 1788359925.392670 mapread 0x03e29b thru 0x03ffff (0x1d65 bytes) 000086 1788359925.396385 write 0x0350be thru 0x036afa (0x1a3d bytes) 000087 1788359925.402600 trunc from 0x040000 to 0x00651b (0x39ae4 bytes) 000088 1788359925.412693 mapread 0x005f0a thru 0x00651b (0x612 bytes) 000089 1788359925.418394 trunc from 0x001c8a to 0x00651b (0x4892 bytes) 000090 1788359925.422291 write 0x02b320 thru 0x02c076 (0xd57 bytes) 000092 1788359925.425950 MIRROR_EXTEND to 2 mirrors 000093 1788359925.722795 trunc from 0x02c077 to 0x00fc33 (0x1c443 bytes) 000094 1788359925.763005 trunc from 0x002f3c to 0x00832f (0x53f4 bytes) 000095 1788359925.770420 MIRROR_RESYNC 2 mirrors 000096 1788359926.009709 write 0x0256e8 thru 0x028142 (0x2a5b bytes) 000097 1788359926.032595 read 0x00469d thru 0x00554a (0xeae bytes) 000098 1788359926.036377 trunc from 0x028143 to 0x01b691 (0xcab1 bytes) 000100 1788359926.064499 punch from 0x000161 to 0x00ab0c (0xa9ac bytes) 000101 1788359926.067617 read 0x018194 thru 0x01b691 (0x34fe bytes) 000102 1788359926.072275 write 0x026e28 thru 0x03481d (0xd9f6 bytes) 000103 1788359926.073937 fallocate 0x01e9f6 thru 0x02a530 (0xbb3b bytes) 000104 1788359926.078431 fallocate 0x008a06 thru 0x01184f (0x8e4a bytes) 000105 1788359926.087359 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3481e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3481e): Mirror 2: 0xbe86ae86 (stale) CRC-32 checksum value for chunk [0, 0x3481e): Mirror 1: 0x27ac2e0d Mirror 3: 0x27ac2e0d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3481e: skipped 000106 1788359926.419643 read 0x0316da thru 0x03481d (0x3144 bytes) 000107 1788359926.427913 mapwrite 0x013759 thru 0x01d589 (0x9e31 bytes) 000108 1788359926.506105 punch from 0x00e161 to 0x00f084 (0xf24 bytes) 000109 1788359926.509506 write 0x027cc5 thru 0x0284d2 (0x80e bytes) 000110 1788359926.519268 mapread 0x00c081 thru 0x013e2b (0x7dab bytes) 000111 1788359926.526011 fallocate 0x01c2ab thru 0x029346 (0xd09c bytes) 000112 1788359926.529678 read 0x0239d9 thru 0x0277ab (0x3dd3 bytes) 000113 1788359926.534794 mapread 0x0100ee thru 0x01a770 (0xa683 bytes) 000114 1788359926.536093 read 0x0126cc thru 0x016b23 (0x4458 bytes) 000115 1788359926.536783 trunc from 0x00c808 to 0x01b477 (0xec70 bytes) 000116 1788359926.544642 fallocate 0x0269a6 thru 0x029ba3 (0x31fe bytes) 000117 1788359926.548507 punch from 0x021c3d to 0x023322 (0x16e6 bytes) 000118 1788359926.552399 trunc from 0x00d5b7 to 0x015262 (0x7cac bytes) 000119 1788359926.558555 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3481e): Mirror 3: 0x27ac2e0d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3481e): Mirror 3: 0x27ac2e0d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3481e: skipped 000120 1788359926.706883 write 0x03f2f1 thru 0x03ffff (0xd0f bytes) 000121 1788359926.709907 read 0x02ec89 thru 0x03c685 (0xd9fd bytes) 000122 1788359926.714880 trunc from 0x013422 to 0x01ca39 (0x9618 bytes) 000123 1788359926.723792 trunc from 0x01451e to 0x01e3c7 (0x9eaa bytes) 000124 1788359926.727267 fallocate 0x013609 thru 0x0217d7 (0xe1cf bytes) 000125 1788359926.730871 mapread 0x03bd65 thru 0x03ffff (0x429b bytes) 000127 1788359926.738399 trunc from 0x040000 to 0x012a4d (0x2d5b2 bytes) 000128 1788359926.762819 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3ee:0x0]: Invalid argument (22) 000129 1788359926.831901 write 0x03d6ea thru 0x03ffff (0x2916 bytes) 000130 1788359926.836954 mapwrite 0x017f91 thru 0x01a60a (0x267a bytes) 000131 1788359926.862133 mapread 0x019903 thru 0x02180e (0x7f0c bytes) 000132 1788359926.863400 trunc from 0x040000 to 0x02ba67 (0x14598 bytes) 000133 1788359926.870126 mapwrite 0x02d83e thru 0x0309c4 (0x3187 bytes) 000134 1788359926.895273 trunc from 0x00a542 to 0x011294 (0x6d53 bytes) 000135 1788359926.898062 trunc from 0x0309c5 to 0x01a0ca (0x168fa bytes) 000136 1788359926.909015 MIRROR_EXTEND to 2 mirrors 000137 1788359927.159849 trunc from 0x007ec8 to 0x00e614 (0x674d bytes) 000138 1788359927.176384 read 0x0180b5 thru 0x01a0ca (0x2016 bytes) 000139 1788359927.182692 write 0x00bc5e thru 0x0192c5 (0xd668 bytes) 000141 1788359927.192815 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3ee:0x0]: Invalid argument (22) 000142 1788359927.267507 write 0x001e10 thru 0x0065bd (0x47ae bytes) 000143 1788359927.275289 trunc from 0x01a0cb to 0x03217c (0x180b2 bytes) 000145 1788359927.284737 punch from 0x009bde to 0x010be7 (0x700a bytes) 000146 1788359927.288089 punch from 0x01d205 to 0x0230f1 (0x5eed bytes) 000148 1788359927.292620 fallocate 0x015274 thru 0x016f3c (0x1cc9 bytes) 000149 1788359927.296407 trunc from 0x03217d to 0x038f79 (0x6dfd bytes) 000150 1788359927.305670 trunc from 0x038f7a to 0x030c34 (0x8345 bytes) 000151 1788359927.315680 read 0x01f750 thru 0x02b018 (0xb8c9 bytes) fallocating to largest ever: 0x40000 000152 1788359927.321665 fallocate 0x03dbc6 thru 0x03ffff (0x243a bytes) 000153 1788359927.326118 trunc from 0x00a8f7 to 0x013fd3 (0x96dd bytes) 000154 1788359927.330105 mapwrite 0x00d9b5 thru 0x018207 (0xa853 bytes) 000155 1788359927.361076 mapread 0x00b1dd thru 0x016652 (0xb476 bytes) 000159 1788359927.366025 trunc from 0x030c35 to 0x00003b (0x30bf9 bytes) 000160 1788359927.374723 mapread 0x000006 thru 0x00003b (0x36 bytes) 000161 1788359927.378965 write 0x01a00e thru 0x0297aa (0xf79d bytes) 000162 1788359927.380942 mapread 0x020324 thru 0x029362 (0x903f bytes) 000163 1788359927.382221 punch from 0x02809d to 0x0297aa (0x170e bytes) 000164 1788359927.391289 mapread 0x01aa55 thru 0x0275b8 (0xcb64 bytes) 000165 1788359927.392378 trunc from 0x0297ab to 0x02a465 (0xcbb bytes) 000166 1788359927.400227 read 0x028d75 thru 0x028e48 (0xd4 bytes) 000167 1788359927.405680 mapwrite 0x0151a9 thru 0x021e4d (0xcca5 bytes) 000168 1788359927.417772 write 0x03bacd thru 0x03ffff (0x4533 bytes) 000170 1788359927.420293 trunc from 0x040000 to 0x0037fa (0x3c805 bytes) 000171 1788359927.433619 trunc from 0x0037fb to 0x0080d2 (0x48d8 bytes) 000172 1788359927.442283 write 0x03ab9e thru 0x03f18c (0x45ef bytes) 000173 1788359927.443649 mapread 0x021d8c thru 0x02a780 (0x89f5 bytes) 000174 1788359927.477292 read 0x01db23 thru 0x02cdcc (0xf2aa bytes) 000176 1788359927.483985 fallocate 0x01a515 thru 0x01e780 (0x426c bytes) 000177 1788359927.488068 punch from 0x0216ab to 0x029e89 (0x87df bytes) 000178 1788359927.490552 punch from 0x009a55 to 0x00afed (0x1599 bytes) 000179 1788359927.495170 read 0x01bb0f thru 0x02a96e (0xee60 bytes) 000181 1788359927.503985 MIRROR_EXTEND to 2 mirrors 000182 1788359927.770525 mapread 0x005e85 thru 0x007084 (0x1200 bytes) 000183 1788359927.779169 read 0x02b439 thru 0x02e96d (0x3535 bytes) 000184 1788359927.783405 read 0x000f9e thru 0x010366 (0xf3c9 bytes) 000185 1788359927.788510 trunc from 0x03f18d to 0x017f83 (0x27209 bytes) 000186 1788359927.821621 mapwrite 0x02c43a thru 0x0392e9 (0xceb0 bytes) 000188 1788359927.897793 MIRROR_RESYNC 2 mirrors 000189 1788359928.197380 trunc from 0x02bc64 to 0x0392e9 (0xd686 bytes) 000190 1788359928.219854 trunc from 0x0392ea to 0x0074cc (0x31e1d bytes) 000192 1788359928.234607 MIRROR_RESYNC 2 mirrors 000193 1788359928.467694 fallocate 0x002f8e thru 0x00d852 (0xa8c5 bytes) 000194 1788359928.486858 write 0x031d76 thru 0x037645 (0x58d0 bytes) 000195 1788359928.488279 trunc from 0x036b95 to 0x037645 (0xab1 bytes) 000196 1788359928.500522 fallocate 0x03bcf5 thru 0x03ffff (0x430b bytes) 000197 1788359928.506182 read 0x029203 thru 0x0303ee (0x71ec bytes) 000198 1788359928.511994 mapwrite 0x0239bc thru 0x029349 (0x598e bytes) 000199 1788359928.539918 punch from 0x03d137 to 0x03ffff (0x2ec9 bytes) 000200 1788359928.543782 mapread 0x039e94 thru 0x03ffff (0x616c bytes) 000201 1788359928.552145 fallocate 0x001719 thru 0x00c959 (0xb241 bytes) 000202 1788359928.558417 mapread 0x01416d thru 0x018d00 (0x4b94 bytes) 000203 1788359928.567180 punch from 0x02c316 to 0x0307c6 (0x44b1 bytes) 000204 1788359928.572586 fallocate 0x0311e1 thru 0x03cf62 (0xbd82 bytes) 000205 1788359928.576149 punch from 0x02cd73 to 0x03bead (0xf13b bytes) 000206 1788359928.582919 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x31c036a8 Mirror 3: 0x31c036a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000207 1788359928.963143 punch from 0x00a948 to 0x00e4fd (0x3bb6 bytes) 000208 1788359928.982157 trunc from 0x026365 to 0x028e75 (0x2b11 bytes) 000209 1788359928.984658 trunc from 0x03fdf9 to 0x03ffff (0x207 bytes) 000210 1788359928.987073 write 0x02f152 thru 0x03dd5d (0xec0c bytes) 000212 1788359928.994871 mapread 0x007ba3 thru 0x014c0c (0xd06a bytes) 000213 1788359929.010661 trunc from 0x03dc13 to 0x03ffff (0x23ed bytes) 000214 1788359929.018144 write 0x00318e thru 0x00d78c (0xa5ff bytes) 000215 1788359929.019402 mapwrite 0x034a9a thru 0x03ffff (0xb566 bytes) 000216 1788359929.034483 write 0x02a2d6 thru 0x02d341 (0x306c bytes) 000217 1788359929.041004 mapwrite 0x0189b1 thru 0x026ece (0xe51e bytes) 000218 1788359929.065042 MIRROR_RESYNC 3 mirrors 000219 1788359929.509829 trunc from 0x0396d7 to 0x03b22c (0x1b56 bytes) 000220 1788359929.530345 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xc1371856 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xc1371856 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000221 1788359929.660776 punch from 0x037572 to 0x03ffff (0x8a8e bytes) 000222 1788359929.664027 mapwrite 0x018936 thru 0x01ca47 (0x4112 bytes) 000223 1788359929.686491 fallocate 0x019a97 thru 0x01b3a0 (0x190a bytes) 000224 1788359929.690279 trunc from 0x02380c to 0x02f94d (0xc142 bytes) 000225 1788359929.693460 punch from 0x027833 to 0x0364fd (0xeccb bytes) 000226 1788359929.699999 mapread 0x0253dc thru 0x03343e (0xe063 bytes) 000227 1788359929.708931 punch from 0x023fd1 to 0x032425 (0xe455 bytes) 000228 1788359929.713691 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3ee:0x0]: Invalid argument (22) 000231 1788359929.776548 read 0x03eade thru 0x03ffff (0x1522 bytes) 000232 1788359929.781635 fallocate 0x001c97 thru 0x00a304 (0x866e bytes) 000233 1788359929.787012 write 0x01cd52 thru 0x02489a (0x7b49 bytes) 000235 1788359929.796881 trunc from 0x028d26 to 0x02e2b9 (0x5594 bytes) 000236 1788359929.799560 punch from 0x025f2d to 0x034862 (0xe936 bytes) 000237 1788359929.801991 trunc from 0x040000 to 0x01c54b (0x23ab4 bytes) 000238 1788359929.820043 fallocate 0x03f539 thru 0x03ffff (0xac7 bytes) 000239 1788359929.824466 trunc from 0x01f711 to 0x020177 (0xa67 bytes) 000240 1788359929.828168 trunc from 0x040000 to 0x00a452 (0x35bad bytes) 000241 1788359929.840331 write 0x03d258 thru 0x03fcf4 (0x2a9d bytes) 000242 1788359929.841604 mapwrite 0x0083ed thru 0x00b848 (0x345c bytes) 000244 1788359929.862827 mapwrite 0x032f8f thru 0x0350e0 (0x2152 bytes) 000246 1788359929.879511 mapwrite 0x028066 thru 0x02815b (0xf6 bytes) 000248 1788359929.891311 fallocate 0x0321f1 thru 0x03acce (0x8ade bytes) 000249 1788359929.895368 fallocate 0x002137 thru 0x00ea38 (0xc902 bytes) 000250 1788359929.898821 punch from 0x01c6fc to 0x02b8c0 (0xf1c5 bytes) 000251 1788359929.901718 mapread 0x03364d thru 0x039f98 (0x694c bytes) 000252 1788359929.902721 mapread 0x0167a7 thru 0x01f036 (0x8890 bytes) 000253 1788359929.913385 fallocate 0x01652d thru 0x020e30 (0xa904 bytes) 000254 1788359929.918986 trunc from 0x0330af to 0x03fcf4 (0xcc46 bytes) 000255 1788359929.922558 punch from 0x01ae04 to 0x01f231 (0x442e bytes) 000256 1788359929.926779 fallocate 0x036d72 thru 0x038da5 (0x2034 bytes) 000257 1788359929.930128 read 0x03c309 thru 0x03fcf4 (0x39ec bytes) 000258 1788359929.936566 write 0x038f9b thru 0x03ffff (0x7065 bytes) 000259 1788359929.937832 mapread 0x025524 thru 0x032100 (0xcbdd bytes) 000261 1788359929.940095 trunc from 0x02f710 to 0x02fd05 (0x5f6 bytes) 000262 1788359929.945602 trunc from 0x040000 to 0x0320e1 (0xdf1e bytes) 000263 1788359929.956439 read 0x005ece thru 0x00a3bf (0x44f2 bytes) 000264 1788359929.957004 mapread 0x00839f thru 0x012942 (0xa5a4 bytes) 000265 1788359929.958092 mapwrite 0x022206 thru 0x03054c (0xe347 bytes) 000266 1788359929.975800 mapread 0x013d1f thru 0x018ff9 (0x52db bytes) 000268 1788359929.977131 punch from 0x01ced4 to 0x028dbf (0xbeec bytes) 000269 1788359929.980545 mapread 0x02d408 thru 0x0320e1 (0x4cda bytes) 000270 1788359929.981990 read 0x005968 thru 0x00ef3e (0x95d7 bytes) 000271 1788359929.982669 fallocate 0x036296 thru 0x0383eb (0x2156 bytes) 000272 1788359929.989622 trunc from 0x0320e2 to 0x00ae5d (0x27284 bytes) 000273 1788359929.998278 punch from 0x001960 to 0x00ae5d (0x94fe bytes) 000274 1788359930.002459 punch from 0x002de2 to 0x00ae5d (0x807c bytes) 000275 1788359930.006591 punch from 0x002bd9 to 0x00680f (0x3c37 bytes) 000277 1788359930.009897 mapwrite 0x02685f thru 0x02ef87 (0x8729 bytes) 000279 1788359930.039073 trunc from 0x02ef88 to 0x007a6e (0x27519 bytes) 000280 1788359930.045801 mapwrite 0x02f06c thru 0x038d02 (0x9c97 bytes) 000281 1788359930.069879 trunc from 0x00dee7 to 0x01bf26 (0xe040 bytes) 000282 1788359930.073433 trunc from 0x01f690 to 0x0264b3 (0x6e24 bytes) 000283 1788359930.076438 trunc from 0x020bf7 to 0x02cd68 (0xc172 bytes) 000284 1788359930.079641 punch from 0x0361d6 to 0x038d02 (0x2b2d bytes) 000285 1788359930.082543 read 0x012080 thru 0x016a6d (0x49ee bytes) 000288 1788359930.090390 fallocate 0x00ff3d thru 0x01bd7d (0xbe41 bytes) 000289 1788359930.095269 mapwrite 0x025bfe thru 0x02efd0 (0x93d3 bytes) 000290 1788359930.106314 read 0x010774 thru 0x01d4d1 (0xcd5e bytes) 000291 1788359930.106934 trunc from 0x038d03 to 0x01d6af (0x1b653 bytes) 000292 1788359930.114839 mapread 0x00915c thru 0x01009b (0x6f40 bytes) 000294 1788359930.116109 mapwrite 0x00b4fa thru 0x01adf6 (0xf8fd bytes) 000295 1788359930.126841 read 0x012e7d thru 0x01bda2 (0x8f26 bytes) 000298 1788359930.129537 trunc from 0x01814a to 0x01d6af (0x5566 bytes) 000300 1788359930.133535 mapwrite 0x03bc1d thru 0x03ffff (0x43e3 bytes) 000301 1788359930.152221 fallocate 0x021307 thru 0x030f90 (0xfc8a bytes) 000302 1788359930.154838 trunc from 0x03b75b to 0x03fb98 (0x443e bytes) 000303 1788359930.157751 punch from 0x003933 to 0x00f6dc (0xbdaa bytes) 000305 1788359930.160967 punch from 0x01cfe8 to 0x025c38 (0x8c51 bytes) 000306 1788359930.164452 trunc from 0x00bfdc to 0x01792b (0xb950 bytes) 000307 1788359930.167502 write 0x033cbf thru 0x039ac6 (0x5e08 bytes) 000308 1788359930.168576 trunc from 0x040000 to 0x007385 (0x38c7a bytes) 000309 1788359930.176458 mapread 0x0012d9 thru 0x007385 (0x60ad bytes) 000310 1788359930.190443 read 0x003416 thru 0x007385 (0x3f70 bytes) 000311 1788359930.191109 mapread 0x004597 thru 0x007385 (0x2def bytes) 000313 1788359930.195276 MIRROR_EXTEND to 2 mirrors 000314 1788359930.440288 mapread 0x005852 thru 0x007385 (0x1b34 bytes) 000315 1788359930.453842 trunc from 0x007386 to 0x026dda (0x1fa55 bytes) 000317 1788359930.500302 trunc from 0x01318d to 0x01c2f5 (0x9169 bytes) 000318 1788359930.504704 fallocate 0x01afe8 thru 0x02a564 (0xf57d bytes) 000319 1788359930.509337 read 0x0024f2 thru 0x00389e (0x13ad bytes) 000320 1788359930.512848 trunc from 0x02a565 to 0x00bf42 (0x1e622 bytes) 000321 1788359930.527918 write 0x0170ce thru 0x0207e5 (0x9718 bytes) 000322 1788359930.529422 punch from 0x016f23 to 0x0207e5 (0x98c3 bytes) 000323 1788359930.537578 mapwrite 0x03c995 thru 0x03ffff (0x366b bytes) 000324 1788359930.568698 write 0x023580 thru 0x02ac4d (0x76ce bytes) 000325 1788359930.574521 read 0x01a3da thru 0x022de1 (0x8a08 bytes) 000326 1788359930.579675 write 0x03ad26 thru 0x03ffff (0x52da bytes) 000328 1788359930.581428 fallocate 0x00bbe6 thru 0x012af7 (0x6f12 bytes) 000329 1788359930.585759 trunc from 0x040000 to 0x02bad9 (0x14526 bytes) 000330 1788359930.596617 write 0x0329e4 thru 0x03b8db (0x8ef8 bytes) 000331 1788359930.597774 mapread 0x02e7f9 thru 0x030f64 (0x276c bytes) 000332 1788359930.608506 read 0x007951 thru 0x015096 (0xd746 bytes) 000333 1788359930.609309 fallocate 0x0096db thru 0x01448f (0xadb5 bytes) 000334 1788359930.613723 trunc from 0x020d69 to 0x02b0e9 (0xa381 bytes) 000335 1788359930.621992 trunc from 0x02f70e to 0x03b8db (0xc1ce bytes) 000337 1788359930.626735 mapread 0x02e1cc thru 0x031104 (0x2f39 bytes) 000338 1788359930.632015 mapread 0x035111 thru 0x03b8db (0x67cb bytes) 000339 1788359930.635535 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3ee:0x0]: Invalid argument (22) 000341 1788359930.698813 mapread 0x02adb7 thru 0x0367f7 (0xba41 bytes) 000342 1788359930.718605 read 0x01beb3 thru 0x01ff54 (0x40a2 bytes) 000343 1788359930.724122 mapread 0x03594e thru 0x03b8db (0x5f8e bytes) 000344 1788359930.725304 read 0x034b37 thru 0x039c85 (0x514f bytes) 000345 1788359930.725957 mapread 0x0304e3 thru 0x03b8db (0xb3f9 bytes) 000346 1788359930.727148 mapread 0x01d0cd thru 0x02446e (0x73a2 bytes) 000348 1788359930.733558 trunc from 0x03b8dc to 0x028ba4 (0x12d37 bytes) 000349 1788359930.742014 write 0x00f167 thru 0x01e341 (0xf1db bytes) 000350 1788359930.743399 write 0x032432 thru 0x035b38 (0x3707 bytes) 000352 1788359930.744924 fallocate 0x02643f thru 0x029c8d (0x384f bytes) 000353 1788359930.748248 trunc from 0x035b39 to 0x02c3a5 (0x9793 bytes) 000354 1788359930.755038 mapwrite 0x01c8b3 thru 0x01f784 (0x2ed2 bytes) 000355 1788359930.765704 mapread 0x01973c thru 0x01e999 (0x525e bytes) 000358 1788359930.768691 write 0x00586b thru 0x0149dc (0xf172 bytes) 000359 1788359930.770090 read 0x004355 thru 0x00d59d (0x9249 bytes) 000360 1788359930.770733 fallocate 0x027b5d thru 0x02e236 (0x66da bytes) 000362 1788359930.775691 fallocate 0x029c6d thru 0x02b009 (0x139d bytes) 000364 1788359930.781092 trunc from 0x00350c to 0x005fb2 (0x2aa7 bytes) 000365 1788359930.791518 write 0x00e27e thru 0x00ed84 (0xb07 bytes) 000366 1788359930.792535 write 0x007064 thru 0x00ef99 (0x7f36 bytes) 000368 1788359930.794709 mapread 0x02a596 thru 0x02cc7b (0x26e6 bytes) 000369 1788359930.801026 punch from 0x01e338 to 0x02305d (0x4d26 bytes) 000370 1788359930.805641 fallocate 0x00c64a thru 0x0156f6 (0x90ad bytes) 000372 1788359930.810118 punch from 0x0175fe to 0x0195f9 (0x1ffc bytes) 000373 1788359930.813552 mapread 0x0242a9 thru 0x029b4b (0x58a3 bytes) 000374 1788359930.814608 punch from 0x0005de to 0x00398b (0x33ae bytes) 000375 1788359930.819270 punch from 0x013b68 to 0x019840 (0x5cd9 bytes) 000377 1788359930.824523 mapwrite 0x01800b thru 0x024daf (0xcda5 bytes) 000379 1788359930.871690 trunc from 0x02e237 to 0x02cbbb (0x167b bytes) 000380 1788359930.880271 mapwrite 0x01ba79 thru 0x0234f2 (0x7a7a bytes) 000381 1788359930.894300 trunc from 0x02cbbc to 0x03404b (0x7490 bytes) 000382 1788359930.904178 punch from 0x0010ce to 0x00b72e (0xa661 bytes) 000383 1788359930.907331 read 0x01cc1e thru 0x020ed1 (0x42b4 bytes) 000384 1788359930.907985 fallocate 0x001b79 thru 0x010a4b (0xeed3 bytes) 000386 1788359930.913791 fallocate 0x02e817 thru 0x03c22a (0xda14 bytes) 000387 1788359930.921881 MIRROR_EXTEND to 2 mirrors 000388 1788359931.322529 MIRROR_EXTEND to 3 mirrors 000389 1788359931.659919 write 0x033f14 thru 0x0354cf (0x15bc bytes) 000390 1788359931.684158 mapwrite 0x009eda thru 0x00da31 (0x3b58 bytes) 000391 1788359931.723730 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3c22b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c22b): Mirror 2: 0xa4d9abcf (stale) Mirror 3: 0xa4d9abcf (stale) CRC-32 checksum value for chunk [0, 0x3c22b): Mirror 1: 0xdd6a60d6 Mirror 4: 0xdd6a60d6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c22b: skipped 000392 1788359932.181447 write 0x013e5f thru 0x0197ce (0x5970 bytes) 000393 1788359932.217759 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c22b): Mirror 2: 0xa4d9abcf (stale) Mirror 4: 0xdd6a60d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c22b): Mirror 2: 0xa4d9abcf (stale) Mirror 4: 0xdd6a60d6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c22b: skipped 000394 1788359932.436782 mapwrite 0x0262b4 thru 0x034dce (0xeb1b bytes) 000395 1788359932.472445 MIRROR_RESYNC 3 mirrors 000396 1788359932.880441 trunc from 0x03c22b to 0x01d049 (0x1f1e1 bytes) 000397 1788359932.929819 fallocate 0x026549 thru 0x03494a (0xe402 bytes) 000398 1788359932.935455 punch from 0x01ae32 to 0x01d049 (0x2218 bytes) 000399 1788359932.939061 trunc from 0x01c241 to 0x01d049 (0xe09 bytes) 000400 1788359932.943738 write 0x013fcf thru 0x017e75 (0x3ea7 bytes) 000401 1788359932.950715 trunc from 0x01213b to 0x01d049 (0xaf0f bytes) 000402 1788359932.959715 read 0x008a46 thru 0x00cea6 (0x4461 bytes) 000403 1788359932.966139 fallocate 0x003ce3 thru 0x01314e (0xf46c bytes) 000404 1788359932.974071 trunc from 0x01d04a to 0x01ca8a (0x5bf bytes) 000405 1788359932.989512 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ca8b): Mirror 2: 0x2dfbc876 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ca8b): Mirror 2: 0x2dfbc876 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ca8b: skipped 000407 1788359933.178985 punch from 0x0000dc to 0x00aa48 (0xa96d bytes) 000408 1788359933.185901 write 0x00a933 thru 0x014a8d (0xa15b bytes) 000409 1788359933.196939 read 0x015d71 thru 0x01ca8a (0x6d1a bytes) 000410 1788359933.202239 trunc from 0x004ca5 to 0x00af8c (0x62e8 bytes) 000411 1788359933.214405 trunc from 0x01ca8b to 0x004819 (0x18271 bytes) 000412 1788359933.228426 trunc from 0x00481a to 0x0105e5 (0xbdcc bytes) 000413 1788359933.238397 trunc from 0x002595 to 0x00ef83 (0xc9ef bytes) 000414 1788359933.243101 mapread 0x00c797 thru 0x0105e5 (0x3e4f bytes) 000415 1788359933.256289 write 0x0329de thru 0x036557 (0x3b7a bytes) 000418 1788359933.258647 write 0x039493 thru 0x03ffff (0x6b6d bytes) 000421 1788359933.261179 write 0x02b3a4 thru 0x037014 (0xbc71 bytes) 000422 1788359933.270500 write 0x0237ef thru 0x029826 (0x6038 bytes) 000423 1788359933.278156 trunc from 0x021c0c to 0x03004c (0xe441 bytes) 000424 1788359933.286840 punch from 0x02cda2 to 0x02e5d8 (0x1837 bytes) 000425 1788359933.294438 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x2cb8c33e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x35230ecb Mirror 3: 0x35230ecb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000426 1788359933.746093 trunc from 0x000073 to 0x008dd6 (0x8d64 bytes) 000427 1788359933.771651 punch from 0x030140 to 0x035015 (0x4ed6 bytes) 000428 1788359933.774813 read 0x0319b0 thru 0x032e39 (0x148a bytes) 000429 1788359933.780705 fallocate 0x003760 thru 0x00d178 (0x9a19 bytes) 000430 1788359933.785612 trunc from 0x033c3b to 0x03a138 (0x64fe bytes) 000431 1788359933.789858 trunc from 0x036f3c to 0x03cce1 (0x5da6 bytes) 000432 1788359933.793375 trunc from 0x03d86d to 0x03ffff (0x2793 bytes) 000433 1788359933.796949 punch from 0x01b80e to 0x02afa9 (0xf79c bytes) 000435 1788359933.804976 MIRROR_RESYNC 3 mirrors 000436 1788359934.263888 write 0x03e8f0 thru 0x03ffff (0x1710 bytes) 000437 1788359934.280166 mapread 0x00b4ac thru 0x01113c (0x5c91 bytes) 000438 1788359934.293986 read 0x01442c thru 0x0208dd (0xc4b2 bytes) 000439 1788359934.299283 write 0x0212e8 thru 0x027f7c (0x6c95 bytes) 000440 1788359934.305665 mapread 0x01cc96 thru 0x02a5cf (0xd93a bytes) 000441 1788359934.314898 mapwrite 0x00ff12 thru 0x01e029 (0xe118 bytes) 000443 1788359934.337472 punch from 0x028eb5 to 0x03222a (0x9376 bytes) 000444 1788359934.344672 mapread 0x017426 thru 0x01a79b (0x3376 bytes) 000445 1788359934.350907 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf8c1a6cb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf8c1a6cb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000446 1788359934.519252 read 0x01739e thru 0x01a08b (0x2cee bytes) 000447 1788359934.525576 trunc from 0x023bc7 to 0x0285cc (0x4a06 bytes) 000448 1788359934.530884 fallocate 0x00a50b thru 0x0169c9 (0xc4bf bytes) 000449 1788359934.534337 trunc from 0x02a7e1 to 0x0322dc (0x7afc bytes) 000450 1788359934.538136 write 0x031acb thru 0x0333f5 (0x192b bytes) 000451 1788359934.546427 trunc from 0x02853c to 0x031947 (0x940c bytes) 000452 1788359934.552963 punch from 0x01b6a5 to 0x026887 (0xb1e3 bytes) 000454 1788359934.557412 punch from 0x0371a8 to 0x03ffff (0x8e58 bytes) 000455 1788359934.561654 mapread 0x005b58 thru 0x011c4c (0xc0f5 bytes) 000456 1788359934.580396 mapread 0x022010 thru 0x02ba4e (0x9a3f bytes) 000457 1788359934.600473 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3ee:0x0]: Invalid argument (22) 000458 1788359934.682033 mapwrite 0x011458 thru 0x01c4f3 (0xb09c bytes) 000459 1788359934.713311 mapwrite 0x008106 thru 0x008640 (0x53b bytes) 000460 1788359934.727072 fallocate 0x039ced thru 0x03fbe9 (0x5efd bytes) 000461 1788359934.730398 read 0x00b049 thru 0x0195af (0xe567 bytes) 000462 1788359934.731296 trunc from 0x03dd87 to 0x03ffff (0x2279 bytes) 000463 1788359934.734416 read 0x0259c6 thru 0x0335b6 (0xdbf1 bytes) 000464 1788359934.742699 trunc from 0x040000 to 0x010e90 (0x2f16f bytes) 000465 1788359934.760327 trunc from 0x010e91 to 0x02764c (0x167bc bytes) 000466 1788359934.771173 punch from 0x0006da to 0x00a648 (0x9f6f bytes) 000467 1788359934.774903 read 0x0126bc thru 0x01550f (0x2e54 bytes) 000468 1788359934.784958 punch from 0x01b1ae to 0x01c31d (0x1170 bytes) 000469 1788359934.788587 read 0x016c8b thru 0x019369 (0x26df bytes) 000471 1788359934.789927 fallocate 0x01c956 thru 0x024279 (0x7924 bytes) 000472 1788359934.795320 mapwrite 0x015c93 thru 0x019aec (0x3e5a bytes) 000473 1788359934.809829 write 0x03cc52 thru 0x03ffff (0x33ae bytes) 000474 1788359934.811039 mapread 0x0072ee thru 0x011a6a (0xa77d bytes) 000475 1788359934.812362 mapread 0x00eeb5 thru 0x0168f9 (0x7a45 bytes) 000476 1788359934.813435 trunc from 0x040000 to 0x01759e (0x28a61 bytes) 000477 1788359934.825898 mapread 0x010412 thru 0x01759e (0x718d bytes) 000478 1788359934.826950 punch from 0x00a065 to 0x013338 (0x92d4 bytes) 000479 1788359934.831134 trunc from 0x01759f to 0x00929d (0xe301 bytes) 000480 1788359934.845660 trunc from 0x00929e to 0x00144c (0x7e51 bytes) 000481 1788359934.859047 read 0x0004de thru 0x00144c (0xf6f bytes) 000482 1788359934.859589 trunc from 0x0005f6 to 0x00144c (0xe57 bytes) 000483 1788359934.863052 trunc from 0x001047 to 0x00144c (0x406 bytes) 000484 1788359934.867647 read 0x000581 thru 0x000ce0 (0x760 bytes) 000485 1788359934.872997 mapwrite 0x00d70b thru 0x013e4f (0x6745 bytes) 000486 1788359934.902770 write 0x0325f6 thru 0x035008 (0x2a13 bytes) 000487 1788359934.904050 trunc from 0x02cc40 to 0x035008 (0x83c9 bytes) 000488 1788359934.912908 fallocate 0x03a85b thru 0x03ffff (0x57a5 bytes) 000489 1788359934.918515 write 0x023370 thru 0x02d90f (0xa5a0 bytes) 000490 1788359934.925267 write 0x027d83 thru 0x03568f (0xd90d bytes) 000491 1788359934.926749 fallocate 0x02dc3e thru 0x035643 (0x7a06 bytes) 000493 1788359934.932815 trunc from 0x03adcc to 0x03e501 (0x3736 bytes) 000494 1788359934.941488 MIRROR_EXTEND to 2 mirrors 000495 1788359935.312348 punch from 0x03d496 to 0x03ef56 (0x1ac1 bytes) 000496 1788359935.339163 fallocate 0x035f90 thru 0x0393e2 (0x3453 bytes) 000497 1788359935.343491 read 0x00b5b9 thru 0x00bc94 (0x6dc bytes) 000498 1788359935.347534 trunc from 0x007954 to 0x00e5f4 (0x6ca1 bytes) 000499 1788359935.351744 mapread 0x01e4ed thru 0x02d584 (0xf098 bytes) 000500 1788359935.367443 fallocate 0x009fa7 thru 0x00bafe (0x1b58 bytes) 000501 1788359935.370293 read 0x0054e5 thru 0x009be1 (0x46fd bytes) 000502 1788359935.376564 read 0x02ce2d thru 0x03747b (0xa64f bytes) 000503 1788359935.377305 trunc from 0x040000 to 0x02d22b (0x12dd4 bytes) 000504 1788359935.389385 fallocate 0x01709d thru 0x01d89f (0x6803 bytes) 000506 1788359935.392883 read 0x00a317 thru 0x00b95a (0x1644 bytes) 000507 1788359935.393521 fallocate 0x02b7e2 thru 0x035b9c (0xa3bb bytes) 000508 1788359935.401977 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3ee:0x0]: Invalid argument (22) 000509 1788359935.476408 read 0x035316 thru 0x035b9c (0x887 bytes) 000511 1788359935.480807 read 0x02e406 thru 0x035b9c (0x7797 bytes) 000512 1788359935.488810 trunc from 0x035b9d to 0x038247 (0x26ab bytes) 000513 1788359935.499139 mapread 0x0336dd thru 0x038247 (0x4b6b bytes) 000514 1788359935.503439 punch from 0x00125c to 0x005e25 (0x4bca bytes) 000515 1788359935.506773 mapread 0x03266a thru 0x038247 (0x5bde bytes) 000516 1788359935.510092 MIRROR_EXTEND to 2 mirrors 000517 1788359935.845268 punch from 0x0218d6 to 0x02a52d (0x8c58 bytes) 000518 1788359935.865518 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x38248): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38248): Mirror 2: 0x1da0ad80 (stale) CRC-32 checksum value for chunk [0, 0x38248): Mirror 1: 0x8f4f564b Mirror 3: 0x8f4f564b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38248: skipped 000519 1788359936.290134 read 0x0312ae thru 0x033178 (0x1ecb bytes) 000520 1788359936.300012 mapwrite 0x00b955 thru 0x00f637 (0x3ce3 bytes) 000521 1788359936.365668 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38248): Mirror 3: 0x8f4f564b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38248): Mirror 3: 0x8f4f564b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38248: skipped 000522 1788359936.526476 punch from 0x027fbb to 0x02ea38 (0x6a7e bytes) 000523 1788359936.529048 mapwrite 0x01a8a5 thru 0x027ae7 (0xd243 bytes) 000524 1788359936.559481 mapwrite 0x015381 thru 0x015927 (0x5a7 bytes) 000525 1788359936.567119 mapwrite 0x010cc0 thru 0x01c682 (0xb9c3 bytes) 000526 1788359936.587412 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x38248): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38248): Mirror 3: 0x8f4f564b (stale) CRC-32 checksum value for chunk [0, 0x38248): Mirror 1: 0x30565fde Mirror 4: 0x30565fde lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38248: skipped 000527 1788359936.938898 fallocate 0x03d7ef thru 0x03ffff (0x2811 bytes) 000528 1788359936.956261 trunc from 0x03083e to 0x03691c (0x60df bytes) 000529 1788359936.959733 trunc from 0x03bff0 to 0x03e743 (0x2754 bytes) 000531 1788359936.962675 mapwrite 0x0224bc thru 0x028d1a (0x685f bytes) 000532 1788359936.980363 mapread 0x0203dd thru 0x0293ef (0x9013 bytes) 000533 1788359936.981642 trunc from 0x040000 to 0x01246d (0x2db92 bytes) 000534 1788359937.008876 write 0x00d7e3 thru 0x0153b0 (0x7bce bytes) 000535 1788359937.014071 mapread 0x00a119 thru 0x00f63f (0x5527 bytes) 000536 1788359937.020954 mapwrite 0x01e236 thru 0x02487e (0x6649 bytes) 000537 1788359937.048488 mapread 0x021877 thru 0x02487e (0x3008 bytes) 000538 1788359937.049566 read 0x021107 thru 0x02487e (0x3778 bytes) 000539 1788359937.050106 write 0x02691f thru 0x02df08 (0x75ea bytes) 000541 1788359937.051531 fallocate 0x031e7b thru 0x03e51e (0xc6a4 bytes) 000542 1788359937.056851 mapread 0x005cd8 thru 0x00866a (0x2993 bytes) 000544 1788359937.062774 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2df09): Mirror 4: 0xb988189e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2df09): Mirror 4: 0xb988189e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2df09: skipped 000545 1788359937.235361 punch from 0x02224d to 0x02df08 (0xbcbc bytes) 000546 1788359937.239860 trunc from 0x02df09 to 0x0175fe (0x1690a bytes) 000547 1788359937.251440 write 0x0258e1 thru 0x02b0a0 (0x57c0 bytes) 000549 1788359937.253493 punch from 0x0164a2 to 0x01d35d (0x6ebc bytes) 000550 1788359937.257345 mapwrite 0x021e2c thru 0x02301b (0x11f0 bytes) 000551 1788359937.285607 fallocate 0x03387a thru 0x0347b3 (0xf3a bytes) 000553 1788359937.289312 trunc from 0x027da6 to 0x0347b3 (0xca0e bytes) 000554 1788359937.292409 trunc from 0x00e659 to 0x016f7e (0x8926 bytes) 000556 1788359937.296299 mapwrite 0x03e771 thru 0x03ffff (0x188f bytes) 000559 1788359937.316021 mapwrite 0x012d5d thru 0x01e5d7 (0xb87b bytes) 000560 1788359937.339375 punch from 0x03f96d to 0x03ffff (0x693 bytes) 000561 1788359937.343787 trunc from 0x040000 to 0x026da6 (0x19259 bytes) 000562 1788359937.352845 trunc from 0x026da7 to 0x0009f9 (0x263ad bytes) 000563 1788359937.363983 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3ee:0x0]: Invalid argument (22) 000564 1788359937.424693 fallocate 0x01ee8f thru 0x022360 (0x34d2 bytes) 000565 1788359937.427944 mapwrite 0x0106ae thru 0x013f71 (0x38c4 bytes) 000566 1788359937.445619 read 0x00e375 thru 0x017b7e (0x980a bytes) 000567 1788359937.446383 mapwrite 0x01992f thru 0x028c54 (0xf326 bytes) 000568 1788359937.479770 write 0x035382 thru 0x03b47b (0x60fa bytes) 000570 1788359937.482106 mapread 0x01786f thru 0x01a1e5 (0x2977 bytes) 000572 1788359937.483767 trunc from 0x01701e to 0x026807 (0xf7ea bytes) 000573 1788359937.488137 mapwrite 0x0020f9 thru 0x0089ba (0x68c2 bytes) 000575 1788359937.505258 fallocate 0x021081 thru 0x021134 (0xb4 bytes) 000577 1788359937.510102 read 0x0395fa thru 0x03b47b (0x1e82 bytes) 000578 1788359937.510762 write 0x027085 thru 0x02cdd1 (0x5d4d bytes) 000579 1788359937.526274 mapread 0x00145a thru 0x0037bb (0x2362 bytes) 000580 1788359937.527290 trunc from 0x03b47c to 0x0194a1 (0x21fda bytes) 000581 1788359937.539141 read 0x009f3d thru 0x00ef28 (0x4fec bytes) 000582 1788359937.539858 mapwrite 0x025de5 thru 0x028d9d (0x2fb9 bytes) 000584 1788359937.568065 read 0x020473 thru 0x02823f (0x7dcd bytes) 000585 1788359937.568772 write 0x035093 thru 0x03ffff (0xaf6d bytes) 000586 1788359937.570240 mapread 0x0077ed thru 0x0147b4 (0xcfc8 bytes) 000587 1788359937.571422 trunc from 0x040000 to 0x01f89a (0x20765 bytes) 000588 1788359937.581063 write 0x002cbc thru 0x00b4b3 (0x87f8 bytes) 000589 1788359937.582293 fallocate 0x02038b thru 0x02fe76 (0xfaec bytes) 000590 1788359937.585590 trunc from 0x02fe77 to 0x035623 (0x57ad bytes) 000591 1788359937.596506 MIRROR_EXTEND to 2 mirrors 000592 1788359937.957007 fallocate 0x01627b thru 0x01b146 (0x4ecc bytes) 000593 1788359937.981042 MIRROR_RESYNC 2 mirrors 000594 1788359938.357582 trunc from 0x035624 to 0x03e66d (0x904a bytes) 000596 1788359938.376534 trunc from 0x01c77d to 0x021bb3 (0x5437 bytes) 000597 1788359938.380699 mapwrite 0x004a53 thru 0x009f03 (0x54b1 bytes) 000598 1788359938.402726 read 0x03dd91 thru 0x03e66d (0x8dd bytes) 000600 1788359938.407113 mapwrite 0x031b7a thru 0x031bdb (0x62 bytes) 000601 1788359938.417725 fallocate 0x017081 thru 0x022c93 (0xbc13 bytes) 000602 1788359938.421801 trunc from 0x03e66e to 0x030369 (0xe304 bytes) 000603 1788359938.431938 mapread 0x000900 thru 0x00ecee (0xe3ef bytes) 000604 1788359938.435535 trunc from 0x03036a to 0x006672 (0x29cf7 bytes) 000605 1788359938.442986 read 0x002922 thru 0x006672 (0x3d51 bytes) 000606 1788359938.443761 read 0x0019b3 thru 0x006672 (0x4cc0 bytes) 000607 1788359938.444591 trunc from 0x006673 to 0x03e2b6 (0x37c44 bytes) 000608 1788359938.451711 write 0x015734 thru 0x01e024 (0x88f1 bytes) 000609 1788359938.457987 punch from 0x00c435 to 0x00e900 (0x24cc bytes) 000610 1788359938.461415 trunc from 0x024824 to 0x029fd1 (0x57ae bytes) 000611 1788359938.464884 trunc from 0x019e8c to 0x025a0b (0xbb80 bytes) 000612 1788359938.474190 read 0x026966 thru 0x032af7 (0xc192 bytes) 000613 1788359938.479253 mapread 0x01da3b thru 0x02486c (0x6e32 bytes) 000614 1788359938.480329 fallocate 0x00eec9 thru 0x016b84 (0x7cbc bytes) 000615 1788359938.484132 mapread 0x027f8c thru 0x037616 (0xf68b bytes) 000616 1788359938.485677 mapread 0x00deb7 thru 0x01d6b4 (0xf7fe bytes) 000617 1788359938.486987 mapread 0x014bb7 thru 0x024551 (0xf99b bytes) 000618 1788359938.488145 mapread 0x0285a4 thru 0x0378ca (0xf327 bytes) 000619 1788359938.489328 trunc from 0x03e2b7 to 0x03e5d9 (0x323 bytes) 000620 1788359938.500487 fallocate 0x03d94b thru 0x03ffff (0x26b5 bytes) 000621 1788359938.503573 fallocate 0x020e00 thru 0x02551d (0x471e bytes) 000622 1788359938.508915 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3ee:0x0]: Invalid argument (22) 000623 1788359938.570351 write 0x01e6f4 thru 0x024352 (0x5c5f bytes) 000625 1788359938.580901 punch from 0x036a02 to 0x0384bf (0x1abe bytes) 000626 1788359938.583828 mapread 0x017a05 thru 0x018778 (0xd74 bytes) 000628 1788359938.591494 trunc from 0x024d09 to 0x0254e8 (0x7e0 bytes) 000630 1788359938.601366 trunc from 0x040000 to 0x03c5ef (0x3a10 bytes) 000631 1788359938.610408 mapread 0x013a8f thru 0x0194a0 (0x5a12 bytes) 000633 1788359938.619343 MIRROR_EXTEND to 2 mirrors 000634 1788359938.927526 trunc from 0x03c5f0 to 0x034f5e (0x7691 bytes) 000635 1788359938.964134 trunc from 0x034f5f to 0x031d2c (0x3232 bytes) 000636 1788359938.973494 mapwrite 0x0224af thru 0x02c18a (0x9cdc bytes) 000637 1788359938.994405 trunc from 0x021424 to 0x02e994 (0xd571 bytes) 000638 1788359938.998516 read 0x00f2a6 thru 0x01dde6 (0xeb41 bytes) 000639 1788359939.003702 write 0x037e77 thru 0x03ffff (0x8189 bytes) 000640 1788359939.007598 MIRROR_RESYNC 2 mirrors 000641 1788359939.304500 punch from 0x021201 to 0x02e997 (0xd797 bytes) 000642 1788359939.321954 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3ba0f0f6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbc5569dd Mirror 3: 0xbc5569dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000644 1788359939.690576 read 0x03fbff thru 0x03ffff (0x401 bytes) 000645 1788359939.695330 mapread 0x0076b4 thru 0x009297 (0x1be4 bytes) 000646 1788359939.710554 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3ee:0x0]: Invalid argument (22) 000647 1788359939.831689 MIRROR_EXTEND to 3 mirrors 000648 1788359940.183346 trunc from 0x040000 to 0x03f1c8 (0xe37 bytes) 000649 1788359940.199297 punch from 0x038d13 to 0x03f1c8 (0x64b6 bytes) 000650 1788359940.202220 trunc from 0x03f1c9 to 0x035320 (0x9ea8 bytes) 000651 1788359940.211632 trunc from 0x035321 to 0x02c6c1 (0x8c5f bytes) 000652 1788359940.217841 mapwrite 0x005dac thru 0x014fc8 (0xf21d bytes) 000653 1788359940.242651 MIRROR_RESYNC 3 mirrors 000654 1788359940.537427 punch from 0x021c5b to 0x025f2b (0x42d1 bytes) 000655 1788359940.552033 mapwrite 0x026183 thru 0x030dac (0xac2a bytes) 000656 1788359940.594763 write 0x03e6b9 thru 0x03ffff (0x1947 bytes) 000657 1788359940.595984 trunc from 0x040000 to 0x02e8fa (0x11705 bytes) 000658 1788359940.606462 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2e8fb): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e8fb): Mirror 3: 0x5f536290 (stale) Mirror 4: 0x5f536290 (stale) CRC-32 checksum value for chunk [0, 0x2e8fb): Mirror 1: 0x937c9b74 Mirror 5: 0x937c9b74 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e8fb: skipped 000659 1788359940.909645 trunc from 0x02e8fb to 0x014921 (0x19fd9 bytes) 000660 1788359940.930169 mapwrite 0x01bd56 thru 0x01e9db (0x2c86 bytes) 000661 1788359940.954873 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1e9dc): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e9dc): Mirror 3: 0x97e544c3 (stale) Mirror 4: 0x97e544c3 (stale) Mirror 5: 0x97e544c3 (stale) CRC-32 checksum value for chunk [0, 0x1e9dc): Mirror 1: 0xeacddce5 Mirror 6: 0xeacddce5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e9dc: skipped 000662 1788359941.280162 trunc from 0x01e9dc to 0x03fa28 (0x2104d bytes) 000664 1788359941.304712 trunc from 0x000846 to 0x00238c (0x1b47 bytes) 000665 1788359941.308270 mapwrite 0x034fc6 thru 0x03749c (0x24d7 bytes) 000666 1788359941.329657 mapwrite 0x009110 thru 0x00dcb8 (0x4ba9 bytes) 000667 1788359941.344824 mapread 0x029c09 thru 0x039b81 (0xff79 bytes) 000668 1788359941.352579 trunc from 0x01fb22 to 0x02d477 (0xd956 bytes) 000669 1788359941.358384 trunc from 0x03fa29 to 0x00c1df (0x33849 bytes) 000670 1788359941.371434 write 0x022dd9 thru 0x031730 (0xe958 bytes) 000671 1788359941.373353 read 0x01b95b thru 0x023752 (0x7df8 bytes) 000672 1788359941.377545 punch from 0x01040c to 0x01f4c7 (0xf0bc bytes) 000673 1788359941.381038 mapwrite 0x006b1c thru 0x012c29 (0xc10e bytes) 000674 1788359941.401476 fallocate 0x02bff5 thru 0x02f0a8 (0x30b4 bytes) 000675 1788359941.407487 read 0x0253b3 thru 0x031730 (0xc37e bytes) 000676 1788359941.408844 fallocate 0x01a7c4 thru 0x02379e (0x8fdb bytes) 000677 1788359941.412917 trunc from 0x031731 to 0x03d6ce (0xbf9e bytes) 000678 1788359941.422346 write 0x03f4f0 thru 0x03ffff (0xb10 bytes) 000679 1788359941.423415 punch from 0x01dba3 to 0x02824c (0xa6aa bytes) 000680 1788359941.426323 fallocate 0x012115 thru 0x018cc0 (0x6bac bytes) 000681 1788359941.429733 trunc from 0x004560 to 0x0109e4 (0xc485 bytes) 000682 1788359941.433846 mapread 0x023029 thru 0x0301fa (0xd1d2 bytes) 000683 1788359941.444490 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xa861769c (stale) Mirror 5: 0xe92dd82d (stale) Mirror 6: 0x4b5325a8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xa861769c (stale) Mirror 5: 0xe92dd82d (stale) Mirror 6: 0x4b5325a8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000684 1788359941.650992 trunc from 0x040000 to 0x03f436 (0xbc9 bytes) 000685 1788359941.661635 fallocate 0x0007da thru 0x00c46d (0xbc94 bytes) 000686 1788359941.664593 punch from 0x02d57a to 0x0358e4 (0x836b bytes) 000687 1788359941.667619 mapwrite 0x001f61 thru 0x00dfb5 (0xc055 bytes) 000688 1788359941.699287 punch from 0x013307 to 0x0225a4 (0xf29e bytes) 000690 1788359941.702628 fallocate 0x038b65 thru 0x03ffff (0x749b bytes) 000691 1788359941.706027 trunc from 0x0300a1 to 0x037ced (0x7c4d bytes) 000692 1788359941.710008 punch from 0x00e045 to 0x0107af (0x276b bytes) 000693 1788359941.715862 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xa861769c (stale) Mirror 5: 0xe92dd82d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xa861769c (stale) Mirror 5: 0xe92dd82d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000694 1788359941.855939 mapwrite 0x03b8d9 thru 0x03ffff (0x4727 bytes) 000695 1788359941.882182 trunc from 0x040000 to 0x03f14d (0xeb2 bytes) 000696 1788359941.896939 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f14e): Mirror 4: 0x62dd3e3b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f14e): Mirror 4: 0x62dd3e3b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f14e: skipped 000697 1788359942.015227 mapwrite 0x009c38 thru 0x00ce90 (0x3259 bytes) 000698 1788359942.033805 trunc from 0x03f14e to 0x03860f (0x6b3e bytes) 000699 1788359942.042127 read 0x01d917 thru 0x02b7e5 (0xdecf bytes) 000700 1788359942.048818 fallocate 0x00e8b8 thru 0x01879a (0x9ee3 bytes) 000701 1788359942.051866 fallocate 0x01a91f thru 0x02776c (0xce4e bytes) 000702 1788359942.054795 punch from 0x025274 to 0x027601 (0x238e bytes) 000703 1788359942.057577 trunc from 0x01d3ae to 0x01f73b (0x238e bytes) 000704 1788359942.060047 trunc from 0x032e7f to 0x03860f (0x5791 bytes) 000705 1788359942.062577 trunc from 0x00fd96 to 0x019436 (0x96a1 bytes) 000706 1788359942.065289 fallocate 0x01534c thru 0x01c598 (0x724d bytes) 000707 1788359942.067962 trunc from 0x030130 to 0x03860f (0x84e0 bytes) 000708 1788359942.070953 mapread 0x01b2f8 thru 0x02651d (0xb226 bytes) 000709 1788359942.079397 mapwrite 0x007d25 thru 0x00b946 (0x3c22 bytes) 000710 1788359942.088403 mapread 0x0167c0 thru 0x01772a (0xf6b bytes) 000712 1788359942.092391 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3ee:0x0]: Invalid argument (22) 000713 1788359942.157503 trunc from 0x038610 to 0x0054d1 (0x3313e bytes) 000715 1788359942.172400 punch from 0x00352a to 0x0054d1 (0x1fa8 bytes) 000716 1788359942.177058 read 0x002e91 thru 0x0054d1 (0x2641 bytes) 000717 1788359942.180868 fallocate 0x027e43 thru 0x032bea (0xada8 bytes) 000718 1788359942.183866 read 0x02b2a2 thru 0x032bea (0x7949 bytes) 000719 1788359942.190078 mapwrite 0x03c729 thru 0x03d55c (0xe34 bytes) 000720 1788359942.218336 mapread 0x008321 thru 0x013d54 (0xba34 bytes) 000721 1788359942.219684 fallocate 0x00a1b6 thru 0x0174c5 (0xd310 bytes) 000722 1788359942.225660 trunc from 0x03d55d to 0x025995 (0x17bc7 bytes) 000723 1788359942.237176 mapwrite 0x02b69b thru 0x02eb07 (0x346d bytes) 000724 1788359942.272422 mapread 0x01dc8d thru 0x0220fa (0x446e bytes) 000725 1788359942.273609 punch from 0x016f9c to 0x018a32 (0x1a97 bytes) 000726 1788359942.276997 punch from 0x02b521 to 0x02eb07 (0x35e7 bytes) 000727 1788359942.280305 trunc from 0x01df25 to 0x02802d (0xa109 bytes) 000729 1788359942.284251 write 0x0295e0 thru 0x03272c (0x914d bytes) 000730 1788359942.288515 read 0x02fd17 thru 0x03272c (0x2a16 bytes) 000732 1788359942.289866 punch from 0x009185 to 0x015906 (0xc782 bytes) 000734 1788359942.297375 mapwrite 0x03187d thru 0x03abdc (0x9360 bytes) 000735 1788359942.326836 trunc from 0x01217c to 0x01dfa1 (0xbe26 bytes) 000736 1788359942.330764 fallocate 0x0388aa thru 0x03cf1f (0x4676 bytes) 000737 1788359942.336614 mapread 0x01e6f1 thru 0x01fceb (0x15fb bytes) 000738 1788359942.339378 mapread 0x00d2d5 thru 0x00f747 (0x2473 bytes) 000739 1788359942.341062 read 0x00e7c8 thru 0x0164e2 (0x7d1b bytes) 000740 1788359942.346267 trunc from 0x03cf20 to 0x03d3fd (0x4de bytes) 000741 1788359942.354176 fallocate 0x00071a thru 0x0058da (0x51c1 bytes) 000742 1788359942.357350 read 0x00b2d1 thru 0x012fa1 (0x7cd1 bytes) 000743 1788359942.357776 write 0x033156 thru 0x03b736 (0x85e1 bytes) 000745 1788359942.359998 punch from 0x005987 to 0x00f679 (0x9cf3 bytes) 000746 1788359942.363902 mapread 0x02c0b6 thru 0x03ba6b (0xf9b6 bytes) 000747 1788359942.365275 fallocate 0x01389d thru 0x0201c3 (0xc927 bytes) 000748 1788359942.369873 fallocate 0x00b833 thru 0x011196 (0x5964 bytes) 000751 1788359942.374847 trunc from 0x011512 to 0x01d85f (0xc34e bytes) 000752 1788359942.378123 trunc from 0x03d3fe to 0x03ca01 (0x9fc bytes) 000754 1788359942.390578 fallocate 0x00c712 thru 0x017b41 (0xb430 bytes) 000755 1788359942.394218 fallocate 0x0105ea thru 0x011891 (0x12a8 bytes) 000756 1788359942.397153 write 0x021f9b thru 0x03082d (0xe893 bytes) 000759 1788359942.402827 write 0x009279 thru 0x016f0c (0xdc94 bytes) 000760 1788359942.413983 trunc from 0x01fb04 to 0x0278ea (0x7de7 bytes) 000761 1788359942.424936 read 0x00ef4b thru 0x0130eb (0x41a1 bytes) 000762 1788359942.425736 read 0x038fb5 thru 0x03ca01 (0x3a4d bytes) 000763 1788359942.426436 mapwrite 0x00a646 thru 0x00bb71 (0x152c bytes) 000764 1788359942.436972 fallocate 0x011820 thru 0x016da2 (0x5583 bytes) 000765 1788359942.440169 punch from 0x00df47 to 0x00f2ad (0x1367 bytes) 000766 1788359942.443081 punch from 0x01e30d to 0x021b8a (0x387e bytes) 000767 1788359942.446793 fallocate 0x010d0a thru 0x0115da (0x8d1 bytes) 000768 1788359942.450086 punch from 0x01421b to 0x022e8f (0xec75 bytes) 000769 1788359942.454031 punch from 0x0086d8 to 0x016d53 (0xe67c bytes) 000770 1788359942.457392 fallocate 0x02d483 thru 0x030675 (0x31f3 bytes) 000771 1788359942.461115 read 0x018450 thru 0x0191df (0xd90 bytes) 000772 1788359942.467558 read 0x0241f3 thru 0x02b001 (0x6e0f bytes) 000773 1788359942.468181 trunc from 0x03ca02 to 0x00f4f0 (0x2d511 bytes) 000774 1788359942.480505 mapread 0x00a230 thru 0x00f4f0 (0x52c1 bytes) 000775 1788359942.481652 fallocate 0x010d84 thru 0x02015b (0xf3d8 bytes) 000776 1788359942.486996 write 0x03aff4 thru 0x03b440 (0x44d bytes) 000777 1788359942.487997 trunc from 0x01ab69 to 0x01d9cc (0x2e64 bytes) 000779 1788359942.491311 read 0x00d373 thru 0x015acc (0x875a bytes) 000780 1788359942.497729 read 0x02406e thru 0x032741 (0xe6d4 bytes) 000781 1788359942.498686 read 0x0188ce thru 0x0248d8 (0xc00b bytes) 000782 1788359942.499409 trunc from 0x03b441 to 0x01ef95 (0x1c4ab bytes) 000783 1788359942.509817 mapwrite 0x009ada thru 0x00acc3 (0x11ea bytes) 000784 1788359942.517270 read 0x00468e thru 0x00a3fa (0x5d6d bytes) 000786 1788359942.518532 trunc from 0x009b82 to 0x016e8c (0xd30b bytes) 000788 1788359942.522858 trunc from 0x01ef96 to 0x0185e7 (0x69ae bytes) 000789 1788359942.534916 fallocate 0x004ffc thru 0x0093b8 (0x43bd bytes) 000790 1788359942.537816 write 0x01b191 thru 0x01d26d (0x20dd bytes) 000791 1788359942.538814 punch from 0x019174 to 0x01d26d (0x40fa bytes) 000792 1788359942.544360 write 0x01d7d0 thru 0x02c153 (0xe984 bytes) 000793 1788359942.549807 trunc from 0x00d80b to 0x015075 (0x786b bytes) 000794 1788359942.553972 mapread 0x00f498 thru 0x01220e (0x2d77 bytes) 000795 1788359942.566306 trunc from 0x00f28d to 0x015299 (0x600d bytes) 000796 1788359942.569703 read 0x0096d3 thru 0x00b55d (0x1e8b bytes) 000797 1788359942.570412 write 0x007ec6 thru 0x014371 (0xc4ac bytes) 000798 1788359942.574675 trunc from 0x02c154 to 0x02926f (0x2ee4 bytes) 000799 1788359942.584829 MIRROR_EXTEND to 2 mirrors 000800 1788359942.876644 MIRROR_RESYNC 2 mirrors 000801 1788359943.037616 mapread 0x026ff9 thru 0x02926f (0x2277 bytes) 000803 1788359943.050943 write 0x0183b9 thru 0x0201b1 (0x7df9 bytes) 000804 1788359943.075955 mapread 0x017f8d thru 0x01fd6d (0x7de1 bytes) 000805 1788359943.084384 mapread 0x022541 thru 0x0242be (0x1d7e bytes) 000806 1788359943.090384 fallocate 0x02eed8 thru 0x03d181 (0xe2aa bytes) 000809 1788359943.098886 MIRROR_RESYNC 2 mirrors 000810 1788359943.406699 mapread 0x0225cf thru 0x02386d (0x129f bytes) 000811 1788359943.418716 trunc from 0x01e8ea to 0x0239af (0x50c6 bytes) 000812 1788359943.445375 mapread 0x00513e thru 0x012876 (0xd739 bytes) 000813 1788359943.459943 read 0x01b12d thru 0x0217ba (0x668e bytes) 000814 1788359943.464544 read 0x00381c thru 0x00f2d7 (0xbabc bytes) 000815 1788359943.465243 fallocate 0x00907b thru 0x013420 (0xa3a6 bytes) 000816 1788359943.468351 write 0x03456d thru 0x0391a8 (0x4c3c bytes) 000817 1788359943.469450 mapwrite 0x0028b3 thru 0x00a987 (0x80d5 bytes) 000818 1788359943.481920 trunc from 0x0391a9 to 0x006e3a (0x3236e bytes) 000820 1788359943.511513 punch from 0x00144d to 0x006e3a (0x59ee bytes) 000821 1788359943.516697 fallocate 0x033d09 thru 0x03ae73 (0x716b bytes) 000822 1788359943.523710 MIRROR_RESYNC 2 mirrors 000823 1788359943.752544 trunc from 0x03ae74 to 0x020309 (0x1ab6a bytes) 000824 1788359943.794853 punch from 0x00fbcf to 0x01e4af (0xe8e1 bytes) 000826 1788359943.798750 read 0x015116 thru 0x018d58 (0x3c43 bytes) 000827 1788359943.802930 mapread 0x01ec9e thru 0x020309 (0x166c bytes) 000828 1788359943.813523 mapwrite 0x029963 thru 0x031afa (0x8198 bytes) 000829 1788359943.844577 read 0x00863c thru 0x0101da (0x7b9f bytes) 000830 1788359943.850154 punch from 0x01d802 to 0x02722c (0x9a2b bytes) 000831 1788359943.853179 punch from 0x0017ac to 0x009f9e (0x87f3 bytes) 000832 1788359943.856648 write 0x020b98 thru 0x0268bf (0x5d28 bytes) 000833 1788359943.865996 read 0x0027f6 thru 0x00c313 (0x9b1e bytes) 000834 1788359943.871905 mapread 0x004a00 thru 0x00ed14 (0xa315 bytes) 000835 1788359943.873535 read 0x00971c thru 0x009e48 (0x72d bytes) 000836 1788359943.874428 mapread 0x028266 thru 0x031afa (0x9895 bytes) 000837 1788359943.875679 punch from 0x02c63b to 0x031afa (0x54c0 bytes) 000838 1788359943.880248 mapwrite 0x00b8e6 thru 0x01a86a (0xef85 bytes) 000840 1788359943.898976 mapwrite 0x033346 thru 0x036d5e (0x3a19 bytes) 000841 1788359943.924368 write 0x0318b3 thru 0x0390b9 (0x7807 bytes) 000842 1788359943.925372 mapread 0x00ae43 thru 0x011ba0 (0x6d5e bytes) 000844 1788359943.926475 mapwrite 0x03a512 thru 0x03ffff (0x5aee bytes) 000845 1788359943.948471 trunc from 0x0373e8 to 0x03ffff (0x8c18 bytes) 000846 1788359943.952144 read 0x02a0a2 thru 0x038246 (0xe1a5 bytes) 000847 1788359943.957088 mapwrite 0x018206 thru 0x025c2f (0xda2a bytes) 000848 1788359943.967922 mapread 0x023633 thru 0x0287aa (0x5178 bytes) 000849 1788359943.968917 fallocate 0x003cdf thru 0x004445 (0x767 bytes) 000850 1788359943.971673 write 0x011e9d thru 0x01c1c5 (0xa329 bytes) 000851 1788359943.972779 write 0x02677e thru 0x030db8 (0xa63b bytes) 000852 1788359943.973906 write 0x03f41b thru 0x03ffff (0xbe5 bytes) 000853 1788359943.974680 trunc from 0x040000 to 0x03dc77 (0x2388 bytes) 000855 1788359943.981135 read 0x02d2c8 thru 0x030ec9 (0x3c02 bytes) 000856 1788359943.981742 punch from 0x01f13c to 0x023f69 (0x4e2e bytes) 000857 1788359943.984664 punch from 0x000854 to 0x007c1a (0x73c7 bytes) 000858 1788359943.987667 trunc from 0x022baf to 0x02f359 (0xc7ab bytes) 000859 1788359943.994338 mapread 0x00f7a7 thru 0x01aa9a (0xb2f4 bytes) 000860 1788359943.998401 MIRROR_RESYNC 2 mirrors 000861 1788359944.268068 trunc from 0x03dc78 to 0x01ea0a (0x1f26d bytes) 000862 1788359944.286526 read 0x016aac thru 0x01ea0a (0x7f5f bytes) 000863 1788359944.292296 punch from 0x0033ec to 0x0093a0 (0x5fb5 bytes) 000865 1788359944.295406 fallocate 0x0201fe thru 0x0265fd (0x6400 bytes) 000866 1788359944.298135 fallocate 0x039ded thru 0x03ff9f (0x61b3 bytes) 000867 1788359944.301069 punch from 0x001000 to 0x001e92 (0xe93 bytes) 000868 1788359944.303552 mapwrite 0x02bf17 thru 0x030e51 (0x4f3b bytes) 000869 1788359944.333405 trunc from 0x030e52 to 0x01267f (0x1e7d2 bytes) 000870 1788359944.341616 punch from 0x000928 to 0x0093ed (0x8ac6 bytes) 000871 1788359944.344865 fallocate 0x027f3c thru 0x031e0a (0x9ecf bytes) 000872 1788359944.347815 trunc from 0x00264b to 0x00e5e4 (0xbf9a bytes) 000873 1788359944.351891 read 0x001358 thru 0x00a08e (0x8d37 bytes) 000874 1788359944.357811 mapwrite 0x029277 thru 0x0343d4 (0xb15e bytes) 000875 1788359944.382274 mapread 0x01189d thru 0x01738b (0x5aef bytes) 000876 1788359944.383355 read 0x01f83d thru 0x02bfa2 (0xc766 bytes) 000877 1788359944.383910 punch from 0x00455f to 0x007cb7 (0x3759 bytes) 000878 1788359944.386835 write 0x021412 thru 0x021c37 (0x826 bytes) 000879 1788359944.387781 mapread 0x01410e thru 0x023fd6 (0xfec9 bytes) 000881 1788359944.389627 trunc from 0x0343d5 to 0x01c6b7 (0x17d1d bytes) 000883 1788359944.398400 fallocate 0x00c693 thru 0x00c9f9 (0x367 bytes) 000884 1788359944.403969 mapwrite 0x03d1f9 thru 0x03ffff (0x2e07 bytes) 000885 1788359944.434043 trunc from 0x007d80 to 0x007eb2 (0x133 bytes) 000886 1788359944.436607 read 0x0293ca thru 0x02fe04 (0x6a3b bytes) 000887 1788359944.437764 read 0x03b029 thru 0x03ffff (0x4fd7 bytes) 000888 1788359944.438390 mapwrite 0x03d2fb thru 0x03ffff (0x2d05 bytes) 000889 1788359944.446581 read 0x02b42c thru 0x02b4c9 (0x9e bytes) 000891 1788359944.447756 mapread 0x020d9b thru 0x021ab3 (0xd19 bytes) 000892 1788359944.448712 write 0x02d7b8 thru 0x034196 (0x69df bytes) 000893 1788359944.450095 trunc from 0x03ba04 to 0x03f416 (0x3a13 bytes) 000894 1788359944.454436 trunc from 0x03233f to 0x0368a1 (0x4563 bytes) 000895 1788359944.462125 mapread 0x0200b2 thru 0x02416c (0x40bb bytes) 000896 1788359944.465723 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc71db951 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd536b068 Mirror 3: 0xd536b068 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000897 1788359944.789837 trunc from 0x03b4ff to 0x03ffff (0x4b01 bytes) 000898 1788359944.805826 trunc from 0x007f9d to 0x016424 (0xe488 bytes) 000899 1788359944.808886 read 0x02e550 thru 0x03c92a (0xe3db bytes) 000900 1788359944.813949 write 0x01878c thru 0x018b87 (0x3fc bytes) 000901 1788359944.819647 mapwrite 0x024eca thru 0x02f571 (0xa6a8 bytes) 000902 1788359944.840039 punch from 0x035514 to 0x037192 (0x1c7f bytes) 000903 1788359944.843772 read 0x0383c1 thru 0x039f7a (0x1bba bytes) 000904 1788359944.844396 trunc from 0x03a480 to 0x03f8e3 (0x5464 bytes) 000905 1788359944.849842 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd536b068 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd536b068 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000906 1788359944.978511 fallocate 0x0226be thru 0x02b0b2 (0x89f5 bytes) 000907 1788359944.981822 trunc from 0x040000 to 0x0305d8 (0xfa27 bytes) 000908 1788359945.004690 punch from 0x02490d to 0x024be8 (0x2dc bytes) 000909 1788359945.007787 mapwrite 0x03a5c2 thru 0x03e00c (0x3a4b bytes) 000910 1788359945.039565 trunc from 0x03e00d to 0x012805 (0x2b807 bytes) 000911 1788359945.049036 mapwrite 0x012eef thru 0x014c94 (0x1da6 bytes) 000913 1788359945.075261 mapread 0x0086f5 thru 0x00ecab (0x65b7 bytes) 000914 1788359945.080970 mapread 0x003917 thru 0x004086 (0x770 bytes) 000915 1788359945.081916 mapwrite 0x02aea9 thru 0x02c1ea (0x1342 bytes) 000916 1788359945.104868 fallocate 0x02d6f7 thru 0x035d00 (0x860a bytes) 000917 1788359945.108356 read 0x02825b thru 0x02c1ea (0x3f90 bytes) 000918 1788359945.108964 punch from 0x02558d to 0x02c1ea (0x6c5e bytes) 000919 1788359945.115982 MIRROR_RESYNC 2 mirrors 000920 1788359945.326695 read 0x009972 thru 0x0133df (0x9a6e bytes) 000921 1788359945.331466 read 0x00c3b6 thru 0x010937 (0x4582 bytes) 000922 1788359945.331903 mapwrite 0x010ad9 thru 0x0172f5 (0x681d bytes) 000923 1788359945.369093 write 0x03fda0 thru 0x03ffff (0x260 bytes) 000925 1788359945.372477 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3ee:0x0]: Invalid argument (22) 000926 1788359945.416669 mapread 0x03a24a thru 0x03ffff (0x5db6 bytes) 000927 1788359945.424330 punch from 0x01335e to 0x015bd2 (0x2875 bytes) 000928 1788359945.426619 trunc from 0x040000 to 0x03cd52 (0x32ad bytes) 000929 1788359945.442397 trunc from 0x00b3b9 to 0x017d69 (0xc9b1 bytes) 000930 1788359945.445141 fallocate 0x02e58a thru 0x03a7b7 (0xc22e bytes) 000931 1788359945.447539 trunc from 0x03cd53 to 0x025c2c (0x17126 bytes) 000933 1788359945.454239 mapwrite 0x001454 thru 0x00ac1a (0x97c7 bytes) 000934 1788359945.468614 trunc from 0x025c2d to 0x006c5a (0x1efd2 bytes) 000937 1788359945.477748 read 0x006a7d thru 0x006c5a (0x1de bytes) 000938 1788359945.478441 write 0x03d2e7 thru 0x03ffff (0x2d19 bytes) 000939 1788359945.479621 write 0x00ada1 thru 0x01540d (0xa66d bytes) 000940 1788359945.484370 punch from 0x024694 to 0x0328aa (0xe217 bytes) 000941 1788359945.487741 read 0x036941 thru 0x039f56 (0x3616 bytes) 000942 1788359945.492450 fallocate 0x004225 thru 0x012f10 (0xecec bytes) 000943 1788359945.495597 read 0x038e70 thru 0x03ffff (0x7190 bytes) 000944 1788359945.496054 trunc from 0x01aebd to 0x01c237 (0x137b bytes) 000945 1788359945.498986 mapwrite 0x01b82b thru 0x0280f2 (0xc8c8 bytes) 000947 1788359945.516032 write 0x009298 thru 0x014650 (0xb3b9 bytes) 000948 1788359945.517383 trunc from 0x040000 to 0x03e1e1 (0x1e1e bytes) 000949 1788359945.524913 punch from 0x014cde to 0x01b240 (0x6563 bytes) 000950 1788359945.533115 punch from 0x038b8a to 0x03e1e1 (0x5658 bytes) 000951 1788359945.536456 punch from 0x002eb4 to 0x00f52d (0xc67a bytes) 000952 1788359945.539864 mapread 0x03d72f thru 0x03e1e1 (0xab3 bytes) 000953 1788359945.545582 mapwrite 0x01f57b thru 0x021aac (0x2532 bytes) 000954 1788359945.554418 mapread 0x0165ab thru 0x01926c (0x2cc2 bytes) 000955 1788359945.563338 mapwrite 0x0021d6 thru 0x00d762 (0xb58d bytes) 000956 1788359945.602313 read 0x01ed7b thru 0x025d76 (0x6ffc bytes) 000957 1788359945.602878 write 0x00a52d thru 0x01a4ab (0xff7f bytes) 000958 1788359945.606292 read 0x038547 thru 0x03e1e1 (0x5c9b bytes) 000959 1788359945.609863 fallocate 0x024857 thru 0x02fcc8 (0xb472 bytes) 000960 1788359945.612817 fallocate 0x010960 thru 0x01a331 (0x99d2 bytes) 000963 1788359945.617229 write 0x031db0 thru 0x03db24 (0xbd75 bytes) 000964 1788359945.619249 mapwrite 0x036944 thru 0x03ffff (0x96bc bytes) 000965 1788359945.638465 fallocate 0x02f5eb thru 0x034822 (0x5238 bytes) 000966 1788359945.641475 fallocate 0x038088 thru 0x03ffff (0x7f78 bytes) 000967 1788359945.643951 trunc from 0x032a9f to 0x0346eb (0x1c4d bytes) 000968 1788359945.646688 read 0x02f34e thru 0x030b3f (0x17f2 bytes) 000969 1788359945.647364 read 0x0203ca thru 0x02e8d6 (0xe50d bytes) 000971 1788359945.648958 mapread 0x02a15b thru 0x02d1e3 (0x3089 bytes) 000972 1788359945.650025 write 0x0009f3 thru 0x00ea11 (0xe01f bytes) 000974 1788359945.652284 write 0x0350db thru 0x0354ae (0x3d4 bytes) 000975 1788359945.653228 punch from 0x03b32b to 0x03dacb (0x27a1 bytes) 000977 1788359945.657513 fallocate 0x00609f thru 0x00f302 (0x9264 bytes) 000978 1788359945.661482 write 0x02e0ec thru 0x038548 (0xa45d bytes) 000979 1788359945.662882 read 0x0167d9 thru 0x019d21 (0x3549 bytes) 000981 1788359945.663887 write 0x00b7a2 thru 0x00d3e3 (0x1c42 bytes) 000982 1788359945.664798 trunc from 0x03c6db to 0x03ffff (0x3925 bytes) 000983 1788359945.667541 read 0x029876 thru 0x033812 (0x9f9d bytes) 000984 1788359945.668212 punch from 0x00ef41 to 0x01b0d7 (0xc197 bytes) 000985 1788359945.677348 fallocate 0x0064e4 thru 0x0098cc (0x33e9 bytes) 000986 1788359945.680620 mapwrite 0x0283a4 thru 0x03410a (0xbd67 bytes) 000987 1788359945.691789 read 0x011087 thru 0x01e8a4 (0xd81e bytes) 000992 1788359945.699739 write 0x01d2ac thru 0x02888b (0xb5e0 bytes) 000993 1788359945.701047 read 0x0176c1 thru 0x01ed19 (0x7659 bytes) 000994 1788359945.701755 read 0x0326c4 thru 0x03efcd (0xc90a bytes) 000995 1788359945.702528 read 0x034151 thru 0x03a2a9 (0x6159 bytes) 000997 1788359945.703917 write 0x001c17 thru 0x00488e (0x2c78 bytes) 000998 1788359945.705070 write 0x00104a thru 0x00f409 (0xe3c0 bytes) 001000 1788359945.707001 mapwrite 0x03a450 thru 0x03ffff (0x5bb0 bytes) All operations completed A-OK! PASS 70a (25s) == sanity-flr test 71: check mirror extend parent fid ==== 10:39:07 (1788359947) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x422:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (4s) == sanity-flr test 72: check append on FLR file ========== 10:39:11 (1788359951) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0989335 s, 21.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551201 s, 19.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (4s) == sanity-flr test 200a: stress test ===================== 10:39:15 (1788359955) Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre2 Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5262048 .. Extending file size to 4681216 .. failed Extending file size to 6767808 .. Extending file size to 6883840 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7233504 .. Extending file size to 8421024 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8501472 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8551136 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8538112 .. Extending file size to 8894144 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8922656 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9120544 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9046816 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9147360 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62730 62731 62732 62733 62736 failed 192.168.204.127@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg427-client.virtnet /mnt/lustre2 (opts:) 192.168.204.127@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg427-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (66s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 10:40:22 (1788360022) Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre2 Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8879136 .. Extending file size to 3980032 .. Extending file size to 8470752 .. Extending file size to 9032928 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9267328 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 65366 65367 65370 65371 65374 done 192.168.204.127@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg427-client.virtnet /mnt/lustre2 (opts:) 192.168.204.127@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg427-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (69s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 10:41:31 (1788360091) Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2416 1268888 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2460 1268844 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18988 3588032 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24904 3582116 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43892 7170148 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2416 1268888 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2460 1268844 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18988 3588032 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24904 3582116 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43892 7170148 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0470801 s, 44.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0672286 s, 31.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x3ebf0c3 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.127@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg427-client.virtnet /mnt/lustre2 (opts:) PASS 200c (7s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 10:41:38 (1788360098) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0473832 s, 44.3 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 10:41:41 (1788360101) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x61:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 10:41:44 (1788360104) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0351011 s, 59.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x594:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x593:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x594:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 10:41:47 (1788360107) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0317513 s, 66.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 10:41:50 (1788360110) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.029484 s, 71.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 10:42:15 (1788360135) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg427-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235493 s, 44.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x595:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x596:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x595:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x595:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x596:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x595:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x596:0x0] } PASS 204d (37s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 10:42:52 (1788360172) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg427-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0481141 s, 43.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (36s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 10:43:28 (1788360208) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg427-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg427-server: Pool lustre.test_204f created Adding targets to pool oleg427-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg427-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.041714 s, 50.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg427-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg427-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg427-server: Pool lustre.test_204f destroyed PASS 204f (50s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 10:44:19 (1788360259) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 10:44:21 (1788360261) PASS 205b (4s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 10:44:25 (1788360265) oleg427-server: Pool lustre.test_206 created Adding targets to pool oleg427-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg427-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg427-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg427-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg427-server: Pool lustre.test_206 destroyed PASS 206 (16s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 10:44:41 (1788360281) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036562 s, 28.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162608 s, 64.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x3e94f557 Mirror 3: 0x3e94f557 PASS 207 (5s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 10:44:46 (1788360286) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 10:44:48 (1788360288) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 10:44:49 (1788360289) Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.017689 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x24f0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1520000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1520000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1520000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1810000 Read this (4 bytes): BBBB 192.168.204.127@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg427-client.virtnet /mnt/lustre2 (opts:) PASS 209a (19s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 10:45:08 (1788360308) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00449739 s, 1.8 MB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 10:45:11 (1788360311) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189852 s, 55.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x82 lmm_fid: [0x200000405:0x82:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x82 lmm_fid: [0x200000405:0x82:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11c:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611266 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-02 10:45:12.000000000 -0400 Modify: 2026-09-02 10:45:12.000000000 -0400 Change: 2026-09-02 10:45:12.000000000 -0400 Birth: 2026-09-02 10:45:13.000000000 -0400 PASS 210a (4s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 10:45:15 (1788360315) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165328 s, 63.4 MB/s oleg427-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg427-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg427-server: O/280000401/d28/13c: File not found by ext2_lookup PASS 210b (16s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 10:45:31 (1788360331) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.105418 s, 389 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0128865 s, 318 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 2 10:45 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x87:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 2 10:45 /mnt/lustre/f211.sanity-flr PASS 211 (4s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 10:45:35 (1788360335) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (3s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 10:45:38 (1788360338) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (4s) == sanity-flr test complete, duration 1033 sec =========== 10:45:42 (1788360342) === sanity-flr: start cleanup 10:45:43 (1788360343) === === sanity-flr: finish cleanup 10:45:47 (1788360347) ===