-----============= acceptance-small: sanity-flr ============----- Wed May 21 02:35:24 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg421-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 02:35:29 (1747809329) === oleg421-client.virtnet: executing check_config_client /mnt/lustre oleg421-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg421-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800adc99000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800adc99000.idle_timeout=debug disable quota as required oleg421-server: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 02:35:36 (1747809336) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7249] == sanity-flr test 0a: lfs mirror create with -N option == 02:35:36 (1747809336) PASS 0a (4s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 02:35:40 (1747809340) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (0s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 02:35:40 (1747809340) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 02:35:41 (1747809341) PASS 0d (4s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 02:35:45 (1747809345) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 02:35:46 (1747809346) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 02:35:47 (1747809347) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800adc99000.stats=clear osc.lustre-OST0001-osc-ffff8800adc99000.stats=clear PASS 0g (1s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 02:35:48 (1747809348) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 02:35:51 (1747809351) PASS 0j (1s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 02:35:52 (1747809352) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 02:35:53 (1747809353) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 02:35:55 (1747809355) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 02:35:57 (1747809357) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (0s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 02:35:57 (1747809357) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 02:35:59 (1747809359) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (1s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 02:36:01 (1747809361) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 02:36:03 (1747809363) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0669835 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0349117 s, 30.0 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 02:36:05 (1747809365) 18+0 records in 18+0 records out 18874368 bytes (19 MB) copied, 0.134875 s, 140 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 02:36:07 (1747809367) Stopping /mnt/lustre-ost1 (opts:) on oleg421-server oleg421-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800adc99000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800adc99000.ost_server_uuid in DISCONN state after 2 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg421-server: oleg421-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg421-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg421-server oleg421-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800adc99000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800adc99000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg421-server: oleg421-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg421-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (20s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 02:36:27 (1747809387) 11+0 records in 11+0 records out 11534336 bytes (12 MB) copied, 0.168674 s, 68.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg421-server oleg421-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800adc99000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800adc99000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg421-server: oleg421-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg421-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (10s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 02:36:37 (1747809397) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg421-server oleg421-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800adc99000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800adc99000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg421-server: oleg421-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg421-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg421-server oleg421-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800adc99000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800adc99000.ost_server_uuid in DISCONN state after 4 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg421-server: oleg421-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg421-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (19s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 02:36:56 (1747809416) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.795543 s, 65.9 MB/s -rw-r--r-- 1 root root 52428800 May 21 02:36 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.202s user 0m0.000s sys 0m0.113s Stopping /mnt/lustre-ost1 (opts:) on oleg421-server oleg421-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800adc99000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800adc99000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.202s user 0m0.000s sys 0m0.106s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg421-server: oleg421-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg421-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg421-server oleg421-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800adc99000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800adc99000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.207s user 0m0.001s sys 0m0.100s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg421-server: oleg421-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg421-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg421-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800adc99000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800adc99000.ost_server_uuid in FULL state after 0 sec PASS 33b (17s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 02:37:13 (1747809433) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (0s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 02:37:13 (1747809433) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 02:37:14 (1747809434) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 02:37:15 (1747809435) PASS 35 (1s) == sanity-flr test 36a: write to mirrored files ========== 02:37:16 (1747809436) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.106307 s, 118 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 02:37:19 (1747809439) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00650904 s, 161 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00729462 s, 144 MB/s PASS 36b (1s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 02:37:21 (1747809441) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 02:37:23 (1747809443) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000560898 s, 1.8 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000502769 s, 2.0 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000677713 s, 1.5 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000496203 s, 2.1 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 02:37:25 (1747809445) create test files with size 15730729 .. 7eb5b14679335374d9af4bf8247fbfdc - 6cd076d60653c1f7d87670732ef3ed13 - e5653ab3d56691bebbe35cea28ca05ed - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg421-client.virtnet /mnt/lustre (opts:) Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre PASS 37 (4s) == sanity-flr test 38: resync ============================ 02:37:30 (1747809450) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (2s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 02:37:32 (1747809452) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 02:37:34 (1747809454) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0187525 s, 55.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00859113 s, 122 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (1s) == sanity-flr test 41: lfs mirror resync check =========== 02:37:35 (1747809455) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0240355 s, 87.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0832043 s, 50.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0368088 s, 57.0 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 02:37:38 (1747809458) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 02:37:39 (1747809459) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (0s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 02:37:39 (1747809459) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00790318 s, 133 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 02:37:41 (1747809461) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 02:37:42 (1747809462) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4e:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (1s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 02:37:44 (1747809464) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0576489 s, 182 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg421-client.virtnet /mnt/lustre (opts:) Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (24s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 02:38:08 (1747809488) llite.lustre-ffff8800b600f800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0557097 s, 188 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg421-client.virtnet /mnt/lustre (opts:) Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff88012a628800.xattr_cache=1 PASS 44e (2s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 02:38:10 (1747809490) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (7s) == sanity-flr test 46: Verify setstripe --copy option ==== 02:38:17 (1747809497) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (6s) == sanity-flr test 47: Verify mirror obj alloc =========== 02:38:23 (1747809503) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (0s) == sanity-flr test 48: Verify snapshot mirror ============ 02:38:24 (1747809504) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0621137 s, 50.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0423423 s, 74.3 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 957ba136996e4d2314434c473749ad93 - mirror 1 checksum: 468daab0cf909fb4cc76d63559389df8 - mirror 2 checksum: 957ba136996e4d2314434c473749ad93 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed May 21 02:38:24 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed May 21 02:38:24 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 957ba136996e4d2314434c473749ad93 - mirror 1 checksum: 468daab0cf909fb4cc76d63559389df8 - mirror 2 checksum: 468daab0cf909fb4cc76d63559389df8 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed May 21 02:38:24 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed May 21 02:38:24 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } PASS 48 (3s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 02:38:26 (1747809506) Failing mds1 on oleg421-server Stopping /mnt/lustre-mds1 (opts:) on oleg421-server 02:38:27 (1747809507) shut down facet: mds1 facet_host: oleg421-server facet_failover_host: oleg421-server Failover mds1 to oleg421-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg421-server: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-MDT0000 02:38:40 (1747809520) targets are mounted 02:38:40 (1747809520) facet_failover done oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00972981 s, 108 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 50A (19s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 02:38:45 (1747809525) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (0s) == sanity-flr test 50b: mirror rsync handles sparseness == 02:38:46 (1747809526) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 02:38:46 (1747809526) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00192431 s, 8.5 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 02:38:48 (1747809528) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 02:38:49 (1747809529) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (0s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 02:38:49 (1747809529) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (4s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 02:38:53 (1747809533) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 02:39:11 (1747809551) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 02:39:28 (1747809568) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (16s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 02:39:44 (1747809584) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 02:39:46 (1747809586) mirror create pid 22773 mirror split pid 22774 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7665: 22774 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4948: 22773 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) PASS 70 (61s) == sanity-flr test 70a: flr mode fsx test ================ 02:40:47 (1747809647) Seed set to 9648 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000003 1747809648.389668 write 0x038013 thru 0x03bc42 (0x3c30 bytes) 000004 1747809648.390352 read 0x00bf94 thru 0x00c550 (0x5bd bytes) 000005 1747809648.393059 read 0x010bcb thru 0x01a53c (0x9972 bytes) 000006 1747809648.397118 read 0x0262e5 thru 0x02e619 (0x8335 bytes) 000008 1747809648.398420 mapread 0x01b5a2 thru 0x01dfd2 (0x2a31 bytes) 000009 1747809648.399082 read 0x01abd4 thru 0x026450 (0xb87d bytes) 000010 1747809648.399457 write 0x02d5cb thru 0x02f585 (0x1fbb bytes) 000011 1747809648.403116 read 0x00ee10 thru 0x01d224 (0xe415 bytes) 000013 1747809648.403708 write 0x012ba2 thru 0x01d675 (0xaad4 bytes) 000018 1747809648.405549 mapwrite 0x01dbd8 thru 0x02cbc8 (0xeff1 bytes) 000019 1747809648.434644 mapwrite 0x02c940 thru 0x03ae08 (0xe4c9 bytes) 000021 1747809648.450710 MIRROR_EXTEND to 2 mirrors 000023 1747809648.551804 mapread 0x033049 thru 0x03ae69 (0x7e21 bytes) 000027 1747809648.561297 read 0x00852a thru 0x00fc31 (0x7708 bytes) 000028 1747809648.564432 read 0x0298cb thru 0x037418 (0xdb4e bytes) 000030 1747809648.569942 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x91f:0x0]: Invalid argument (22) truncating to largest ever: 0xcc5d 000032 1747809648.601090 trunc from 0x03bc43 to 0x00cc5c (0x2efe6 bytes) 000033 1747809648.609456 trunc from 0x00cc5d to 0x000c13 (0xc049 bytes) 000034 1747809648.612918 write 0x006a2b thru 0x00ce74 (0x644a bytes) 000035 1747809648.613538 write 0x03b9d9 thru 0x03ee5c (0x3484 bytes) truncating to largest ever: 0x3ad86 000036 1747809648.614157 trunc from 0x03ee5d to 0x03ad85 (0x40d7 bytes) 000037 1747809648.618174 write 0x03d805 thru 0x03ffff (0x27fb bytes) 000038 1747809648.618787 trunc from 0x040000 to 0x036c17 (0x93e8 bytes) truncating to largest ever: 0x3bf43 000040 1747809648.622452 trunc from 0x036c18 to 0x03bf42 (0x532b bytes) 000043 1747809648.626339 mapwrite 0x02422c thru 0x031eb3 (0xdc88 bytes) 000044 1747809648.647210 read 0x00a4f7 thru 0x0129b9 (0x84c3 bytes) 000045 1747809648.649527 trunc from 0x03bf43 to 0x028d46 (0x131fc bytes) 000047 1747809648.653534 mapwrite 0x01c48b thru 0x024076 (0x7bec bytes) 000048 1747809648.670556 MIRROR_EXTEND to 2 mirrors 000049 1747809648.764520 trunc from 0x028d47 to 0x00d4d3 (0x1b873 bytes) 000050 1747809648.784217 trunc from 0x00d4d4 to 0x009cb2 (0x3821 bytes) 000052 1747809648.788437 trunc from 0x009cb3 to 0x02e10c (0x2445a bytes) 000053 1747809648.792343 read 0x010beb thru 0x010f2a (0x340 bytes) 000054 1747809648.794008 mapwrite 0x00f0f7 thru 0x015903 (0x680d bytes) 000057 1747809648.812647 mapread 0x011738 thru 0x017ac6 (0x638f bytes) 000058 1747809648.813068 mapwrite 0x017bb6 thru 0x0229c2 (0xae0d bytes) 000059 1747809648.827633 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x91f:0x0]: Invalid argument (22) 000062 1747809648.859568 write 0x03ea75 thru 0x03ffff (0x158b bytes) 000064 1747809648.860761 mapread 0x007d53 thru 0x00cedb (0x5189 bytes) 000065 1747809648.866738 MIRROR_EXTEND to 2 mirrors 000066 1747809648.974892 mapread 0x00d8cc thru 0x018f4f (0xb684 bytes) 000072 1747809648.988201 read 0x00c677 thru 0x01219d (0x5b27 bytes) truncating to largest ever: 0x3c04c 000074 1747809648.990288 trunc from 0x040000 to 0x03c04b (0x3fb4 bytes) 000075 1747809649.010029 mapwrite 0x00ebf3 thru 0x00ed55 (0x163 bytes) 000076 1747809649.023193 write 0x03fc0c thru 0x03ffff (0x3f4 bytes) 000077 1747809649.023732 write 0x01ec5c thru 0x01f07a (0x41f bytes) 000078 1747809649.027472 MIRROR_RESYNC 2 mirrors 000079 1747809649.122688 read 0x011ef8 thru 0x01dc8d (0xbd96 bytes) 000080 1747809649.124772 mapwrite 0x03674b thru 0x03b554 (0x4e0a bytes) 000081 1747809649.167925 read 0x0071ee thru 0x00b3eb (0x41fe bytes) 000082 1747809649.172738 MIRROR_RESYNC 2 mirrors 000085 1747809649.260537 write 0x03f0a8 thru 0x03ffff (0xf58 bytes) 000089 1747809649.277655 read 0x003efa thru 0x009ba8 (0x5caf bytes) 000093 1747809649.280720 mapread 0x02aaf8 thru 0x02d3ee (0x28f7 bytes) 000094 1747809649.285448 trunc from 0x040000 to 0x025e08 (0x1a1f7 bytes) 000096 1747809649.294178 mapwrite 0x0096d5 thru 0x013e5f (0xa78b bytes) 000098 1747809649.314668 read 0x021446 thru 0x025e08 (0x49c3 bytes) 000099 1747809649.317274 write 0x01067f thru 0x01c7cc (0xc14e bytes) 000100 1747809649.317991 trunc from 0x025e09 to 0x000d9e (0x2506a bytes) 000102 1747809649.323716 MIRROR_EXTEND to 3 mirrors 000104 1747809649.407961 trunc from 0x000d9f to 0x018adf (0x17d41 bytes) 000105 1747809649.426019 read 0x009f13 thru 0x018adf (0xebcd bytes) 000106 1747809649.428662 trunc from 0x018ae0 to 0x02ca9f (0x13fc0 bytes) 000107 1747809649.433964 mapwrite 0x00bd6f thru 0x01a8f2 (0xeb84 bytes) 000108 1747809649.451402 mapread 0x014fe8 thru 0x0184f9 (0x3512 bytes) 000109 1747809649.453596 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x91f:0x0]: Invalid argument (22) 000110 1747809649.501773 mapread 0x01eb97 thru 0x022a42 (0x3eac bytes) 000111 1747809649.506984 write 0x00ffa3 thru 0x01bad5 (0xbb33 bytes) 000116 1747809649.512464 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x91f:0x0]: Invalid argument (22) 000117 1747809649.552004 write 0x00571c thru 0x0074b1 (0x1d96 bytes) 000118 1747809649.556051 mapread 0x00a17a thru 0x0123eb (0x8272 bytes) 000120 1747809649.560649 mapwrite 0x02ea78 thru 0x0367ca (0x7d53 bytes) 000121 1747809649.591077 mapwrite 0x001bc8 thru 0x00b7c1 (0x9bfa bytes) 000126 1747809649.608266 read 0x02b837 thru 0x03449b (0x8c65 bytes) 000127 1747809649.608752 mapread 0x00865b thru 0x009c7a (0x1620 bytes) 000128 1747809649.609293 mapwrite 0x02658d thru 0x0348cc (0xe340 bytes) 000129 1747809649.626847 read 0x027bda thru 0x02d3f5 (0x581c bytes) 000130 1747809649.627180 mapwrite 0x03e694 thru 0x03ffff (0x196c bytes) 000134 1747809649.645944 mapread 0x034f66 thru 0x03ffff (0xb09a bytes) 000138 1747809649.647484 mapread 0x03ae45 thru 0x03ffff (0x51bb bytes) 000140 1747809649.649440 MIRROR_EXTEND to 2 mirrors 000142 1747809649.749300 mapread 0x011682 thru 0x01e863 (0xd1e2 bytes) 000143 1747809649.758777 MIRROR_EXTEND to 3 mirrors 000146 1747809649.856098 write 0x002056 thru 0x00245c (0x407 bytes) 000147 1747809649.872697 read 0x038836 thru 0x03f608 (0x6dd3 bytes) 000148 1747809649.875286 write 0x0284e2 thru 0x029910 (0x142f bytes) 000150 1747809649.879383 read 0x01cdca thru 0x022d59 (0x5f90 bytes) 000152 1747809649.883061 mapread 0x00f30f thru 0x01a23c (0xaf2e bytes) 000153 1747809649.883680 mapwrite 0x03d82b thru 0x03ffff (0x27d5 bytes) 000154 1747809649.899933 mapread 0x030eaa thru 0x0315a7 (0x6fe bytes) 000155 1747809649.900766 write 0x00e7bb thru 0x017cbe (0x9504 bytes) 000156 1747809649.901836 mapread 0x02a87a thru 0x03909c (0xe823 bytes) 000157 1747809649.902668 write 0x01930a thru 0x02117f (0x7e76 bytes) 000158 1747809649.903592 read 0x01ba46 thru 0x02b12c (0xf6e7 bytes) 000159 1747809649.904078 write 0x03e662 thru 0x03ffff (0x199e bytes) 000161 1747809649.904989 read 0x019dbd thru 0x01e90e (0x4b52 bytes) 000163 1747809649.905463 write 0x01ec59 thru 0x0253d9 (0x6781 bytes) 000164 1747809649.905986 mapread 0x015d35 thru 0x019b49 (0x3e15 bytes) 000165 1747809649.908186 MIRROR_EXTEND to 4 mirrors 000168 1747809650.021713 mapread 0x029bc0 thru 0x03255f (0x89a0 bytes) 000170 1747809650.031050 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x91f:0x0]: Invalid argument (22) 000172 1747809650.094954 mapwrite 0x0041e2 thru 0x012751 (0xe570 bytes) 000173 1747809650.139968 mapread 0x030d51 thru 0x03b68a (0xa93a bytes) 000174 1747809650.145825 MIRROR_EXTEND to 4 mirrors 000175 1747809650.245775 write 0x03b5e8 thru 0x03ffff (0x4a18 bytes) 000178 1747809650.262069 read 0x009959 thru 0x01044a (0x6af2 bytes) 000179 1747809650.264201 mapwrite 0x002114 thru 0x00350d (0x13fa bytes) 000180 1747809650.280393 mapread 0x0377e7 thru 0x03ffff (0x8819 bytes) 000181 1747809650.283080 write 0x006ce8 thru 0x01358f (0xc8a8 bytes) 000182 1747809650.284953 mapread 0x021169 thru 0x02b062 (0x9efa bytes) 000183 1747809650.289301 mapread 0x03a5b0 thru 0x03ffff (0x5a50 bytes) 000187 1747809650.290925 mapwrite 0x033e54 thru 0x037965 (0x3b12 bytes) 000188 1747809650.303305 mapread 0x002142 thru 0x00a56b (0x842a bytes) 000189 1747809650.307965 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x91f:0x0]: Invalid argument (22) 000192 1747809650.357793 MIRROR_EXTEND to 4 mirrors 000193 1747809650.459972 write 0x03f91e thru 0x03ffff (0x6e2 bytes) 000194 1747809650.475688 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x91f:0x0]: Invalid argument (22) 000199 1747809650.527893 mapread 0x0271f8 thru 0x032909 (0xb712 bytes) 000201 1747809650.536605 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x91f:0x0]: Invalid argument (22) 000202 1747809650.583270 read 0x039dac thru 0x03ffff (0x6254 bytes) 000205 1747809650.586446 read 0x00d718 thru 0x01b3d4 (0xdcbd bytes) 000206 1747809650.592054 MIRROR_EXTEND to 3 mirrors 000207 1747809650.686119 mapread 0x0160b5 thru 0x01913b (0x3087 bytes) 000208 1747809650.690352 write 0x02283b thru 0x02a4f4 (0x7cba bytes) 000210 1747809650.708248 trunc from 0x040000 to 0x009e6f (0x36190 bytes) 000211 1747809650.717501 MIRROR_RESYNC 3 mirrors 000213 1747809650.824726 mapread 0x006688 thru 0x009e6f (0x37e8 bytes) 000214 1747809650.829358 read 0x003d8f thru 0x007b39 (0x3dab bytes) 000215 1747809650.829725 write 0x0381b5 thru 0x03b23f (0x308b bytes) 000217 1747809650.845212 MIRROR_EXTEND to 4 mirrors 000218 1747809650.943697 mapwrite 0x00db3f thru 0x015bee (0x80b0 bytes) 000219 1747809650.991738 write 0x037148 thru 0x03b6ee (0x45a7 bytes) 000223 1747809650.996083 mapwrite 0x014380 thru 0x01464d (0x2ce bytes) 000224 1747809651.008472 trunc from 0x03b6ef to 0x008902 (0x32dec bytes) 000228 1747809651.018136 trunc from 0x008903 to 0x033257 (0x2a955 bytes) 000233 1747809651.022360 read 0x021900 thru 0x02aef5 (0x95f6 bytes) 000236 1747809651.026086 mapread 0x003395 thru 0x006fd9 (0x3c45 bytes) 000238 1747809651.029301 mapread 0x020953 thru 0x023ce9 (0x3397 bytes) 000240 1747809651.032010 write 0x01865e thru 0x01a2e1 (0x1c84 bytes) 000242 1747809651.034365 trunc from 0x033258 to 0x00b10c (0x2814b bytes) 000243 1747809651.038164 mapread 0x0019e1 thru 0x009337 (0x7957 bytes) 000244 1747809651.038680 trunc from 0x00b10d to 0x021331 (0x16225 bytes) 000246 1747809651.041919 write 0x010144 thru 0x01f10e (0xefcb bytes) 000247 1747809651.045419 mapwrite 0x03d3ae thru 0x03ffff (0x2c52 bytes) 000248 1747809651.063972 write 0x038ff3 thru 0x03c975 (0x3983 bytes) 000249 1747809651.064652 trunc from 0x040000 to 0x02f083 (0x10f7c bytes) 000250 1747809651.068685 trunc from 0x02f084 to 0x00de24 (0x2125f bytes) 000251 1747809651.072226 write 0x004f09 thru 0x013a54 (0xeb4c bytes) 000253 1747809651.073661 read 0x00272f thru 0x00850b (0x5ddd bytes) 000256 1747809651.074542 mapread 0x008de3 thru 0x00c215 (0x3433 bytes) 000260 1747809651.075933 mapwrite 0x023ca8 thru 0x033771 (0xfaca bytes) 000263 1747809651.099123 write 0x03d50d thru 0x03ffff (0x2af3 bytes) 000268 1747809651.100822 trunc from 0x040000 to 0x0392f8 (0x6d07 bytes) 000272 1747809651.106381 trunc from 0x0392f9 to 0x022cdd (0x1661b bytes) 000273 1747809651.110358 read 0x02132a thru 0x022cdd (0x19b4 bytes) 000278 1747809651.111787 mapwrite 0x015158 thru 0x023a94 (0xe93d bytes) 000282 1747809651.132117 mapread 0x00a600 thru 0x00babb (0x14bc bytes) 000283 1747809651.132841 write 0x01f36e thru 0x0274ef (0x8182 bytes) 000284 1747809651.133770 read 0x001c6d thru 0x008540 (0x68d4 bytes) 000285 1747809651.134220 mapread 0x00964f thru 0x0156cd (0xc07f bytes) 000287 1747809651.136177 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x91f:0x0]: Invalid argument (22) 000289 1747809651.187966 read 0x01a0b3 thru 0x0265c4 (0xc512 bytes) 000290 1747809651.190514 mapread 0x0086cd thru 0x012692 (0x9fc6 bytes) 000292 1747809651.198116 write 0x002b75 thru 0x0064c3 (0x394f bytes) 000297 1747809651.199617 trunc from 0x0274f0 to 0x018dd4 (0xe71b bytes) 000298 1747809651.203585 mapread 0x0079f0 thru 0x01473e (0xcd4f bytes) 000301 1747809651.205193 mapwrite 0x037dae thru 0x038f81 (0x11d4 bytes) 000302 1747809651.223821 read 0x02def2 thru 0x037544 (0x9653 bytes) 000304 1747809651.227220 mapwrite 0x0027af thru 0x00b17d (0x89cf bytes) 000305 1747809651.240304 trunc from 0x038f82 to 0x02c59d (0xc9e4 bytes) 000306 1747809651.244381 mapwrite 0x01aa06 thru 0x028540 (0xdb3b bytes) 000307 1747809651.259787 mapread 0x025e40 thru 0x02c59d (0x675e bytes) 000309 1747809651.260693 read 0x022297 thru 0x0251d1 (0x2f3b bytes) 000311 1747809651.261166 mapwrite 0x02f3a1 thru 0x032268 (0x2ec8 bytes) 000314 1747809651.277582 trunc from 0x032269 to 0x0042d1 (0x2df97 bytes) 000315 1747809651.282570 MIRROR_EXTEND to 4 mirrors 000317 1747809651.375454 mapread 0x000022 thru 0x002299 (0x2278 bytes) 000318 1747809651.382034 read 0x000100 thru 0x000fd2 (0xed3 bytes) 000319 1747809651.384126 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x91f:0x0]: Invalid argument (22) 000320 1747809651.435145 mapread 0x000189 thru 0x0042d1 (0x4149 bytes) 000325 1747809651.440973 trunc from 0x0042d2 to 0x02bc11 (0x27940 bytes) 000326 1747809651.461055 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x91f:0x0]: Invalid argument (22) 000327 1747809651.509693 trunc from 0x02bc12 to 0x01aa1c (0x111f5 bytes) 000328 1747809651.516213 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x91f:0x0]: Invalid argument (22) 000329 1747809651.547320 MIRROR_EXTEND to 2 mirrors 000330 1747809651.644461 mapread 0x0129cf thru 0x013d02 (0x1334 bytes) 000333 1747809651.650585 trunc from 0x01aa1d to 0x021d0c (0x72f0 bytes) 000334 1747809651.670823 read 0x001ddd thru 0x003237 (0x145b bytes) 000337 1747809651.672918 write 0x02e037 thru 0x037268 (0x9232 bytes) 000338 1747809651.673604 mapwrite 0x01c84d thru 0x029d29 (0xd4dd bytes) 000339 1747809651.693345 mapread 0x02f3a6 thru 0x037268 (0x7ec3 bytes) 000340 1747809651.694026 mapread 0x009bfa thru 0x00a509 (0x910 bytes) 000341 1747809651.697272 mapwrite 0x018042 thru 0x021d50 (0x9d0f bytes) 000342 1747809651.711101 mapwrite 0x016f53 thru 0x01e394 (0x7442 bytes) 000343 1747809651.723013 write 0x010f31 thru 0x013ab7 (0x2b87 bytes) 000344 1747809651.725177 mapread 0x02f246 thru 0x037268 (0x8023 bytes) 000345 1747809651.726114 trunc from 0x037269 to 0x036ebc (0x3ac bytes) 000347 1747809651.732579 mapread 0x01f806 thru 0x02f489 (0xfc84 bytes) 000348 1747809651.733607 mapread 0x001f2e thru 0x003b31 (0x1c04 bytes) 000350 1747809651.734819 read 0x0337e8 thru 0x03643b (0x2c54 bytes) 000351 1747809651.735383 mapwrite 0x0282b8 thru 0x028b2c (0x875 bytes) 000352 1747809651.748834 trunc from 0x036ebd to 0x000154 (0x36d68 bytes) 000353 1747809651.753256 write 0x006c31 thru 0x00de6a (0x723a bytes) 000355 1747809651.754599 read 0x0067e7 thru 0x00b69d (0x4eb7 bytes) 000356 1747809651.756315 MIRROR_EXTEND to 3 mirrors 000360 1747809651.852613 write 0x033c82 thru 0x03ffff (0xc37e bytes) 000362 1747809651.866028 mapwrite 0x0151bb thru 0x0159ed (0x833 bytes) 000363 1747809651.880274 write 0x01955f thru 0x02773f (0xe1e1 bytes) 000364 1747809651.883259 read 0x02f5d2 thru 0x0362c8 (0x6cf7 bytes) 000367 1747809651.885598 write 0x01dc69 thru 0x026bbf (0x8f57 bytes) 000368 1747809651.886195 write 0x035612 thru 0x038721 (0x3110 bytes) 000369 1747809651.886652 trunc from 0x040000 to 0x016ffc (0x29003 bytes) 000370 1747809651.894204 write 0x0280e0 thru 0x02eb91 (0x6ab2 bytes) 000371 1747809651.895898 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x91f:0x0]: Invalid argument (22) 000374 1747809651.945346 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x91f:0x0]: Invalid argument (22) 000376 1747809651.978048 write 0x028339 thru 0x028731 (0x3f9 bytes) 000377 1747809651.979853 write 0x02612b thru 0x02c59f (0x6475 bytes) 000379 1747809651.983169 write 0x0000fe thru 0x00bf2b (0xbe2e bytes) 000380 1747809651.987299 MIRROR_EXTEND to 2 mirrors 000381 1747809652.100791 MIRROR_EXTEND to 3 mirrors 000382 1747809652.203765 write 0x02c5f5 thru 0x0318e9 (0x52f5 bytes) 000383 1747809652.219376 write 0x014504 thru 0x01634a (0x1e47 bytes) 000385 1747809652.223017 read 0x02ba4c thru 0x0318e9 (0x5e9e bytes) 000386 1747809652.224360 trunc from 0x0318ea to 0x01ff2d (0x119bc bytes) 000387 1747809652.230873 MIRROR_EXTEND to 4 mirrors 000388 1747809652.333455 mapread 0x00534f thru 0x012985 (0xd637 bytes) 000389 1747809652.344003 MIRROR_EXTEND to 5 mirrors 000390 1747809652.451059 mapwrite 0x02577a thru 0x026dee (0x1675 bytes) 000392 1747809652.489013 write 0x028047 thru 0x02f229 (0x71e3 bytes) 000396 1747809652.491101 write 0x00d6d5 thru 0x00ec65 (0x1591 bytes) 000397 1747809652.494137 write 0x0105a4 thru 0x017185 (0x6be2 bytes) 000399 1747809652.497587 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x91f:0x0]: Invalid argument (22) 000400 1747809652.552067 read 0x025290 thru 0x029f9d (0x4d0e bytes) 000402 1747809652.556149 MIRROR_RESYNC 4 mirrors 000403 1747809652.686360 MIRROR_RESYNC 4 mirrors 000404 1747809652.725946 write 0x03520c thru 0x0377f9 (0x25ee bytes) 000405 1747809652.740165 MIRROR_EXTEND to 5 mirrors 000407 1747809652.835635 MIRROR_EXTEND to 6 mirrors truncating to largest ever: 0x3f431 000408 1747809652.936933 trunc from 0x0377fa to 0x03f430 (0x7c37 bytes) 000409 1747809652.955423 write 0x009dd2 thru 0x01333a (0x9569 bytes) 000412 1747809652.958634 write 0x004a1c thru 0x01416c (0xf751 bytes) 000414 1747809652.963307 trunc from 0x03f431 to 0x00c534 (0x32efc bytes) 000417 1747809652.968141 read 0x0035bb thru 0x009e27 (0x686d bytes) 000418 1747809652.968556 mapwrite 0x02dfb4 thru 0x03c6f5 (0xe742 bytes) 000419 1747809652.991497 trunc from 0x03c6f6 to 0x02cd31 (0xf9c4 bytes) 000420 1747809652.995315 trunc from 0x02cd32 to 0x03d204 (0x104d3 bytes) 000426 1747809653.000474 read 0x029727 thru 0x03589b (0xc175 bytes) 000428 1747809653.004602 mapwrite 0x03dcc4 thru 0x03ffff (0x233c bytes) 000430 1747809653.022840 mapwrite 0x02275b thru 0x0231c7 (0xa6d bytes) 000431 1747809653.032391 mapwrite 0x028c45 thru 0x02e88c (0x5c48 bytes) 000432 1747809653.043105 mapread 0x02f8ad thru 0x03c81b (0xcf6f bytes) 000433 1747809653.043832 trunc from 0x040000 to 0x0037b5 (0x3c84a bytes) 000435 1747809653.048185 write 0x01bfe3 thru 0x01f91a (0x3938 bytes) 000437 1747809653.049223 mapwrite 0x01df6f thru 0x02d659 (0xf6eb bytes) 000438 1747809653.081502 mapwrite 0x032b01 thru 0x035e34 (0x3334 bytes) 000439 1747809653.104320 read 0x005c1f thru 0x00edcb (0x91ad bytes) 000441 1747809653.109388 MIRROR_RESYNC 6 mirrors 000445 1747809653.273603 mapwrite 0x0116a4 thru 0x01589d (0x41fa bytes) 000449 1747809653.321044 write 0x0348d4 thru 0x03ce55 (0x8582 bytes) 000451 1747809653.323889 mapread 0x008859 thru 0x00cc0e (0x43b6 bytes) 000452 1747809653.327683 MIRROR_RESYNC 6 mirrors 000453 1747809653.511166 read 0x0348fd thru 0x03ce55 (0x8559 bytes) 000455 1747809653.513205 read 0x02f85d thru 0x036686 (0x6e2a bytes) 000457 1747809653.515878 mapwrite 0x018f9a thru 0x01ec4b (0x5cb2 bytes) 000458 1747809653.560607 mapwrite 0x021056 thru 0x022ea4 (0x1e4f bytes) 000460 1747809653.574653 mapread 0x02f6be thru 0x0378e3 (0x8226 bytes) 000462 1747809653.584239 mapwrite 0x00dd7c thru 0x010052 (0x22d7 bytes) 000463 1747809653.600107 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x91f:0x0]: Invalid argument (22) 000465 1747809653.652628 mapwrite 0x019f84 thru 0x0281d8 (0xe255 bytes) 000469 1747809653.677552 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x91f:0x0]: Invalid argument (22) 000474 1747809653.728663 MIRROR_RESYNC 4 mirrors 000475 1747809653.858365 mapwrite 0x026c2a thru 0x0319d8 (0xadaf bytes) 000477 1747809653.907127 trunc from 0x03ce56 to 0x0347dc (0x8679 bytes) 000480 1747809653.917324 mapwrite 0x005a00 thru 0x00fa01 (0xa002 bytes) 000481 1747809653.935134 mapread 0x00bcd2 thru 0x010108 (0x4437 bytes) 000482 1747809653.935790 write 0x020754 thru 0x02773a (0x6fe7 bytes) 000483 1747809653.936416 mapread 0x00c008 thru 0x00d504 (0x14fd bytes) 000485 1747809653.939108 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x91f:0x0]: Invalid argument (22) 000488 1747809653.992222 trunc from 0x0347dd to 0x012f8b (0x21851 bytes) 000490 1747809653.996091 mapread 0x0061f9 thru 0x007a35 (0x183d bytes) 000493 1747809653.999270 trunc from 0x012f8c to 0x001fbb (0x10fd0 bytes) 000494 1747809654.002376 read 0x00016f thru 0x000993 (0x825 bytes) 000497 1747809654.004162 write 0x034702 thru 0x03c5ea (0x7ee9 bytes) 000498 1747809654.005066 read 0x00cd42 thru 0x013a2c (0x6ceb bytes) 000499 1747809654.008175 read 0x013ff9 thru 0x020077 (0xc07f bytes) 000500 1747809654.008593 trunc from 0x03c5eb to 0x00ac84 (0x31966 bytes) 000501 1747809654.012362 read 0x0099b7 thru 0x00ac84 (0x12ce bytes) 000503 1747809654.013217 write 0x019550 thru 0x02652f (0xcfe0 bytes) 000504 1747809654.014055 mapread 0x008d8b thru 0x0176de (0xe954 bytes) 000505 1747809654.016798 write 0x0083c4 thru 0x00e87b (0x64b8 bytes) 000506 1747809654.017306 mapwrite 0x02279d thru 0x02428d (0x1af1 bytes) 000507 1747809654.030945 trunc from 0x026530 to 0x034597 (0xe068 bytes) 000508 1747809654.034730 write 0x03d24d thru 0x03ffff (0x2db3 bytes) 000509 1747809654.035280 read 0x007fd2 thru 0x012939 (0xa968 bytes) 000514 1747809654.036666 mapread 0x00009e thru 0x00d565 (0xd4c8 bytes) 000515 1747809654.037122 mapread 0x01b45d thru 0x021db6 (0x695a bytes) 000518 1747809654.039156 MIRROR_EXTEND to 4 mirrors 000519 1747809654.136845 mapwrite 0x018489 thru 0x019183 (0xcfb bytes) 000520 1747809654.185222 mapwrite 0x027628 thru 0x02f93e (0x8317 bytes) 000521 1747809654.202338 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x91f:0x0]: Invalid argument (22) 000523 1747809654.255844 mapwrite 0x00a6b9 thru 0x013408 (0x8d50 bytes) 000524 1747809654.274430 write 0x01cd19 thru 0x01fba4 (0x2e8c bytes) 000526 1747809654.278889 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x91f:0x0]: Invalid argument (22) 000527 1747809654.329455 mapread 0x00b7ec thru 0x00fedc (0x46f1 bytes) 000528 1747809654.334977 trunc from 0x040000 to 0x015d69 (0x2a296 bytes) 000530 1747809654.343736 trunc from 0x015d6a to 0x02bc8d (0x15f24 bytes) 000532 1747809654.347230 mapread 0x025696 thru 0x02a2f3 (0x4c5e bytes) 000533 1747809654.349933 mapwrite 0x0397ab thru 0x03ffff (0x6855 bytes) 000534 1747809654.370810 read 0x00f86e thru 0x013175 (0x3908 bytes) 000535 1747809654.371144 read 0x01ac12 thru 0x021584 (0x6973 bytes) 000536 1747809654.374529 trunc from 0x040000 to 0x02801a (0x17fe5 bytes) 000537 1747809654.378235 trunc from 0x02801b to 0x03662d (0xe613 bytes) 000540 1747809654.382272 trunc from 0x03662e to 0x007ffc (0x2e631 bytes) 000542 1747809654.385952 mapread 0x0039c0 thru 0x007ffc (0x463d bytes) 000544 1747809654.386756 write 0x039dc8 thru 0x03dffb (0x4234 bytes) 000546 1747809654.387524 mapread 0x02ea1b thru 0x03473e (0x5d24 bytes) 000547 1747809654.396258 trunc from 0x03dffc to 0x003d60 (0x3a29b bytes) 000549 1747809654.400098 mapwrite 0x03ef91 thru 0x03ffff (0x106f bytes) 000551 1747809654.419605 mapread 0x033c8f thru 0x03ffff (0xc371 bytes) 000555 1747809654.432246 write 0x00844d thru 0x00c8de (0x4492 bytes) 000559 1747809654.435600 trunc from 0x040000 to 0x02471a (0x1b8e5 bytes) 000560 1747809654.439601 mapread 0x014cd8 thru 0x020b72 (0xbe9b bytes) 000562 1747809654.454757 trunc from 0x02471b to 0x032b0f (0xe3f5 bytes) 000563 1747809654.458549 read 0x01a5e4 thru 0x020317 (0x5d34 bytes) 000564 1747809654.458895 trunc from 0x032b10 to 0x01e161 (0x149ae bytes) 000565 1747809654.462546 mapwrite 0x02c538 thru 0x034cc2 (0x878b bytes) 000566 1747809654.484725 mapwrite 0x0067ea thru 0x00e4af (0x7cc6 bytes) 000567 1747809654.498131 mapread 0x023dc6 thru 0x0262d4 (0x250f bytes) 000568 1747809654.498674 mapwrite 0x01f684 thru 0x02f54e (0xfecb bytes) truncating to largest ever: 0x3fd3b 000570 1747809654.515240 trunc from 0x034cc3 to 0x03fd3a (0xb078 bytes) 000572 1747809654.521197 MIRROR_RESYNC 2 mirrors 000573 1747809654.615414 mapwrite 0x010183 thru 0x0122ad (0x212b bytes) 000576 1747809654.660672 mapread 0x0323b1 thru 0x03be48 (0x9a98 bytes) 000577 1747809654.665298 trunc from 0x03fd3b to 0x002176 (0x3dbc4 bytes) 000581 1747809654.674734 trunc from 0x002177 to 0x03433a (0x321c4 bytes) 000584 1747809654.680799 MIRROR_EXTEND to 3 mirrors 000585 1747809654.777245 trunc from 0x03433b to 0x020e27 (0x13513 bytes) 000586 1747809654.797661 mapwrite 0x000676 thru 0x003fa2 (0x392d bytes) 000587 1747809654.819830 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x91f:0x0]: Invalid argument (22) 000588 1747809654.866643 trunc from 0x020e28 to 0x009b57 (0x172d0 bytes) 000592 1747809654.870901 read 0x008e9d thru 0x009b57 (0xcbb bytes) 000593 1747809654.872600 read 0x0054c1 thru 0x009b57 (0x4697 bytes) 000594 1747809654.875102 trunc from 0x009b58 to 0x00a061 (0x50a bytes) 000599 1747809654.879369 trunc from 0x00a062 to 0x020045 (0x15fe4 bytes) 000600 1747809654.882789 mapwrite 0x03d99c thru 0x03ffff (0x2664 bytes) 000602 1747809654.904172 mapwrite 0x0245e3 thru 0x026309 (0x1d27 bytes) 000605 1747809654.915382 mapwrite 0x025542 thru 0x033ef0 (0xe9af bytes) 000606 1747809654.931075 read 0x02e3d5 thru 0x032a3d (0x4669 bytes) 000607 1747809654.931655 mapread 0x001a6b thru 0x00ab9b (0x9131 bytes) 000609 1747809654.932449 mapread 0x00fb13 thru 0x017273 (0x7761 bytes) 000610 1747809654.933020 mapwrite 0x0275e7 thru 0x02e235 (0x6c4f bytes) 000611 1747809654.945524 mapwrite 0x028af2 thru 0x02baf4 (0x3003 bytes) 000616 1747809654.957168 mapread 0x00059c thru 0x0027f4 (0x2259 bytes) 000617 1747809654.959180 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x91f:0x0]: Invalid argument (22) 000618 1747809654.988909 mapwrite 0x016fd4 thru 0x0191fb (0x2228 bytes) 000619 1747809655.004066 trunc from 0x040000 to 0x03f5f2 (0xa0d bytes) 000621 1747809655.009027 trunc from 0x03f5f3 to 0x01380e (0x2bde4 bytes) 000624 1747809655.013672 write 0x000a94 thru 0x00bf0c (0xb479 bytes) 000627 1747809655.016892 trunc from 0x01380f to 0x02b443 (0x17c35 bytes) 000630 1747809655.021553 mapread 0x01c4d5 thru 0x023234 (0x6d60 bytes) 000631 1747809655.024545 write 0x008dd8 thru 0x00f866 (0x6a8f bytes) 000632 1747809655.025167 mapread 0x009a8a thru 0x00eb8b (0x5102 bytes) 000633 1747809655.025631 write 0x00c088 thru 0x016811 (0xa78a bytes) 000634 1747809655.027639 mapwrite 0x00ad9c thru 0x017d56 (0xcfbb bytes) 000636 1747809655.045606 mapwrite 0x032cc6 thru 0x039a1c (0x6d57 bytes) 000637 1747809655.067206 mapwrite 0x015650 thru 0x0169cb (0x137c bytes) 000638 1747809655.079327 mapread 0x0205ca thru 0x0264a1 (0x5ed8 bytes) 000640 1747809655.080148 write 0x03bf1e thru 0x03ffff (0x40e2 bytes) 000641 1747809655.080774 trunc from 0x040000 to 0x038fcb (0x7034 bytes) 000642 1747809655.085039 trunc from 0x038fcc to 0x00732c (0x31c9f bytes) 000646 1747809655.089907 mapread 0x006f29 thru 0x00732c (0x404 bytes) 000647 1747809655.090619 mapwrite 0x01c519 thru 0x02afea (0xead2 bytes) 000648 1747809655.119104 mapwrite 0x034937 thru 0x03ffff (0xb6c9 bytes) 000651 1747809655.140574 trunc from 0x040000 to 0x016dfc (0x29203 bytes) 000655 1747809655.145433 mapread 0x00c608 thru 0x016dfc (0xa7f5 bytes) 000656 1747809655.150181 trunc from 0x016dfd to 0x01c944 (0x5b48 bytes) 000659 1747809655.154633 mapread 0x0001cc thru 0x00a491 (0xa2c6 bytes) 000660 1747809655.155213 write 0x03d31d thru 0x03ffff (0x2ce3 bytes) 000661 1747809655.156038 mapwrite 0x000e14 thru 0x0089a3 (0x7b90 bytes) 000662 1747809655.170950 read 0x0044fe thru 0x00deb4 (0x99b7 bytes) 000663 1747809655.171462 write 0x01a0b5 thru 0x01d06d (0x2fb9 bytes) 000667 1747809655.175398 read 0x03f144 thru 0x03ffff (0xebc bytes) 000668 1747809655.175763 trunc from 0x040000 to 0x02d61b (0x129e4 bytes) 000669 1747809655.180076 mapread 0x005710 thru 0x00b13b (0x5a2c bytes) 000670 1747809655.180661 trunc from 0x02d61c to 0x02b6ee (0x1f2d bytes) 000672 1747809655.184930 write 0x013085 thru 0x020997 (0xd913 bytes) 000673 1747809655.185743 read 0x022e8c thru 0x022efe (0x73 bytes) 000674 1747809655.186084 write 0x0181c5 thru 0x02386d (0xb6a9 bytes) 000687 1747809655.192594 trunc from 0x02b6ef to 0x01f03f (0xc6af bytes) 000689 1747809655.197053 mapread 0x003036 thru 0x00816f (0x513a bytes) 000691 1747809655.197768 mapwrite 0x00298c thru 0x0035f0 (0xc65 bytes) 000693 1747809655.212535 mapwrite 0x01aa75 thru 0x01d2eb (0x2877 bytes) 000696 1747809655.225279 trunc from 0x01f040 to 0x033e5b (0x14e1c bytes) 000697 1747809655.229361 mapread 0x02877f thru 0x03384e (0xb0d0 bytes) 000698 1747809655.235743 mapread 0x014557 thru 0x020941 (0xc3eb bytes) 000699 1747809655.237660 read 0x0126b7 thru 0x018f1d (0x6867 bytes) 000700 1747809655.237970 trunc from 0x033e5c to 0x02752d (0xc92e bytes) 000703 1747809655.242738 trunc from 0x02752e to 0x007a12 (0x1fb1b bytes) 000704 1747809655.247153 trunc from 0x007a13 to 0x0164ab (0xea99 bytes) 000705 1747809655.251299 trunc from 0x0164ac to 0x02c934 (0x16489 bytes) 000706 1747809655.255322 mapread 0x00ca44 thru 0x01222f (0x57ec bytes) 000707 1747809655.259441 mapread 0x00eb1f thru 0x01648b (0x796d bytes) 000708 1747809655.261824 MIRROR_EXTEND to 2 mirrors 000711 1747809655.366281 mapwrite 0x01b297 thru 0x0283b5 (0xd11f bytes) 000712 1747809655.417389 read 0x02b0c1 thru 0x02c934 (0x1874 bytes) 000713 1747809655.417757 write 0x012855 thru 0x016dfa (0x45a6 bytes) 000714 1747809655.420718 read 0x001c0e thru 0x009a6b (0x7e5e bytes) 000716 1747809655.423705 write 0x03decc thru 0x03ffff (0x2134 bytes) 000719 1747809655.425231 read 0x0091a8 thru 0x00c35b (0x31b4 bytes) 000720 1747809655.425572 write 0x003632 thru 0x00c7dd (0x91ac bytes) 000721 1747809655.426434 read 0x010325 thru 0x012608 (0x22e4 bytes) 000723 1747809655.427052 mapread 0x008f7c thru 0x017157 (0xe1dc bytes) 000724 1747809655.427696 trunc from 0x040000 to 0x012c84 (0x2d37b bytes) 000725 1747809655.438258 MIRROR_EXTEND to 3 mirrors 000726 1747809655.549181 MIRROR_EXTEND to 4 mirrors 000727 1747809655.650075 read 0x01278c thru 0x012c84 (0x4f9 bytes) 000728 1747809655.652237 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x91f:0x0]: Invalid argument (22) 000730 1747809655.705924 trunc from 0x012c85 to 0x0388c0 (0x25c3c bytes) 000732 1747809655.726785 MIRROR_RESYNC 3 mirrors 000735 1747809655.848653 trunc from 0x0388c1 to 0x01ee26 (0x19a9a bytes) 000736 1747809655.866916 MIRROR_RESYNC 3 mirrors 000737 1747809655.975060 read 0x000c93 thru 0x00e6d9 (0xda47 bytes) 000739 1747809655.977668 trunc from 0x01ee27 to 0x031c27 (0x12e01 bytes) 000740 1747809655.996559 read 0x00534e thru 0x009723 (0x43d6 bytes) 000741 1747809656.001693 MIRROR_EXTEND to 4 mirrors 000742 1747809656.117817 mapread 0x0062ae thru 0x0161ef (0xff42 bytes) 000743 1747809656.127560 write 0x02615a thru 0x0341ab (0xe052 bytes) 000747 1747809656.144978 trunc from 0x0341ac to 0x016475 (0x1dd36 bytes) 000748 1747809656.150506 read 0x0150c3 thru 0x016475 (0x13b3 bytes) 000751 1747809656.153255 mapwrite 0x00d4f8 thru 0x00d81a (0x323 bytes) 000754 1747809656.168569 trunc from 0x016476 to 0x02b5c5 (0x15150 bytes) 000755 1747809656.173138 write 0x00d323 thru 0x01bd92 (0xea70 bytes) 000756 1747809656.175410 read 0x01a234 thru 0x025edc (0xbca9 bytes) 000757 1747809656.178314 read 0x00fc9f thru 0x01b460 (0xb7c2 bytes) 000759 1747809656.178880 mapread 0x00c2de thru 0x017f97 (0xbcba bytes) 000760 1747809656.179376 trunc from 0x02b5c6 to 0x00f274 (0x1c351 bytes) 000761 1747809656.184522 MIRROR_EXTEND to 5 mirrors 000763 1747809656.274502 MIRROR_EXTEND to 6 mirrors 000765 1747809656.366134 trunc from 0x00f275 to 0x010143 (0xecf bytes) 000767 1747809656.386874 MIRROR_RESYNC 6 mirrors 000769 1747809656.567442 write 0x01af18 thru 0x020bae (0x5c97 bytes) 000771 1747809656.582195 write 0x004b0a thru 0x00986a (0x4d61 bytes) 000773 1747809656.585350 mapwrite 0x01552a thru 0x023a84 (0xe55b bytes) 000774 1747809656.619891 trunc from 0x023a85 to 0x02802d (0x45a9 bytes) 000775 1747809656.623966 write 0x02c917 thru 0x036173 (0x985d bytes) 000777 1747809656.625147 read 0x00e0f8 thru 0x011c21 (0x3b2a bytes) 000778 1747809656.625481 write 0x01c42b thru 0x028746 (0xc31c bytes) 000779 1747809656.628652 trunc from 0x036174 to 0x026812 (0xf961 bytes) 000780 1747809656.632735 mapread 0x0225ee thru 0x026812 (0x4225 bytes) 000781 1747809656.634790 MIRROR_RESYNC 6 mirrors 000782 1747809656.799179 mapread 0x019138 thru 0x026812 (0xd6db bytes) 000783 1747809656.805023 mapread 0x022104 thru 0x026812 (0x470f bytes) 000784 1747809656.805468 mapread 0x00122e thru 0x00a20e (0x8fe1 bytes) 000786 1747809656.810076 read 0x02008f thru 0x021636 (0x15a8 bytes) 000788 1747809656.811679 MIRROR_RESYNC 6 mirrors 000791 1747809656.857810 trunc from 0x026813 to 0x033dc0 (0xd5ae bytes) 000792 1747809656.877814 mapread 0x01b5b3 thru 0x01f55a (0x3fa8 bytes) 000793 1747809656.882811 mapread 0x00c692 thru 0x019b57 (0xd4c6 bytes) 000794 1747809656.889075 MIRROR_RESYNC 6 mirrors 000795 1747809657.069857 MIRROR_RESYNC 6 mirrors 000796 1747809657.120149 mapread 0x0022ce thru 0x010d12 (0xea45 bytes) 000797 1747809657.127205 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x91f:0x0]: Invalid argument (22) 000802 1747809657.189006 mapread 0x030e01 thru 0x033dc0 (0x2fc0 bytes) 000803 1747809657.194001 mapread 0x01c443 thru 0x01ddf7 (0x19b5 bytes) 000807 1747809657.197858 mapwrite 0x037006 thru 0x03ffff (0x8ffa bytes) 000809 1747809657.231082 mapread 0x0153f4 thru 0x01866c (0x3279 bytes) 000810 1747809657.233828 read 0x0396b2 thru 0x03b80c (0x215b bytes) 000811 1747809657.234157 mapwrite 0x020572 thru 0x03047e (0xff0d bytes) 000812 1747809657.251742 MIRROR_EXTEND to 6 mirrors 000813 1747809657.355158 read 0x027520 thru 0x029883 (0x2364 bytes) 000823 1747809657.367025 mapwrite 0x02c13b thru 0x0317e7 (0x56ad bytes) 000824 1747809657.413958 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x91f:0x0]: Invalid argument (22) 000825 1747809657.468299 write 0x02f0f2 thru 0x034ce8 (0x5bf7 bytes) 000826 1747809657.471312 trunc from 0x040000 to 0x01b82f (0x247d0 bytes) 000827 1747809657.480012 trunc from 0x01b830 to 0x019607 (0x2228 bytes) 000828 1747809657.483898 mapread 0x0059dc thru 0x00c7b8 (0x6ddd bytes) 000830 1747809657.487944 trunc from 0x019608 to 0x00db00 (0xbb07 bytes) 000832 1747809657.492264 trunc from 0x00db01 to 0x02e30f (0x2080f bytes) 000835 1747809657.496646 mapwrite 0x0365cb thru 0x03ffff (0x9a35 bytes) 000839 1747809657.519029 mapread 0x033fa7 thru 0x039f78 (0x5fd2 bytes) 000841 1747809657.520245 write 0x00a681 thru 0x012fe0 (0x8960 bytes) 000844 1747809657.522898 read 0x03a5e8 thru 0x03cb92 (0x25ab bytes) 000846 1747809657.523389 mapread 0x00163b thru 0x00d345 (0xbd0b bytes) 000847 1747809657.525173 mapread 0x0238d5 thru 0x02991f (0x604b bytes) 000848 1747809657.527823 mapread 0x02ab64 thru 0x02b06b (0x508 bytes) 000849 1747809657.528303 mapwrite 0x009826 thru 0x00f9f8 (0x61d3 bytes) 000850 1747809657.541925 mapread 0x0189d6 thru 0x023744 (0xad6f bytes) 000851 1747809657.546397 mapwrite 0x0180e9 thru 0x021fc3 (0x9edb bytes) 000852 1747809657.558327 mapread 0x02fd5c thru 0x03a6be (0xa963 bytes) 000855 1747809657.561164 MIRROR_EXTEND to 6 mirrors 000858 1747809657.673679 trunc from 0x040000 to 0x01d57e (0x22a81 bytes) 000859 1747809657.693290 read 0x0159ca thru 0x01cacc (0x7103 bytes) 000860 1747809657.695607 write 0x024bf0 thru 0x02f813 (0xac24 bytes) 000861 1747809657.696282 mapread 0x004d21 thru 0x011bf4 (0xced4 bytes) 000863 1747809657.703116 read 0x003d5a thru 0x008014 (0x42bb bytes) 000864 1747809657.703403 read 0x007404 thru 0x00d258 (0x5e55 bytes) 000865 1747809657.703708 write 0x031b2d thru 0x03cf00 (0xb3d4 bytes) 000866 1747809657.704354 write 0x02554b thru 0x025673 (0x129 bytes) 000868 1747809657.704854 write 0x0274a7 thru 0x0278f5 (0x44f bytes) 000869 1747809657.705109 mapwrite 0x0393f6 thru 0x03a36a (0xf75 bytes) 000872 1747809657.718347 mapread 0x038ef8 thru 0x03cf00 (0x4009 bytes) 000873 1747809657.718799 write 0x002a1e thru 0x010065 (0xd648 bytes) 000874 1747809657.719471 trunc from 0x03cf01 to 0x01850d (0x249f3 bytes) 000875 1747809657.726554 MIRROR_RESYNC 6 mirrors 000876 1747809657.907177 mapwrite 0x01e842 thru 0x02a2c8 (0xba87 bytes) 000879 1747809657.948558 mapwrite 0x00e75d thru 0x00f547 (0xdeb bytes) 000881 1747809657.961853 write 0x02e593 thru 0x03baca (0xd538 bytes) 000884 1747809657.964498 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x91f:0x0]: Invalid argument (22) 000887 1747809658.015625 mapwrite 0x0069a1 thru 0x00f699 (0x8cf9 bytes) 000888 1747809658.035993 trunc from 0x03bacb to 0x031a24 (0xa0a6 bytes) 000890 1747809658.040808 mapread 0x01860b thru 0x01a5e5 (0x1fdb bytes) 000891 1747809658.043820 read 0x0124b5 thru 0x01861b (0x6167 bytes) 000892 1747809658.045774 trunc from 0x031a25 to 0x01c8cd (0x15157 bytes) 000895 1747809658.049711 read 0x004008 thru 0x007d9b (0x3d94 bytes) 000896 1747809658.050027 read 0x01b5ff thru 0x01c8cd (0x12cf bytes) 000897 1747809658.050314 mapread 0x00ee41 thru 0x016c87 (0x7e47 bytes) 000899 1747809658.052828 trunc from 0x01c8ce to 0x023f56 (0x7689 bytes) 000900 1747809658.056312 mapwrite 0x034e5a thru 0x03585c (0xa03 bytes) 000902 1747809658.075128 write 0x004de7 thru 0x0068e7 (0x1b01 bytes) 000903 1747809658.076112 mapread 0x0265f0 thru 0x0276b2 (0x10c3 bytes) 000905 1747809658.077406 write 0x000612 thru 0x00a1ad (0x9b9c bytes) 000907 1747809658.078945 read 0x019d89 thru 0x01c1c8 (0x2440 bytes) 000908 1747809658.079327 trunc from 0x03585d to 0x0026e0 (0x3317c bytes) 000913 1747809658.084595 write 0x008909 thru 0x0115ac (0x8ca4 bytes) 000914 1747809658.085197 write 0x01ebba thru 0x02612c (0x7573 bytes) 000915 1747809658.085790 write 0x03cbcd thru 0x03ffff (0x3433 bytes) 000916 1747809658.086255 mapwrite 0x036f8d thru 0x03ffff (0x9073 bytes) 000917 1747809658.106539 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x91f:0x0]: Invalid argument (22) 000918 1747809658.156952 write 0x03d256 thru 0x03fce8 (0x2a93 bytes) 000919 1747809658.159728 mapread 0x0241a0 thru 0x02b2f2 (0x7153 bytes) 000920 1747809658.162306 mapread 0x00aea2 thru 0x014a5a (0x9bb9 bytes) 000922 1747809658.167109 mapread 0x030a3d thru 0x031104 (0x6c8 bytes) 000923 1747809658.167664 write 0x03f53e thru 0x03ffff (0xac2 bytes) 000926 1747809658.168343 mapwrite 0x02575f thru 0x02fcea (0xa58c bytes) 000928 1747809658.182512 trunc from 0x040000 to 0x0280de (0x17f21 bytes) 000929 1747809658.188158 MIRROR_EXTEND to 5 mirrors 000930 1747809658.282697 trunc from 0x0280df to 0x0000ed (0x27ff1 bytes) 000931 1747809658.301543 write 0x00fde9 thru 0x017e86 (0x809e bytes) 000932 1747809658.302435 mapwrite 0x012d90 thru 0x01dff7 (0xb268 bytes) 000933 1747809658.325184 MIRROR_RESYNC 5 mirrors 000934 1747809658.468928 trunc from 0x01dff8 to 0x01e285 (0x28e bytes) 000935 1747809658.488017 MIRROR_RESYNC 5 mirrors 000938 1747809658.622869 read 0x01ca74 thru 0x01e285 (0x1812 bytes) 000939 1747809658.624191 read 0x01d94e thru 0x01e285 (0x938 bytes) 000940 1747809658.624380 trunc from 0x01e286 to 0x024224 (0x5f9f bytes) 000942 1747809658.642527 mapwrite 0x02e952 thru 0x02eb1b (0x1ca bytes) 000943 1747809658.661571 mapread 0x02e1e1 thru 0x02eb1b (0x93b bytes) 000945 1747809658.662648 read 0x02c885 thru 0x02eb1b (0x2297 bytes) 000946 1747809658.662938 read 0x01707a thru 0x01e078 (0x6fff bytes) 000947 1747809658.663261 mapread 0x01cdca thru 0x0248a3 (0x7ada bytes) 000948 1747809658.663903 mapwrite 0x030461 thru 0x03a40d (0x9fad bytes) 000949 1747809658.683133 write 0x015d7d thru 0x022cf0 (0xcf74 bytes) 000950 1747809658.683900 mapread 0x020686 thru 0x0254e9 (0x4e64 bytes) 000951 1747809658.684300 mapwrite 0x004b09 thru 0x011e6e (0xd366 bytes) 000953 1747809658.698378 write 0x004e19 thru 0x00f237 (0xa41f bytes) 000956 1747809658.700094 read 0x0352c3 thru 0x03a40d (0x514b bytes) 000957 1747809658.700579 trunc from 0x03a40e to 0x03423f (0x61ce bytes) 000959 1747809658.704968 mapread 0x007e80 thru 0x0170fd (0xf27e bytes) 000960 1747809658.705762 read 0x003e8d thru 0x00ab07 (0x6c7b bytes) 000961 1747809658.706245 trunc from 0x034240 to 0x0001b6 (0x34089 bytes) 000962 1747809658.710586 read 0x000093 thru 0x0001b6 (0x124 bytes) 000963 1747809658.710885 trunc from 0x0001b7 to 0x00c4f4 (0xc33e bytes) 000964 1747809658.714182 write 0x012f3a thru 0x017edb (0x4fa2 bytes) 000965 1747809658.714777 mapread 0x00cc79 thru 0x017edb (0xb263 bytes) 000966 1747809658.717395 mapwrite 0x03a519 thru 0x03c8d0 (0x23b8 bytes) 000967 1747809658.736525 mapread 0x01a9c5 thru 0x020456 (0x5a92 bytes) 000968 1747809658.737144 read 0x0034ae thru 0x00465b (0x11ae bytes) 000969 1747809658.737619 mapwrite 0x005244 thru 0x00af6e (0x5d2b bytes) 000972 1747809658.750565 mapread 0x03b673 thru 0x03c8d0 (0x125e bytes) 000975 1747809658.751519 trunc from 0x03c8d1 to 0x01ec20 (0x1dcb0 bytes) 000977 1747809658.756524 write 0x01fa92 thru 0x02f2c7 (0xf836 bytes) 000978 1747809658.759149 MIRROR_EXTEND to 6 mirrors 000979 1747809658.870751 mapread 0x02cef4 thru 0x02ead1 (0x1bde bytes) 000984 1747809658.880873 mapwrite 0x00a1f0 thru 0x015aad (0xb8be bytes) 000986 1747809658.930278 mapread 0x02ae7b thru 0x02f2c7 (0x444d bytes) 000987 1747809658.932912 mapwrite 0x02352e thru 0x03290e (0xf3e1 bytes) 000992 1747809658.960058 write 0x00f055 thru 0x01cc7d (0xdc29 bytes) 000993 1747809658.962428 mapwrite 0x014144 thru 0x01b816 (0x76d3 bytes) 000996 1747809658.977725 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x91f:0x0]: Invalid argument (22) 000999 1747809659.026773 read 0x002a4b thru 0x00ee26 (0xc3dc bytes) 001000 1747809659.030008 write 0x00ed79 thru 0x017ecc (0x9154 bytes) All operations completed A-OK! PASS 70a (13s) == sanity-flr test 71: check mirror extend parent fid ==== 02:41:00 (1747809660) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (0s) == sanity-flr test 200a: stress test ===================== 02:41:00 (1747809660) Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre2 Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2983168 .. Extending file size to 4638912 .. done Extending file size to 6428320 .. Extending file size to 4037856 .. Extending file size to 6462144 .. Extending file size to 5667552 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7964768 .. Extending file size to 6462240 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 7368896 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7612512 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7821824 .. Extending file size to 8568128 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9208928 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8770240 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8958048 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9312224 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9223680 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 30229 30230 30232 30234 30236 done 192.168.204.121@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg421-client.virtnet /mnt/lustre2 (opts:) 192.168.204.121@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg421-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 02:42:04 (1747809724) Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre2 Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3673504 .. Extending file size to 641472 .. Extending file size to 2076544 .. Extending file size to 6696160 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6258848 .. Extending file size to 7756416 .. Extending file size to 6870304 .. Extending file size to 8896352 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9286176 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9238880 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9325248 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 2038 2039 2041 2043 2045 192.168.204.121@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg421-client.virtnet /mnt/lustre2 (opts:) 192.168.204.121@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg421-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 02:43:08 (1747809788) Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 5376 2202880 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 18432 3750912 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 20480 3748864 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 5376 2202880 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 18432 3750912 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 20480 3748864 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0346259 s, 60.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0952586 s, 22.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.121@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg421-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 02:43:13 (1747809793) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0242004 s, 86.7 MB/s PASS 202 (1s) == sanity-flr test 203: mirror file preserve mirror ID === 02:43:14 (1747809794) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x969:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc8b:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc8b:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc8c:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0xc90:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 02:43:16 (1747809796) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0266221 s, 78.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc8d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc91:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc92:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 02:43:18 (1747809798) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0254656 s, 82.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc8e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc8f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc93:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc8e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc8f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc93:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc94:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 02:43:20 (1747809800) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0374457 s, 56.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc90:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc95:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc90:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc95:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc96:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 02:43:43 (1747809823) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg421-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0229998 s, 45.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc91:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc98:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc97:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc91:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc98:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc97:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc92:0x0] } PASS 204d (43s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 02:44:26 (1747809866) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg421-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc93:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc94:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0290657 s, 72.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc93:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc94:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc99:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc93:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc94:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc99:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (44s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 02:45:11 (1747809911) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg421-server mds-ost sync done. Creating new pool test_204f oleg421-server: Pool lustre.test_204f created Adding targets to pool oleg421-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg421-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0383528 s, 54.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc95:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc95:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg421-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg421-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg421-server: Pool lustre.test_204f destroyed PASS 204f (57s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 02:46:08 (1747809968) PASS 205a (1s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 02:46:09 (1747809969) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 02:46:11 (1747809971) oleg421-server: Pool lustre.test_206 created Adding targets to pool oleg421-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg421-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg421-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg421-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg421-server: Pool lustre.test_206 destroyed PASS 206 (11s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 02:46:22 (1747809982) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0165701 s, 63.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00807657 s, 130 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x54da808b Mirror 3: 0x54da808b PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 02:46:24 (1747809984) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 02:46:25 (1747809985) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 02:46:26 (1747809986) Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0175453 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1ca0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1360000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1360000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1360000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1be0000 Read this (4 bytes): BBBB 192.168.204.121@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg421-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 02:46:39 (1747809999) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00290167 s, 2.8 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 02:46:41 (1747810001) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00759459 s, 138 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x98e lmm_fid: [0x200000404:0x98e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x98e lmm_fid: [0x200000404:0x98e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xca4:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322836366 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-21 02:46:41.000000000 -0400 Modify: 2025-05-21 02:46:41.000000000 -0400 Change: 2025-05-21 02:46:42.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 02:46:43 (1747810003) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (0s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 02:46:43 (1747810003) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0974844 s, 420 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0127174 s, 322 kB/s size after second write -rw-r--r-- 1 root root 4096 May 21 02:46 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x990:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 May 21 02:46 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test complete, duration 680 sec ============ 02:46:45 (1747810005) === sanity-flr: start cleanup 02:46:46 (1747810006) === === sanity-flr: finish cleanup 02:46:47 (1747810007) ===