-----============= acceptance-small: sanity-flr ============----- Sat Nov 16 04:36:22 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 6 201 44c 49a === sanity-flr: start setup 04:36:27 (1731749787) === oleg242-client.virtnet: executing check_config_client /mnt/lustre oleg242-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg242-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012a4df000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012a4df000.idle_timeout=debug disable quota as required oleg242-server: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 04:36:34 (1731749794) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f6969] == sanity-flr test 0a: lfs mirror create with -N option == 04:36:35 (1731749795) PASS 0a (5s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 04:36:40 (1731749800) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 04:36:41 (1731749801) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (0s) == sanity-flr test 0d: lfs mirror extend with -N option == 04:36:42 (1731749802) PASS 0d (6s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 04:36:48 (1731749808) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 04:36:49 (1731749809) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 04:36:50 (1731749810) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88012a4df000.stats=clear osc.lustre-OST0001-osc-ffff88012a4df000.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 04:36:52 (1731749812) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 04:36:55 (1731749815) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 04:36:58 (1731749818) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 04:36:59 (1731749819) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 04:37:01 (1731749821) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 04:37:03 (1731749823) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 04:37:04 (1731749824) lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 04:37:06 (1731749826) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (1s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 04:37:08 (1731749828) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 04:37:10 (1731749830) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0683857 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0581397 s, 18.0 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 04:37:13 (1731749833) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB) copied, 0.0478409 s, 132 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 04:37:15 (1731749835) Stopping /mnt/lustre-ost1 (opts:) on oleg242-server oleg242-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a4df000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a4df000.ost_server_uuid in DISCONN state after 3 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg242-server: oleg242-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg242-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg242-server oleg242-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a4df000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a4df000.ost_server_uuid in DISCONN state after 3 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg242-server: oleg242-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg242-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (31s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 04:37:46 (1731749866) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.073807 s, 42.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg242-server oleg242-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a4df000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a4df000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg242-server: oleg242-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg242-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (18s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 04:38:04 (1731749884) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg242-server oleg242-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a4df000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a4df000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg242-server: oleg242-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg242-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg242-server oleg242-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a4df000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a4df000.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg242-server: oleg242-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg242-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (24s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 04:38:28 (1731749908) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.958345 s, 54.7 MB/s -rw-r--r-- 1 root root 52428800 Nov 16 04:38 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.313s user 0m0.002s sys 0m0.118s Stopping /mnt/lustre-ost1 (opts:) on oleg242-server oleg242-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a4df000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a4df000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.330s user 0m0.000s sys 0m0.124s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg242-server: oleg242-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg242-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg242-server oleg242-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a4df000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a4df000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.214s user 0m0.000s sys 0m0.109s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg242-server: oleg242-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg242-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg242-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012a4df000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a4df000.ost_server_uuid in FULL state after 0 sec PASS 33b (27s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 04:38:55 (1731749935) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (0s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 04:38:56 (1731749936) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 04:38:56 (1731749936) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 04:38:57 (1731749937) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 04:39:00 (1731749940) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.187299 s, 67.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 04:39:03 (1731749943) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0157205 s, 66.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0094735 s, 111 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 04:39:05 (1731749945) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 04:39:08 (1731749948) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000871268 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000927773 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00069127 s, 1.5 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00054161 s, 1.9 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 04:39:11 (1731749951) create test files with size 15760680 .. 7e6c3746ad5bfebe38bedf4c052f20a6 - 63f8f02606486b3c093a62f19986eab6 - 8d21838b165f4398d945b3d16ef908a4 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg242-client.virtnet /mnt/lustre (opts:) Starting client: oleg242-client.virtnet: -o user_xattr,flock 192.168.202.142@tcp:/lustre /mnt/lustre PASS 37 (8s) == sanity-flr test 38: resync ============================ 04:39:19 (1731749959) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (4s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 04:39:23 (1731749963) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 04:39:25 (1731749965) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0440477 s, 23.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0096099 s, 109 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 04:39:28 (1731749968) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.041315 s, 50.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.115519 s, 36.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0598967 s, 35.0 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 04:39:31 (1731749971) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 04:39:32 (1731749972) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (0s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 04:39:33 (1731749973) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00878667 s, 119 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 04:39:35 (1731749975) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 04:39:36 (1731749976) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4e:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x13:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 04:39:39 (1731749979) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0712845 s, 147 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg242-client.virtnet /mnt/lustre (opts:) Starting client: oleg242-client.virtnet: -o user_xattr,flock 192.168.202.142@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (3s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 04:39:42 (1731749982) llite.lustre-ffff8800a2253000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0675792 s, 155 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg242-client.virtnet /mnt/lustre (opts:) Starting client: oleg242-client.virtnet: -o user_xattr,flock 192.168.202.142@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff88012aa95000.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 04:39:45 (1731749985) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (9s) == sanity-flr test 46: Verify setstripe --copy option ==== 04:39:54 (1731749994) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (8s) == sanity-flr test 47: Verify mirror obj alloc =========== 04:40:02 (1731750002) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 04:40:03 (1731750003) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0589028 s, 53.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0484723 s, 64.9 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: eb3fcce3c9b867a5dbf4b8ebeadabeb1 - mirror 1 checksum: f8e12a587a8e74def9ac4e34aa9d25f4 - mirror 2 checksum: eb3fcce3c9b867a5dbf4b8ebeadabeb1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sat Nov 16 04:40:02 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sat Nov 16 04:40:02 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: eb3fcce3c9b867a5dbf4b8ebeadabeb1 - mirror 1 checksum: f8e12a587a8e74def9ac4e34aa9d25f4 - mirror 2 checksum: f8e12a587a8e74def9ac4e34aa9d25f4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sat Nov 16 04:40:02 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sat Nov 16 04:40:02 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } PASS 48 (2s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 04:40:06 (1731750006) Failing mds1 on oleg242-server Stopping /mnt/lustre-mds1 (opts:) on oleg242-server 04:40:08 (1731750008) shut down Failover mds1 to oleg242-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0000 04:40:21 (1731750021) targets are mounted 04:40:21 (1731750021) facet_failover done oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102317 s, 102 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } PASS 50A (21s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 04:40:27 (1731750027) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 04:40:28 (1731750028) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (0s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 04:40:28 (1731750028) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00245992 s, 6.7 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 04:40:30 (1731750030) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 04:40:31 (1731750031) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 04:40:32 (1731750032) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (5s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 04:40:37 (1731750037) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 04:40:55 (1731750055) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 04:41:12 (1731750072) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 04:41:29 (1731750089) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 04:41:31 (1731750091) mirror create pid 22280 mirror split pid 22281 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4851: 22280 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4851: 22281 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 04:42:33 (1731750153) Seed set to 153 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000004 1731750153.785786 write 0x00dc12 thru 0x01152f (0x391e bytes) 000005 1731750153.786661 mapread 0x000826 thru 0x001418 (0xbf3 bytes) 000007 1731750153.791246 mapread 0x009ab6 thru 0x00ee5b (0x53a6 bytes) 000009 1731750153.794608 read 0x005c32 thru 0x010e3b (0xb20a bytes) 000012 1731750153.797227 MIRROR_EXTEND to 2 mirrors 000013 1731750153.918119 mapwrite 0x023ca5 thru 0x0305d2 (0xc92e bytes) truncating to largest ever: 0x1afe0 000014 1731750153.969587 trunc from 0x0305d3 to 0x01afdf (0x155f3 bytes) 000015 1731750153.975782 write 0x021b91 thru 0x03070b (0xeb7b bytes) 000016 1731750153.976737 mapread 0x00cce2 thru 0x01a03b (0xd35a bytes) 000017 1731750153.982772 mapwrite 0x01b231 thru 0x02723c (0xc00c bytes) 000018 1731750154.005786 MIRROR_EXTEND to 3 mirrors 000022 1731750154.143260 mapread 0x02d7dc thru 0x03070b (0x2f30 bytes) 000024 1731750154.154092 trunc from 0x03070c to 0x00e602 (0x22109 bytes) 000027 1731750154.175913 read 0x002e49 thru 0x003ace (0xc86 bytes) 000028 1731750154.179374 MIRROR_RESYNC 3 mirrors truncating to largest ever: 0x305ee 000029 1731750154.302455 trunc from 0x00e603 to 0x0305ed (0x21feb bytes) 000032 1731750154.323123 mapwrite 0x01e974 thru 0x01f0dc (0x769 bytes) 000033 1731750154.342691 mapwrite 0x00d52b thru 0x017fa6 (0xaa7c bytes) 000035 1731750154.362219 mapread 0x01532c thru 0x022e0d (0xdae2 bytes) 000036 1731750154.365727 mapread 0x019acc thru 0x01c6f4 (0x2c29 bytes) 000037 1731750154.366201 mapwrite 0x004ae4 thru 0x00fc4d (0xb16a bytes) 000038 1731750154.384320 trunc from 0x0305ee to 0x01ebf5 (0x119f8 bytes) 000039 1731750154.388915 read 0x009c8c thru 0x00feac (0x6221 bytes) 000040 1731750154.389321 mapwrite 0x024522 thru 0x02b00c (0x6aeb bytes) 000041 1731750154.411717 read 0x007015 thru 0x00bc5e (0x4c4a bytes) 000043 1731750154.412457 mapread 0x01e486 thru 0x02b00c (0xcb87 bytes) 000044 1731750154.413088 mapread 0x02591f thru 0x02af56 (0x5638 bytes) 000045 1731750154.413625 write 0x036c70 thru 0x03ffff (0x9390 bytes) 000046 1731750154.414499 trunc from 0x040000 to 0x027377 (0x18c88 bytes) 000048 1731750154.419397 mapread 0x00c32b thru 0x00d583 (0x1259 bytes) 000049 1731750154.419909 mapread 0x02385a thru 0x023929 (0xd0 bytes) 000050 1731750154.420393 write 0x006016 thru 0x014490 (0xe47b bytes) 000053 1731750154.421769 mapread 0x005325 thru 0x00b111 (0x5ded bytes) truncating to largest ever: 0x3daa8 000056 1731750154.422893 trunc from 0x027378 to 0x03daa7 (0x16730 bytes) 000057 1731750154.429634 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x7de:0x0]: Invalid argument (22) 000060 1731750154.493627 mapread 0x005126 thru 0x014d74 (0xfc4f bytes) 000062 1731750154.503080 write 0x016c6c thru 0x01bfd0 (0x5365 bytes) 000063 1731750154.505685 mapread 0x0135d5 thru 0x01b61d (0x8049 bytes) 000064 1731750154.506276 mapwrite 0x02fbe8 thru 0x0344a5 (0x48be bytes) 000066 1731750154.527688 mapwrite 0x0396a1 thru 0x03ffff (0x695f bytes) 000067 1731750154.551819 mapwrite 0x014d1a thru 0x01d6f7 (0x89de bytes) 000068 1731750154.570835 write 0x026e00 thru 0x02d476 (0x6677 bytes) 000072 1731750154.574027 trunc from 0x040000 to 0x0188ea (0x27715 bytes) 000073 1731750154.584607 write 0x002b01 thru 0x00fabe (0xcfbe bytes) 000074 1731750154.590328 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x7de:0x0]: Invalid argument (22) 000076 1731750154.630003 mapread 0x00466f thru 0x014565 (0xfef7 bytes) 000077 1731750154.638896 read 0x00aba1 thru 0x00c1b9 (0x1619 bytes) 000078 1731750154.639516 trunc from 0x0188eb to 0x0331b8 (0x1a8ce bytes) 000080 1731750154.645568 mapwrite 0x0299d9 thru 0x035a00 (0xc028 bytes) 000083 1731750154.677938 read 0x0044e1 thru 0x00ffbd (0xbadd bytes) 000086 1731750154.679317 trunc from 0x035a01 to 0x0032ae (0x32752 bytes) 000089 1731750154.684536 trunc from 0x0032af to 0x00a804 (0x7556 bytes) 000091 1731750154.689187 mapwrite 0x03bd24 thru 0x03ffff (0x42dc bytes) 000093 1731750154.721199 write 0x02e58a thru 0x03740d (0x8e84 bytes) 000094 1731750154.722099 mapwrite 0x02e2cf thru 0x03d14c (0xee7e bytes) 000095 1731750154.739841 trunc from 0x040000 to 0x007d1f (0x382e0 bytes) 000096 1731750154.744275 mapread 0x0009b8 thru 0x001c81 (0x12ca bytes) 000098 1731750154.744951 read 0x007a7d thru 0x007d1f (0x2a3 bytes) 000102 1731750154.746070 write 0x01657e thru 0x024a45 (0xe4c8 bytes) 000103 1731750154.746872 read 0x00c4d8 thru 0x01bd30 (0xf859 bytes) 000106 1731750154.751202 trunc from 0x024a46 to 0x038a69 (0x14024 bytes) 000111 1731750154.757054 write 0x005ae6 thru 0x008717 (0x2c32 bytes) 000112 1731750154.757699 read 0x0216b9 thru 0x02d20f (0xbb57 bytes) 000113 1731750154.761575 mapread 0x032e81 thru 0x033742 (0x8c2 bytes) 000114 1731750154.762068 write 0x012633 thru 0x01f52e (0xcefc bytes) 000115 1731750154.762846 mapread 0x022326 thru 0x029daf (0x7a8a bytes) 000116 1731750154.763407 write 0x039820 thru 0x03ffff (0x67e0 bytes) 000119 1731750154.764659 trunc from 0x040000 to 0x01ae06 (0x251f9 bytes) 000120 1731750154.770495 trunc from 0x01ae07 to 0x013602 (0x7804 bytes) 000122 1731750154.775418 mapread 0x0037b7 thru 0x00d332 (0x9b7c bytes) 000123 1731750154.776051 write 0x006789 thru 0x00a8fe (0x4176 bytes) 000124 1731750154.778411 MIRROR_EXTEND to 2 mirrors 000125 1731750154.906871 write 0x00f003 thru 0x016ec2 (0x7ec0 bytes) 000126 1731750154.922266 mapread 0x013759 thru 0x016ec2 (0x376a bytes) 000127 1731750154.922942 write 0x028c4d thru 0x02f7de (0x6b92 bytes) 000129 1731750154.924117 mapread 0x029655 thru 0x02d1e9 (0x3b95 bytes) 000131 1731750154.925009 mapread 0x0276cd thru 0x02d009 (0x593d bytes) 000133 1731750154.927781 mapwrite 0x00e7f7 thru 0x0110be (0x28c8 bytes) 000135 1731750154.951341 read 0x020f4d thru 0x029e8a (0x8f3e bytes) 000137 1731750154.957219 MIRROR_EXTEND to 3 mirrors 000138 1731750155.094400 write 0x00fee8 thru 0x0167c4 (0x68dd bytes) 000142 1731750155.115339 MIRROR_EXTEND to 4 mirrors 000144 1731750155.248452 trunc from 0x02f7df to 0x00994d (0x25e91 bytes) 000147 1731750155.267137 write 0x035cd2 thru 0x03ffff (0xa32e bytes) 000151 1731750155.269056 trunc from 0x040000 to 0x024c37 (0x1b3c8 bytes) 000152 1731750155.274289 read 0x00baff thru 0x019166 (0xd668 bytes) 000153 1731750155.277417 trunc from 0x024c38 to 0x0278a4 (0x2c6d bytes) 000154 1731750155.281870 mapwrite 0x02235b thru 0x02f6b4 (0xd35a bytes) 000155 1731750155.313500 mapwrite 0x03ae14 thru 0x03ffff (0x51ec bytes) 000156 1731750155.341260 mapwrite 0x01976a thru 0x01cd81 (0x3618 bytes) 000157 1731750155.360122 MIRROR_EXTEND to 5 mirrors 000158 1731750155.486018 trunc from 0x040000 to 0x006709 (0x398f6 bytes) 000160 1731750155.508786 trunc from 0x00670a to 0x020eb7 (0x1a7ae bytes) 000161 1731750155.514019 write 0x00fd4d thru 0x012876 (0x2b2a bytes) 000162 1731750155.517272 write 0x010b72 thru 0x014596 (0x3a25 bytes) 000163 1731750155.520131 write 0x02e451 thru 0x0338c5 (0x5475 bytes) 000167 1731750155.521720 mapread 0x01733b thru 0x01b794 (0x445a bytes) 000168 1731750155.523848 MIRROR_RESYNC 5 mirrors 000169 1731750155.729641 write 0x039650 thru 0x039fe7 (0x998 bytes) 000170 1731750155.745114 write 0x01d8a4 thru 0x022cf7 (0x5454 bytes) 000171 1731750155.749553 MIRROR_EXTEND to 6 mirrors 000172 1731750155.878825 mapread 0x0266bd thru 0x033509 (0xce4d bytes) 000175 1731750155.892026 mapwrite 0x010e9c thru 0x019a63 (0x8bc8 bytes) 000177 1731750155.951929 trunc from 0x039fe8 to 0x010c62 (0x29385 bytes) 000178 1731750155.963181 mapwrite 0x03bd53 thru 0x03ffff (0x42ad bytes) 000180 1731750155.990904 MIRROR_EXTEND to 7 mirrors 000181 1731750156.148994 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x7de:0x0]: Invalid argument (22) 000182 1731750156.252779 trunc from 0x040000 to 0x0255af (0x1aa50 bytes) 000183 1731750156.283809 MIRROR_RESYNC 6 mirrors 000184 1731750156.562161 mapread 0x0134d8 thru 0x01b8b6 (0x83df bytes) 000185 1731750156.568327 write 0x01f83b thru 0x02305a (0x3820 bytes) 000186 1731750156.591414 write 0x037002 thru 0x03ffff (0x8ffe bytes) 000188 1731750156.592450 read 0x03cc14 thru 0x03ffff (0x33ec bytes) 000189 1731750156.595710 write 0x00baa9 thru 0x017362 (0xb8ba bytes) 000194 1731750156.601722 mapwrite 0x027193 thru 0x032f5b (0xbdc9 bytes) 000195 1731750156.639848 MIRROR_EXTEND to 7 mirrors 000197 1731750156.769684 read 0x01ed69 thru 0x026861 (0x7af9 bytes) 000198 1731750156.772948 mapwrite 0x020b94 thru 0x028d20 (0x818d bytes) 000200 1731750156.832631 MIRROR_RESYNC 7 mirrors 000202 1731750157.082381 read 0x034a2b thru 0x03b3ed (0x69c3 bytes) 000204 1731750157.089024 MIRROR_EXTEND to 8 mirrors 000205 1731750157.318137 mapwrite 0x02aa32 thru 0x02cf1d (0x24ec bytes) 000206 1731750157.379525 trunc from 0x040000 to 0x01eb22 (0x214dd bytes) 000207 1731750157.387969 MIRROR_RESYNC 8 mirrors 000208 1731750157.776483 write 0x00a454 thru 0x00c503 (0x20b0 bytes) 000211 1731750157.806443 trunc from 0x01eb23 to 0x0317bc (0x12c9a bytes) 000213 1731750157.813672 read 0x016598 thru 0x017f4b (0x19b4 bytes) 000214 1731750157.817822 MIRROR_EXTEND to 9 mirrors 000219 1731750157.966735 mapread 0x01d60f thru 0x02a8fb (0xd2ed bytes) 000220 1731750157.980247 mapwrite 0x0209b1 thru 0x029c42 (0x9292 bytes) 000221 1731750158.065372 MIRROR_RESYNC 9 mirrors 000222 1731750158.601811 mapread 0x014e5b thru 0x015808 (0x9ae bytes) 000226 1731750158.613645 MIRROR_RESYNC 9 mirrors 000227 1731750158.715832 mapread 0x0168ce thru 0x021b8d (0xb2c0 bytes) 000229 1731750158.729947 write 0x003af4 thru 0x00ff0b (0xc418 bytes) 000230 1731750158.765916 mapwrite 0x014f9c thru 0x01b54f (0x65b4 bytes) 000233 1731750158.803191 write 0x0144c8 thru 0x019caa (0x57e3 bytes) 000234 1731750158.803934 mapwrite 0x00dcb8 thru 0x01462e (0x6977 bytes) 000235 1731750158.831467 mapwrite 0x020f10 thru 0x020f1f (0x10 bytes) 000237 1731750158.858788 read 0x01bcb6 thru 0x01d334 (0x167f bytes) 000238 1731750158.861130 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x7de:0x0]: Invalid argument (22) 000242 1731750158.943216 mapwrite 0x00e2a6 thru 0x01c861 (0xe5bc bytes) 000247 1731750158.983282 read 0x00bfce thru 0x00e673 (0x26a6 bytes) 000249 1731750158.987502 MIRROR_RESYNC 8 mirrors 000250 1731750159.361212 write 0x02a173 thru 0x02fbb7 (0x5a45 bytes) 000253 1731750159.400836 mapread 0x00d0d8 thru 0x01c3e5 (0xf30e bytes) 000254 1731750159.407996 write 0x027b87 thru 0x035d06 (0xe180 bytes) 000255 1731750159.414754 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x7de:0x0]: Invalid argument (22) 000256 1731750159.496990 read 0x00f6b3 thru 0x01b570 (0xbebe bytes) 000260 1731750159.503111 trunc from 0x035d07 to 0x034453 (0x18b3 bytes) 000263 1731750159.511299 mapwrite 0x0058a6 thru 0x012bb1 (0xd30c bytes) 000266 1731750159.553902 MIRROR_EXTEND to 8 mirrors 000268 1731750159.721335 mapwrite 0x031468 thru 0x03d541 (0xc0da bytes) 000272 1731750159.785834 trunc from 0x03d542 to 0x027bab (0x15996 bytes) 000273 1731750159.792574 mapwrite 0x01833e thru 0x0218e0 (0x95a3 bytes) 000276 1731750159.820831 read 0x0233df thru 0x027bab (0x47cd bytes) 000277 1731750159.821223 read 0x018aec thru 0x020c47 (0x815c bytes) 000279 1731750159.821895 mapread 0x0279de thru 0x027bab (0x1ce bytes) 000280 1731750159.822379 trunc from 0x027bac to 0x025e0e (0x1d9d bytes) 000281 1731750159.828529 write 0x0169ae thru 0x01ebe3 (0x8236 bytes) 000282 1731750159.829328 read 0x0221cb thru 0x025c17 (0x3a4d bytes) 000283 1731750159.829745 mapread 0x016f3d thru 0x0172aa (0x36e bytes) 000286 1731750159.830854 read 0x00adb8 thru 0x0109e5 (0x5c2e bytes) 000289 1731750159.835676 mapread 0x01655a thru 0x01be60 (0x5907 bytes) 000290 1731750159.836311 write 0x00c12d thru 0x01812c (0xc000 bytes) 000291 1731750159.837237 read 0x00ab17 thru 0x0136bd (0x8ba7 bytes) 000292 1731750159.837663 read 0x02227e thru 0x025e0e (0x3b91 bytes) 000293 1731750159.838062 mapread 0x0217e8 thru 0x025e0e (0x4627 bytes) 000297 1731750159.839624 trunc from 0x025e0f to 0x01d048 (0x8dc6 bytes) 000298 1731750159.846936 MIRROR_EXTEND to 9 mirrors 000299 1731750159.985207 MIRROR_EXTEND to 10 mirrors 000301 1731750160.124790 mapwrite 0x00759a thru 0x008be0 (0x1647 bytes) 000302 1731750160.181940 mapwrite 0x02c38c thru 0x038016 (0xbc8b bytes) 000303 1731750160.227673 write 0x02aa79 thru 0x02ec45 (0x41cd bytes) 000304 1731750160.230413 MIRROR_EXTEND to 11 mirrors 000305 1731750160.381503 mapread 0x0113a2 thru 0x017e4a (0x6aa9 bytes) 000306 1731750160.389217 mapread 0x00247f thru 0x00414a (0x1ccc bytes) 000307 1731750160.395499 MIRROR_EXTEND to 12 mirrors 000310 1731750160.548077 MIRROR_RESYNC 12 mirrors 000315 1731750160.932026 mapwrite 0x01af90 thru 0x0294c4 (0xe535 bytes) 000317 1731750161.002255 write 0x00bef2 thru 0x00c4d5 (0x5e4 bytes) 000318 1731750161.005663 mapread 0x02df50 thru 0x038016 (0xa0c7 bytes) 000319 1731750161.010812 trunc from 0x038017 to 0x019fa3 (0x1e073 bytes) 000321 1731750161.024741 read 0x0152b7 thru 0x019fa3 (0x4ced bytes) 000322 1731750161.025185 trunc from 0x019fa4 to 0x0238c2 (0x991f bytes) 000323 1731750161.030716 mapwrite 0x01133c thru 0x012800 (0x14c5 bytes) 000324 1731750161.052951 MIRROR_RESYNC 12 mirrors 000325 1731750161.486370 mapread 0x021c42 thru 0x0238c2 (0x1c81 bytes) 000326 1731750161.489596 trunc from 0x0238c3 to 0x039a89 (0x161c7 bytes) 000328 1731750161.512578 mapread 0x0029ae thru 0x00e269 (0xb8bc bytes) 000330 1731750161.518433 mapwrite 0x020e7b thru 0x021396 (0x51c bytes) 000332 1731750161.536985 MIRROR_EXTEND to 13 mirrors 000334 1731750161.672989 trunc from 0x039a8a to 0x025fdd (0x13aac bytes) 000335 1731750161.699607 MIRROR_RESYNC 13 mirrors 000337 1731750162.174050 mapwrite 0x039e80 thru 0x03ffff (0x6180 bytes) 000338 1731750162.229601 read 0x002a4d thru 0x008da8 (0x635c bytes) 000339 1731750162.232451 mapwrite 0x00e246 thru 0x018b6f (0xa92a bytes) 000340 1731750162.254864 trunc from 0x040000 to 0x030cb6 (0xf349 bytes) 000342 1731750162.262028 write 0x03be5e thru 0x03ffff (0x41a2 bytes) 000343 1731750162.264806 MIRROR_RESYNC 13 mirrors 000345 1731750162.668270 read 0x03df00 thru 0x03ffff (0x2100 bytes) 000346 1731750162.670582 mapwrite 0x01e34a thru 0x026241 (0x7ef8 bytes) 000347 1731750162.733763 trunc from 0x040000 to 0x0015d2 (0x3ea2d bytes) 000348 1731750162.748136 read 0x000e92 thru 0x0015d2 (0x741 bytes) 000350 1731750162.750551 mapread 0x001396 thru 0x0015d2 (0x23d bytes) 000351 1731750162.751046 read 0x0007bd thru 0x0015d2 (0xe16 bytes) 000352 1731750162.751457 mapread 0x0008a8 thru 0x0015d2 (0xd2b bytes) 000353 1731750162.751936 mapread 0x00135a thru 0x0015d2 (0x279 bytes) 000356 1731750162.752908 read 0x0010f5 thru 0x0015d2 (0x4de bytes) 000357 1731750162.753256 read 0x000ac0 thru 0x0015d2 (0xb13 bytes) 000359 1731750162.753827 write 0x011bfd thru 0x01ffce (0xe3d2 bytes) 000361 1731750162.755169 write 0x01bb94 thru 0x01c596 (0xa03 bytes) 000362 1731750162.755734 write 0x03967f thru 0x03ffff (0x6981 bytes) 000365 1731750162.757109 write 0x0077c5 thru 0x0081dc (0xa18 bytes) 000366 1731750162.764419 MIRROR_SPLIT mirror 9 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x7de:0x0]: Invalid argument (22) 000367 1731750162.829311 MIRROR_SPLIT mirror 6 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x7de:0x0]: Invalid argument (22) 000368 1731750162.892871 read 0x00d23d thru 0x0171d4 (0x9f98 bytes) 000370 1731750162.897184 mapwrite 0x02c636 thru 0x03c34c (0xfd17 bytes) 000371 1731750162.934129 mapread 0x031625 thru 0x03c180 (0xab5c bytes) 000372 1731750162.934776 mapwrite 0x0087f1 thru 0x00f610 (0x6e20 bytes) 000373 1731750162.956454 mapread 0x0187fa thru 0x023371 (0xab78 bytes) 000374 1731750162.961221 read 0x03c71d thru 0x03ffff (0x38e3 bytes) 000375 1731750162.961653 mapwrite 0x01121d thru 0x017d99 (0x6b7d bytes) 000376 1731750162.975426 write 0x00142a thru 0x00ff69 (0xeb40 bytes) 000379 1731750162.977069 trunc from 0x040000 to 0x0362cd (0x9d32 bytes) 000380 1731750162.985199 MIRROR_RESYNC 11 mirrors 000381 1731750163.382605 read 0x01daf9 thru 0x022cbb (0x51c3 bytes) 000383 1731750163.385808 write 0x01b76b thru 0x023e4d (0x86e3 bytes) 000386 1731750163.408553 write 0x010bd2 thru 0x01b81a (0xac49 bytes) 000388 1731750163.412437 trunc from 0x0362ce to 0x00b06d (0x2b260 bytes) 000389 1731750163.420445 trunc from 0x00b06e to 0x014743 (0x96d6 bytes) 000390 1731750163.425106 mapread 0x00d53c thru 0x013ce8 (0x67ad bytes) 000391 1731750163.428517 trunc from 0x014744 to 0x00c04f (0x86f4 bytes) 000393 1731750163.433384 write 0x021439 thru 0x02903d (0x7c05 bytes) 000394 1731750163.434326 read 0x01999a thru 0x01d0b5 (0x371c bytes) 000395 1731750163.438060 trunc from 0x02903e to 0x0001bf (0x28e7e bytes) 000396 1731750163.443579 mapwrite 0x014cbc thru 0x016d68 (0x20ad bytes) 000398 1731750163.470695 mapwrite 0x01822d thru 0x026cdc (0xeab0 bytes) 000401 1731750163.513994 mapwrite 0x025b8b thru 0x029af2 (0x3f68 bytes) 000402 1731750163.547220 mapwrite 0x0067e4 thru 0x008d4a (0x2567 bytes) 000405 1731750163.569259 mapread 0x00e3b8 thru 0x015581 (0x71ca bytes) 000407 1731750163.578587 write 0x008d80 thru 0x0184a8 (0xf729 bytes) 000410 1731750163.581728 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x7de:0x0]: Invalid argument (22) 000411 1731750163.639266 read 0x01cde0 thru 0x022653 (0x5874 bytes) 000413 1731750163.642966 write 0x0235b2 thru 0x02454e (0xf9d bytes) 000415 1731750163.646994 read 0x007520 thru 0x007bc8 (0x6a9 bytes) 000416 1731750163.651606 read 0x00a5d9 thru 0x0135eb (0x9013 bytes) 000417 1731750163.652068 mapread 0x00cdd5 thru 0x01be62 (0xf08e bytes) 000419 1731750163.655044 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x7de:0x0]: Invalid argument (22) 000421 1731750163.715951 mapwrite 0x0295cd thru 0x0320f0 (0x8b24 bytes) 000422 1731750163.751406 MIRROR_RESYNC 9 mirrors 000423 1731750164.086387 mapread 0x0111fe thru 0x020e34 (0xfc37 bytes) 000424 1731750164.100003 read 0x0110fd thru 0x015e01 (0x4d05 bytes) 000425 1731750164.103044 MIRROR_RESYNC 9 mirrors 000426 1731750164.178595 write 0x00c1c7 thru 0x015dee (0x9c28 bytes) 000427 1731750164.212316 MIRROR_EXTEND to 10 mirrors 000428 1731750164.401878 mapwrite 0x009e9a thru 0x016f7f (0xd0e6 bytes) 000430 1731750164.473028 write 0x0078da thru 0x015f10 (0xe637 bytes) 000433 1731750164.474772 read 0x02db42 thru 0x0320f0 (0x45af bytes) 000436 1731750164.479002 read 0x018bc8 thru 0x01be40 (0x3279 bytes) 000437 1731750164.479504 mapwrite 0x015502 thru 0x0214d9 (0xbfd8 bytes) 000441 1731750164.512139 mapread 0x01fc59 thru 0x026f0c (0x72b4 bytes) 000442 1731750164.512806 mapread 0x00cfb3 thru 0x0103bc (0x340a bytes) 000445 1731750164.513998 write 0x00f648 thru 0x014ee4 (0x589d bytes) 000446 1731750164.517110 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x7de:0x0]: Invalid argument (22) 000449 1731750164.599322 read 0x00c6ab thru 0x01a11d (0xda73 bytes) 000450 1731750164.603227 trunc from 0x0320f1 to 0x03224e (0x15e bytes) 000451 1731750164.609456 mapwrite 0x02cfec thru 0x037549 (0xa55e bytes) 000452 1731750164.669888 MIRROR_RESYNC 9 mirrors 000453 1731750165.165776 trunc from 0x03754a to 0x016b70 (0x209d9 bytes) 000454 1731750165.190440 mapwrite 0x015835 thru 0x020728 (0xaef4 bytes) 000455 1731750165.242334 mapread 0x01eeee thru 0x020728 (0x183b bytes) 000456 1731750165.242860 mapwrite 0x037f8e thru 0x03f631 (0x76a4 bytes) 000460 1731750165.292550 MIRROR_EXTEND to 10 mirrors 000461 1731750165.490672 mapwrite 0x004087 thru 0x006cf4 (0x2c6e bytes) 000462 1731750165.573308 read 0x017e96 thru 0x0269f9 (0xeb64 bytes) 000464 1731750165.580006 MIRROR_EXTEND to 11 mirrors 000465 1731750165.742762 write 0x010214 thru 0x01a8d4 (0xa6c1 bytes) 000466 1731750165.770094 trunc from 0x03f632 to 0x0162b7 (0x2937a bytes) 000467 1731750165.781314 trunc from 0x0162b8 to 0x023940 (0xd689 bytes) 000469 1731750165.786255 read 0x015f59 thru 0x01c01f (0x60c7 bytes) 000470 1731750165.789928 mapread 0x01add0 thru 0x01b9a2 (0xbd3 bytes) 000473 1731750165.792786 MIRROR_EXTEND to 12 mirrors 000474 1731750165.958393 read 0x00cdc2 thru 0x013001 (0x6240 bytes) 000476 1731750165.965920 read 0x005550 thru 0x0113a3 (0xbe54 bytes) 000479 1731750165.975288 mapwrite 0x028c6e thru 0x0336e5 (0xaa78 bytes) 000481 1731750166.049681 read 0x01f774 thru 0x02f185 (0xfa12 bytes) 000482 1731750166.055926 trunc from 0x0336e6 to 0x019e5c (0x19889 bytes) 000484 1731750166.066409 mapwrite 0x01c412 thru 0x02138a (0x4f79 bytes) 000485 1731750166.106950 write 0x018c15 thru 0x026bf5 (0xdfe1 bytes) 000487 1731750166.108385 mapread 0x00a492 thru 0x00fe4f (0x59be bytes) 000488 1731750166.111191 MIRROR_RESYNC 12 mirrors 000489 1731750166.598233 MIRROR_EXTEND to 13 mirrors 000491 1731750166.799662 read 0x01395e thru 0x01fa5a (0xc0fd bytes) 000493 1731750166.805400 mapread 0x00ea97 thru 0x0183b8 (0x9922 bytes) 000494 1731750166.812915 mapread 0x012314 thru 0x01ef96 (0xcc83 bytes) 000495 1731750166.815626 mapread 0x00f76d thru 0x01217f (0x2a13 bytes) 000496 1731750166.818175 write 0x023b2b thru 0x031823 (0xdcf9 bytes) 000498 1731750166.840979 trunc from 0x031824 to 0x02197d (0xfea6 bytes) 000501 1731750166.850676 mapwrite 0x02d987 thru 0x036ed5 (0x954f bytes) 000503 1731750166.878298 mapwrite 0x00dba2 thru 0x01b27c (0xd6db bytes) 000504 1731750166.902692 write 0x0100cd thru 0x0129fe (0x2932 bytes) 000505 1731750166.903310 mapread 0x01b0fe thru 0x02428f (0x9192 bytes) 000506 1731750166.905968 read 0x018266 thru 0x02291e (0xa6b9 bytes) 000508 1731750166.906711 mapread 0x017e26 thru 0x018a07 (0xbe2 bytes) 000509 1731750166.907245 trunc from 0x036ed6 to 0x031cec (0x51e9 bytes) 000510 1731750166.912538 write 0x006a31 thru 0x0109b3 (0x9f83 bytes) 000511 1731750166.915139 MIRROR_EXTEND to 14 mirrors 000512 1731750167.042383 MIRROR_RESYNC 14 mirrors 000513 1731750167.601549 read 0x0317f7 thru 0x031cec (0x4f6 bytes) 000514 1731750167.601853 trunc from 0x031ced to 0x031ec4 (0x1d8 bytes) 000516 1731750167.634546 write 0x03a0b1 thru 0x03ffff (0x5f4f bytes) 000517 1731750167.635308 mapread 0x039c36 thru 0x03ffff (0x63ca bytes) 000518 1731750167.641315 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x7de:0x0]: Invalid argument (22) 000519 1731750167.712428 MIRROR_SPLIT mirror 3 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x7de:0x0]: Invalid argument (22) 000520 1731750167.788649 mapread 0x024816 thru 0x033693 (0xee7e bytes) 000525 1731750167.800594 write 0x038a7b thru 0x03ffff (0x7585 bytes) 000526 1731750167.801309 mapwrite 0x000053 thru 0x009106 (0x90b4 bytes) 000527 1731750167.827195 mapwrite 0x005971 thru 0x00d1f4 (0x7884 bytes) 000530 1731750167.845741 mapwrite 0x001cc0 thru 0x00bf1f (0xa260 bytes) 000531 1731750167.863947 trunc from 0x040000 to 0x0388ec (0x7713 bytes) 000533 1731750167.869897 mapwrite 0x03c516 thru 0x03ffff (0x3aea bytes) 000535 1731750167.897981 trunc from 0x040000 to 0x014b5e (0x2b4a1 bytes) 000536 1731750167.904766 write 0x039ab1 thru 0x03c8aa (0x2dfa bytes) 000537 1731750167.905514 mapwrite 0x037b01 thru 0x03e8f8 (0x6df8 bytes) 000539 1731750167.940595 MIRROR_EXTEND to 13 mirrors 000540 1731750168.103465 write 0x01789f thru 0x01acfe (0x3460 bytes) 000542 1731750168.130932 mapread 0x0256a6 thru 0x0323a0 (0xccfb bytes) 000543 1731750168.138255 write 0x0251b5 thru 0x034ae9 (0xf935 bytes) 000544 1731750168.139198 trunc from 0x03e8f9 to 0x013777 (0x2b181 bytes) 000545 1731750168.147693 write 0x0170d5 thru 0x020477 (0x93a3 bytes) 000548 1731750168.149699 mapread 0x00701e thru 0x015b4f (0xeb32 bytes) 000549 1731750168.156903 mapread 0x01fad9 thru 0x020477 (0x99f bytes) 000550 1731750168.162392 MIRROR_EXTEND to 14 mirrors 000551 1731750168.295519 write 0x007e99 thru 0x017b34 (0xfc9c bytes) 000553 1731750168.317487 trunc from 0x020478 to 0x013ebf (0xc5b8 bytes) 000554 1731750168.327704 read 0x00c2c8 thru 0x013ebf (0x7bf8 bytes) 000555 1731750168.329531 MIRROR_SPLIT mirror 2 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x7de:0x0]: Invalid argument (22) 000556 1731750168.387910 MIRROR_RESYNC 13 mirrors 000557 1731750168.807729 trunc from 0x013ec0 to 0x016aec (0x2c2d bytes) 000558 1731750168.835180 mapwrite 0x00e856 thru 0x01bdfe (0xd5a9 bytes) 000559 1731750168.882083 MIRROR_EXTEND to 14 mirrors 000560 1731750169.007164 MIRROR_EXTEND to 15 mirrors 000561 1731750169.150595 write 0x00d83f thru 0x0175fd (0x9dbf bytes) 000565 1731750169.173758 write 0x0369c2 thru 0x03b136 (0x4775 bytes) 000566 1731750169.174555 trunc from 0x03b137 to 0x011f74 (0x291c2 bytes) 000567 1731750169.185168 write 0x019991 thru 0x01cd7e (0x33ee bytes) 000568 1731750169.186064 read 0x01c1b5 thru 0x01cd7e (0xbca bytes) 000569 1731750169.186417 trunc from 0x01cd7f to 0x008fd7 (0x13da7 bytes) 000570 1731750169.190566 write 0x0176c0 thru 0x027032 (0xf973 bytes) 000571 1731750169.191637 trunc from 0x027033 to 0x01d332 (0x9d00 bytes) 000572 1731750169.195803 write 0x034d08 thru 0x036967 (0x1c60 bytes) 000573 1731750169.196504 write 0x01dd02 thru 0x02387a (0x5b79 bytes) 000575 1731750169.198844 trunc from 0x036968 to 0x0148a0 (0x220c7 bytes) 000578 1731750169.205423 MIRROR_RESYNC 15 mirrors 000580 1731750169.716179 read 0x011f02 thru 0x012ba1 (0xca0 bytes) 000581 1731750169.717876 mapwrite 0x03eb46 thru 0x03ffff (0x14ba bytes) 000582 1731750169.769230 MIRROR_RESYNC 15 mirrors 000583 1731750170.297982 trunc from 0x040000 to 0x016e20 (0x291df bytes) 000584 1731750170.323568 mapwrite 0x033dbe thru 0x03ffff (0xc242 bytes) 000585 1731750170.382199 trunc from 0x040000 to 0x02adfa (0x15205 bytes) 000586 1731750170.390640 trunc from 0x02adfb to 0x000483 (0x2a977 bytes) 000588 1731750170.397140 write 0x01f8f1 thru 0x0208d8 (0xfe8 bytes) 000590 1731750170.398166 trunc from 0x0208d9 to 0x019a52 (0x6e86 bytes) 000592 1731750170.406872 MIRROR_EXTEND to 16 mirrors 000593 1731750170.567678 MIRROR_RESYNC 16 mirrors 000594 1731750171.096597 trunc from 0x019a53 to 0x027c1e (0xe1cc bytes) 000595 1731750171.118982 trunc from 0x027c1f to 0x009daf (0x1de6f bytes) 000596 1731750171.124667 mapwrite 0x01c812 thru 0x029458 (0xcc47 bytes) 000597 1731750171.158730 trunc from 0x029459 to 0x000008 (0x29450 bytes) 000598 1731750171.163380 write 0x03e5f3 thru 0x03ffff (0x1a0d bytes) 000601 1731750171.164794 read 0x02ad78 thru 0x031ccc (0x6f55 bytes) 000603 1731750171.168458 write 0x03a789 thru 0x03ffff (0x5877 bytes) 000604 1731750171.172081 mapwrite 0x03c7b6 thru 0x03ffff (0x384a bytes) 000605 1731750171.194859 MIRROR_SPLIT mirror 5 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x7de:0x0]: Invalid argument (22) 000606 1731750171.255487 MIRROR_EXTEND to 16 mirrors 000607 1731750171.425511 mapread 0x0126d8 thru 0x020c03 (0xe52c bytes) 000609 1731750171.441009 read 0x017a9d thru 0x01e4bf (0x6a23 bytes) 000610 1731750171.442947 mapwrite 0x0062e1 thru 0x00764b (0x136b bytes) 000612 1731750171.514005 mapwrite 0x028ac4 thru 0x02e284 (0x57c1 bytes) 000613 1731750171.540656 read 0x026cb4 thru 0x0361ab (0xf4f8 bytes) 000614 1731750171.547221 read 0x028692 thru 0x033110 (0xaa7f bytes) 000616 1731750171.547961 read 0x038220 thru 0x03ffff (0x7de0 bytes) 000622 1731750171.549941 trunc from 0x040000 to 0x00ca2d (0x335d2 bytes) 000623 1731750171.565468 mapread 0x0093b1 thru 0x00ca2d (0x367d bytes) 000625 1731750171.566536 write 0x00aa69 thru 0x010245 (0x57dd bytes) 000626 1731750171.567357 mapwrite 0x026aba thru 0x032016 (0xb55d bytes) 000627 1731750171.611242 write 0x00cb78 thru 0x019062 (0xc4eb bytes) 000628 1731750171.612209 write 0x035bbc thru 0x03f533 (0x9978 bytes) 000629 1731750171.613111 mapread 0x00721f thru 0x010267 (0x9049 bytes) 000631 1731750171.614049 mapread 0x02ac13 thru 0x03939c (0xe78a bytes) 000632 1731750171.617037 write 0x01a090 thru 0x01d647 (0x35b8 bytes) 000634 1731750171.618863 write 0x037462 thru 0x03ffff (0x8b9e bytes) 000635 1731750171.621485 MIRROR_RESYNC 16 mirrors 000637 1731750172.043204 read 0x0065a3 thru 0x00c2ad (0x5d0b bytes) 000638 1731750172.046518 mapread 0x01579e thru 0x01868a (0x2eed bytes) 000639 1731750172.054200 read 0x025ee9 thru 0x026f70 (0x1088 bytes) 000641 1731750172.058914 trunc from 0x040000 to 0x00dd05 (0x322fa bytes) 000643 1731750172.086700 mapwrite 0x0312e8 thru 0x037a42 (0x675b bytes) 000644 1731750172.123358 mapwrite 0x02c6a6 thru 0x02da62 (0x13bd bytes) 000645 1731750172.146402 mapwrite 0x00e1fa thru 0x013c8b (0x5a92 bytes) 000646 1731750172.180391 MIRROR_SPLIT mirror 15 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x7de:0x0]: Invalid argument (22) 000649 1731750172.250077 trunc from 0x037a43 to 0x011865 (0x261dd bytes) 000650 1731750172.255530 trunc from 0x011866 to 0x02e002 (0x1c79d bytes) 000653 1731750172.261637 trunc from 0x02e003 to 0x02d850 (0x7b2 bytes) 000655 1731750172.267507 mapwrite 0x03af5f thru 0x03bf3f (0xfe1 bytes) 000657 1731750172.294743 read 0x007d68 thru 0x00f807 (0x7aa0 bytes) 000660 1731750172.300720 MIRROR_RESYNC 15 mirrors 000661 1731750172.768652 write 0x005ecd thru 0x00d534 (0x7668 bytes) 000664 1731750172.792979 trunc from 0x03bf40 to 0x038f73 (0x2fcc bytes) 000668 1731750172.808683 mapread 0x024633 thru 0x0294ad (0x4e7b bytes) 000670 1731750172.812854 mapwrite 0x02edf1 thru 0x0351eb (0x63fb bytes) 000672 1731750172.834546 mapread 0x029dfe thru 0x03875a (0xe95d bytes) 000674 1731750172.835540 mapwrite 0x02e326 thru 0x03a743 (0xc41e bytes) 000676 1731750172.859760 read 0x012c69 thru 0x016d04 (0x409c bytes) 000680 1731750172.863853 trunc from 0x03a744 to 0x02625b (0x144e8 bytes) 000681 1731750172.868700 trunc from 0x02625c to 0x000444 (0x25e17 bytes) 000683 1731750172.874113 read 0x00003c thru 0x000444 (0x409 bytes) 000685 1731750172.877004 mapwrite 0x0086c1 thru 0x00e47e (0x5dbe bytes) 000689 1731750172.907410 mapread 0x008742 thru 0x00e47e (0x5d3d bytes) 000694 1731750172.909348 mapwrite 0x019dde thru 0x01c1fd (0x2420 bytes) 000695 1731750172.938637 mapread 0x010ec8 thru 0x018505 (0x763e bytes) 000697 1731750172.939595 write 0x029bf7 thru 0x032e65 (0x926f bytes) 000699 1731750172.940786 mapwrite 0x0153d0 thru 0x021e82 (0xcab3 bytes) 000700 1731750172.964770 trunc from 0x032e66 to 0x00eac2 (0x243a3 bytes) 000703 1731750172.970694 mapwrite 0x02a4cb thru 0x0313eb (0x6f21 bytes) 000705 1731750173.012033 write 0x00f187 thru 0x0160c0 (0x6f3a bytes) 000706 1731750173.019127 MIRROR_RESYNC 15 mirrors 000707 1731750173.534464 mapread 0x010e63 thru 0x016a4d (0x5beb bytes) 000708 1731750173.540828 read 0x00a520 thru 0x00b0ed (0xbce bytes) 000714 1731750173.543532 mapread 0x006ef2 thru 0x00947d (0x258c bytes) 000716 1731750173.546847 mapread 0x0148b0 thru 0x0174be (0x2c0f bytes) 000718 1731750173.548872 MIRROR_SPLIT mirror 2 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x7de:0x0]: Invalid argument (22) 000721 1731750173.673340 mapread 0x00c88c thru 0x018b78 (0xc2ed bytes) 000725 1731750173.682908 mapread 0x011fdb thru 0x017267 (0x528d bytes) 000727 1731750173.683607 read 0x003772 thru 0x00fc12 (0xc4a1 bytes) 000728 1731750173.685941 mapwrite 0x029965 thru 0x02b7a4 (0x1e40 bytes) 000729 1731750173.741810 MIRROR_EXTEND to 15 mirrors 000730 1731750173.887433 write 0x03317a thru 0x03ffff (0xce86 bytes) 000731 1731750173.907346 read 0x0238cb thru 0x028a6f (0x51a5 bytes) 000733 1731750173.910517 mapread 0x00208b thru 0x00a0d9 (0x804f bytes) 000736 1731750173.916299 write 0x03f74b thru 0x03ffff (0x8b5 bytes) 000737 1731750173.918222 MIRROR_EXTEND to 16 mirrors 000738 1731750174.094598 mapread 0x02ce78 thru 0x033023 (0x61ac bytes) 000739 1731750174.106796 trunc from 0x040000 to 0x029df2 (0x1620d bytes) 000740 1731750174.128632 write 0x03071e thru 0x037d33 (0x7616 bytes) 000741 1731750174.129460 trunc from 0x037d34 to 0x03d91c (0x5be9 bytes) 000743 1731750174.137388 mapread 0x023087 thru 0x029269 (0x61e3 bytes) 000750 1731750174.143991 mapwrite 0x012933 thru 0x013ccb (0x1399 bytes) 000752 1731750174.169135 read 0x000982 thru 0x010073 (0xf6f2 bytes) 000753 1731750174.172186 trunc from 0x03d91d to 0x03bfc6 (0x1956 bytes) 000754 1731750174.177011 mapread 0x00e812 thru 0x01cbeb (0xe3da bytes) 000756 1731750174.181043 read 0x027974 thru 0x0354d4 (0xdb61 bytes) 000757 1731750174.181475 mapwrite 0x030edb thru 0x036ce9 (0x5e0f bytes) 000758 1731750174.199742 mapread 0x02a775 thru 0x033545 (0x8dd1 bytes) 000760 1731750174.200764 read 0x037d9b thru 0x0395e4 (0x184a bytes) 000761 1731750174.203895 read 0x00a340 thru 0x00a3b5 (0x76 bytes) 000764 1731750174.206503 MIRROR_RESYNC 16 mirrors 000767 1731750174.756420 trunc from 0x03bfc7 to 0x00dc9c (0x2e32a bytes) 000769 1731750174.779973 mapwrite 0x022588 thru 0x0323b9 (0xfe32 bytes) 000771 1731750174.816969 mapread 0x01a976 thru 0x026d54 (0xc3df bytes) 000773 1731750174.821167 mapwrite 0x006ad5 thru 0x013775 (0xcca1 bytes) 000774 1731750174.852635 read 0x02efd8 thru 0x0323b9 (0x33e2 bytes) 000775 1731750174.852907 write 0x00839f thru 0x011f1e (0x9b80 bytes) 000777 1731750174.853931 trunc from 0x0323ba to 0x03470c (0x2353 bytes) 000779 1731750174.860902 mapwrite 0x018004 thru 0x01a22b (0x2228 bytes) 000780 1731750174.883635 read 0x00d849 thru 0x01b050 (0xd808 bytes) 000781 1731750174.884097 mapread 0x03218b thru 0x03470c (0x2582 bytes) 000782 1731750174.887678 mapread 0x021c93 thru 0x029dee (0x815c bytes) 000783 1731750174.888477 write 0x018f73 thru 0x0206e3 (0x7771 bytes) 000785 1731750174.891439 MIRROR_SPLIT mirror 1 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x7de:0x0]: Invalid argument (22) 000786 1731750174.965051 MIRROR_EXTEND to 16 mirrors 000787 1731750175.134729 MIRROR_RESYNC 16 mirrors 000788 1731750175.676847 MIRROR_SPLIT mirror 1 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x7de:0x0]: Invalid argument (22) 000789 1731750175.814718 trunc from 0x03470d to 0x019ce0 (0x1aa2c bytes) 000790 1731750175.840465 read 0x012828 thru 0x017914 (0x50ed bytes) 000792 1731750175.844483 read 0x00b055 thru 0x016f25 (0xbed1 bytes) 000795 1731750175.849296 write 0x036da1 thru 0x03ffff (0x925f bytes) 000796 1731750175.852841 MIRROR_RESYNC 15 mirrors 000797 1731750176.338747 mapread 0x035754 thru 0x03c15b (0x6a08 bytes) 000798 1731750176.344675 mapread 0x00ec44 thru 0x012e18 (0x41d5 bytes) 000799 1731750176.347515 mapread 0x026f7e thru 0x0336ee (0xc771 bytes) 000801 1731750176.352376 read 0x00ec4a thru 0x0116dc (0x2a93 bytes) 000802 1731750176.352631 mapread 0x006f8c thru 0x014d5a (0xddcf bytes) 000805 1731750176.356060 write 0x00f2f9 thru 0x012164 (0x2e6c bytes) 000807 1731750176.383698 write 0x008d2c thru 0x01754a (0xe81f bytes) 000811 1731750176.386066 read 0x01025d thru 0x014ebe (0x4c62 bytes) 000813 1731750176.386772 mapread 0x00630f thru 0x015ce8 (0xf9da bytes) 000814 1731750176.392481 trunc from 0x040000 to 0x032dae (0xd251 bytes) 000815 1731750176.403184 mapread 0x023389 thru 0x02a6fd (0x7375 bytes) 000817 1731750176.404284 mapwrite 0x029b4d thru 0x035663 (0xbb17 bytes) 000818 1731750176.440909 MIRROR_EXTEND to 16 mirrors 000819 1731750176.608330 mapwrite 0x00e819 thru 0x01d3f1 (0xebd9 bytes) 000821 1731750176.702065 write 0x009139 thru 0x00fff6 (0x6ebe bytes) 000822 1731750176.703195 mapwrite 0x00235d thru 0x00aef1 (0x8b95 bytes) 000825 1731750176.730846 mapwrite 0x025a3c thru 0x029af1 (0x40b6 bytes) 000827 1731750176.753307 mapread 0x004a42 thru 0x00947a (0x4a39 bytes) 000828 1731750176.753895 mapwrite 0x0083f2 thru 0x015e0e (0xda1d bytes) 000830 1731750176.784241 mapwrite 0x02103e thru 0x02fd94 (0xed57 bytes) 000831 1731750176.818759 read 0x00a5d4 thru 0x015085 (0xaab2 bytes) 000832 1731750176.819342 write 0x0036ea thru 0x00443b (0xd52 bytes) 000833 1731750176.820256 read 0x018dca thru 0x01e74d (0x5984 bytes) 000836 1731750176.821458 write 0x02ee31 thru 0x031884 (0x2a54 bytes) 000839 1731750176.823069 read 0x02c5f9 thru 0x035663 (0x906b bytes) 000840 1731750176.823537 trunc from 0x035664 to 0x005720 (0x2ff43 bytes) 000842 1731750176.851525 MIRROR_SPLIT mirror 10 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x7de:0x0]: Invalid argument (22) 000843 1731750176.930183 write 0x023353 thru 0x02fa37 (0xc6e5 bytes) 000844 1731750176.931289 trunc from 0x02fa38 to 0x025d82 (0x9cb5 bytes) 000845 1731750176.936944 read 0x0120a0 thru 0x01f221 (0xd182 bytes) 000846 1731750176.941383 read 0x0203c5 thru 0x02162e (0x126a bytes) 000847 1731750176.945596 read 0x02086a thru 0x025d82 (0x5519 bytes) 000849 1731750176.946386 trunc from 0x025d83 to 0x017a32 (0xe350 bytes) 000850 1731750176.953378 write 0x00318a thru 0x01109d (0xdf14 bytes) 000851 1731750176.956729 MIRROR_EXTEND to 16 mirrors 000853 1731750177.206665 mapread 0x0153d9 thru 0x017a32 (0x265a bytes) 000855 1731750177.220333 write 0x01c604 thru 0x021ef8 (0x58f5 bytes) 000858 1731750177.244163 trunc from 0x021ef9 to 0x01ac77 (0x7281 bytes) 000859 1731750177.252895 read 0x00e3ba thru 0x0100e9 (0x1d30 bytes) 000862 1731750177.256986 read 0x01548f thru 0x01ac77 (0x57e9 bytes) 000863 1731750177.262682 read 0x018ec5 thru 0x01ac77 (0x1db3 bytes) 000864 1731750177.263284 read 0x01a531 thru 0x01ac77 (0x747 bytes) 000865 1731750177.264181 write 0x00398b thru 0x00d4b5 (0x9b2b bytes) 000866 1731750177.264883 mapread 0x016d3a thru 0x01ac77 (0x3f3e bytes) 000870 1731750177.266490 read 0x00ef7c thru 0x01ac77 (0xbcfc bytes) 000871 1731750177.266920 read 0x00ca5b thru 0x016194 (0x973a bytes) 000872 1731750177.269640 MIRROR_SPLIT mirror 7 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x7de:0x0]: Invalid argument (22) 000873 1731750177.377637 read 0x01055e thru 0x012d72 (0x2815 bytes) 000875 1731750177.380827 read 0x00d2f3 thru 0x0158c4 (0x85d2 bytes) 000876 1731750177.389012 read 0x015d8c thru 0x01ac77 (0x4eec bytes) 000877 1731750177.389501 read 0x016e9f thru 0x01ac77 (0x3dd9 bytes) 000878 1731750177.393381 MIRROR_EXTEND to 16 mirrors 000879 1731750177.596356 MIRROR_RESYNC 16 mirrors 000881 1731750178.266669 mapwrite 0x030771 thru 0x035de7 (0x5677 bytes) 000882 1731750178.327700 mapwrite 0x02b772 thru 0x02f479 (0x3d08 bytes) 000883 1731750178.351328 mapwrite 0x0193ba thru 0x01e818 (0x545f bytes) 000885 1731750178.374214 read 0x02cc09 thru 0x02e677 (0x1a6f bytes) 000886 1731750178.374705 trunc from 0x035de8 to 0x009050 (0x2cd97 bytes) 000887 1731750178.381505 read 0x000940 thru 0x009050 (0x8711 bytes) 000888 1731750178.384825 read 0x001822 thru 0x002c09 (0x13e8 bytes) 000890 1731750178.385563 mapwrite 0x00b196 thru 0x0154de (0xa349 bytes) 000895 1731750178.417004 mapwrite 0x00ab63 thru 0x0121ee (0x768c bytes) 000897 1731750178.433085 mapwrite 0x00410e thru 0x007d7f (0x3c72 bytes) 000898 1731750178.445790 read 0x00caf6 thru 0x0154de (0x89e9 bytes) 000899 1731750178.446243 write 0x006776 thru 0x007b5b (0x13e6 bytes) 000900 1731750178.446763 mapread 0x00a59e thru 0x011d3d (0x77a0 bytes) 000902 1731750178.447622 read 0x006e6a thru 0x00b870 (0x4a07 bytes) 000903 1731750178.447957 write 0x036bae thru 0x03ffff (0x9452 bytes) 000905 1731750178.449019 write 0x004777 thru 0x012bd3 (0xe45d bytes) 000907 1731750178.452242 MIRROR_RESYNC 16 mirrors 000909 1731750178.920907 trunc from 0x040000 to 0x010494 (0x2fb6b bytes) 000911 1731750178.957956 mapread 0x00262d thru 0x005ae0 (0x34b4 bytes) 000913 1731750178.965225 mapread 0x005090 thru 0x008527 (0x3498 bytes) 000915 1731750178.966123 read 0x000fea thru 0x007638 (0x664f bytes) 000916 1731750178.968894 trunc from 0x010495 to 0x024c7c (0x147e8 bytes) 000917 1731750178.977074 mapread 0x01c4c2 thru 0x024444 (0x7f83 bytes) 000918 1731750178.985208 MIRROR_SPLIT mirror 1 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x7de:0x0]: Invalid argument (22) 000920 1731750179.062724 mapread 0x0092cf thru 0x0148ad (0xb5df bytes) 000925 1731750179.082270 MIRROR_SPLIT mirror 10 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x7de:0x0]: Invalid argument (22) 000926 1731750179.152485 trunc from 0x024c7d to 0x02751f (0x28a3 bytes) 000930 1731750179.160702 MIRROR_RESYNC 14 mirrors 000931 1731750179.732376 write 0x001fa1 thru 0x005609 (0x3669 bytes) 000933 1731750179.757999 mapwrite 0x033242 thru 0x0344c5 (0x1284 bytes) 000935 1731750179.791279 mapread 0x0168f6 thru 0x01a7c4 (0x3ecf bytes) 000936 1731750179.797096 MIRROR_RESYNC 14 mirrors 000938 1731750180.444651 write 0x00039e thru 0x00f67e (0xf2e1 bytes) 000940 1731750180.468991 read 0x00b7ff thru 0x01723d (0xba3f bytes) 000941 1731750180.472058 read 0x017fc0 thru 0x021862 (0x98a3 bytes) 000942 1731750180.475419 mapwrite 0x022200 thru 0x02239c (0x19d bytes) 000944 1731750180.490688 mapread 0x01e18c thru 0x024113 (0x5f88 bytes) 000949 1731750180.492756 read 0x00a891 thru 0x01355d (0x8ccd bytes) 000950 1731750180.493248 write 0x0005a8 thru 0x007dc6 (0x781f bytes) 000951 1731750180.494059 write 0x0363dd thru 0x03ffff (0x9c23 bytes) 000952 1731750180.496788 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x7de:0x0]: Invalid argument (22) 000953 1731750180.576557 mapread 0x022f34 thru 0x026de9 (0x3eb6 bytes) 000954 1731750180.584001 mapwrite 0x01dd5f thru 0x01ddd6 (0x78 bytes) 000955 1731750180.604651 MIRROR_SPLIT mirror 3 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x7de:0x0]: Invalid argument (22) 000956 1731750180.690583 read 0x03ef6d thru 0x03ffff (0x1093 bytes) 000957 1731750180.694551 mapread 0x00f8bb thru 0x011cbe (0x2404 bytes) 000959 1731750180.702795 trunc from 0x040000 to 0x03d355 (0x2caa bytes) 000960 1731750180.710911 MIRROR_RESYNC 12 mirrors 000961 1731750181.165489 trunc from 0x03d356 to 0x033e80 (0x94d5 bytes) 000962 1731750181.186359 mapwrite 0x015ac6 thru 0x021329 (0xb864 bytes) 000963 1731750181.216544 trunc from 0x033e81 to 0x001c70 (0x32210 bytes) 000964 1731750181.227248 MIRROR_EXTEND to 13 mirrors 000966 1731750181.380163 MIRROR_SPLIT mirror 7 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x7de:0x0]: Invalid argument (22) 000967 1731750181.469615 mapread 0x00134a thru 0x001c70 (0x927 bytes) 000968 1731750181.487689 MIRROR_EXTEND to 13 mirrors 000969 1731750181.635311 read 0x00006a thru 0x001c70 (0x1c07 bytes) 000970 1731750181.638681 write 0x0231a2 thru 0x031a01 (0xe860 bytes) 000973 1731750181.657372 mapread 0x010a1c thru 0x011ae0 (0x10c5 bytes) 000975 1731750181.661038 read 0x01400b thru 0x019ffd (0x5ff3 bytes) 000976 1731750181.664827 read 0x028423 thru 0x028c3e (0x81c bytes) 000977 1731750181.665318 read 0x01fbf4 thru 0x02709a (0x74a7 bytes) 000978 1731750181.665709 trunc from 0x031a02 to 0x013bca (0x1de37 bytes) truncating to largest ever: 0x3ed55 000979 1731750181.673081 trunc from 0x013bcb to 0x03ed54 (0x2b18a bytes) 000980 1731750181.678864 read 0x0319e5 thru 0x039a82 (0x809e bytes) 000981 1731750181.684338 write 0x020b4c thru 0x027712 (0x6bc7 bytes) 000982 1731750181.685113 mapwrite 0x030874 thru 0x03f166 (0xe8f3 bytes) 000984 1731750181.718644 trunc from 0x03f167 to 0x022676 (0x1caf0 bytes) 000986 1731750181.723738 write 0x008ea9 thru 0x01662a (0xd782 bytes) 000987 1731750181.726202 MIRROR_EXTEND to 14 mirrors 000988 1731750181.879820 read 0x019551 thru 0x01b1a7 (0x1c57 bytes) 000990 1731750181.883977 trunc from 0x022677 to 0x033847 (0x111d1 bytes) 000992 1731750181.910643 write 0x031e7d thru 0x033ae3 (0x1c67 bytes) 000993 1731750181.914414 mapread 0x028618 thru 0x033ae3 (0xb4cc bytes) 000994 1731750181.923596 MIRROR_RESYNC 14 mirrors 000995 1731750182.433576 mapwrite 0x02c424 thru 0x034cf9 (0x88d6 bytes) 000996 1731750182.490051 mapread 0x0116f1 thru 0x01f4ef (0xddff bytes) 000997 1731750182.497335 read 0x02127a thru 0x0278e4 (0x666b bytes) 000998 1731750182.497959 read 0x031f19 thru 0x034cf9 (0x2de1 bytes) 001000 1731750182.498756 mapread 0x0151a5 thru 0x018e28 (0x3c84 bytes) All operations completed A-OK! PASS 70a (30s) == sanity-flr test 200a: stress test ===================== 04:43:03 (1731750183) Starting client: oleg242-client.virtnet: -o user_xattr,flock 192.168.202.142@tcp:/lustre /mnt/lustre2 Starting client: oleg242-client.virtnet: -o user_xattr,flock 192.168.202.142@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1185088 .. Extending file size to 8798016 .. done Extending file size to 8838880 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 3603968 .. Extending file size to 8451968 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8640640 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8714848 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 9216256 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8945664 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8987712 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9283552 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 31370 31371 31373 31374 31377 192.168.202.142@tcp:/lustre /mnt/lustre2 lustre rw,checksum,csumfix,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg242-client.virtnet /mnt/lustre2 (opts:) 192.168.202.142@tcp:/lustre /mnt/lustre3 lustre rw,checksum,csumfix,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg242-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 04:44:07 (1731750247) Starting client: oleg242-client.virtnet: -o user_xattr,flock 192.168.202.142@tcp:/lustre /mnt/lustre2 Starting client: oleg242-client.virtnet: -o user_xattr,flock 192.168.202.142@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4442048 .. Extending file size to 1977504 .. done Extending file size to 4851296 .. Extending file size to 2726272 .. Extending file size to 5645312 .. Extending file size to 7176096 .. Extending file size to 8201568 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7720544 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8344288 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8753536 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8676352 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9263488 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8809184 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9297952 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 8886016 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8940480 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 1701 1702 1703 1705 1708 192.168.202.142@tcp:/lustre /mnt/lustre2 lustre rw,checksum,csumfix,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg242-client.virtnet /mnt/lustre2 (opts:) 192.168.202.142@tcp:/lustre /mnt/lustre3 lustre rw,checksum,csumfix,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg242-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 04:45:11 (1731750311) Starting client: oleg242-client.virtnet: -o user_xattr,flock 192.168.202.142@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4992 2203392 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4992 2203392 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0340751 s, 61.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0847937 s, 24.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.142@tcp:/lustre /mnt/lustre2 lustre rw,checksum,csumfix,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg242-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 04:45:15 (1731750315) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0230511 s, 91.0 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 04:45:17 (1731750317) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x82b:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x94a:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x94a:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x94b:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x945:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 04:45:19 (1731750319) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0310505 s, 67.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x94c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x946:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x947:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 04:45:21 (1731750321) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0309596 s, 67.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x94d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x94e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x948:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x94d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x94e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x948:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x949:0x0] } PASS 204b (1s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 04:45:23 (1731750323) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0300817 s, 69.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x94f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x94a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x94f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x94a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x94b:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 04:45:45 (1731750345) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg242-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0258291 s, 40.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x950:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x94d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x94c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x950:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x94d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x94c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x951:0x0] } PASS 204d (41s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 04:46:26 (1731750386) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg242-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x94e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x94f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0326917 s, 64.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x94e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x94f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x952:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x94e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x94f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x952:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (40s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 04:47:06 (1731750426) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg242-server mds-ost sync done. Creating new pool oleg242-server: Pool lustre.test_204f created Adding targets to pool oleg242-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg242-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.027233 s, 77.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x950:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x953:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x950:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x953:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg242-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg242-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg242-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (54s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 04:48:00 (1731750480) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 04:48:02 (1731750482) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 04:48:04 (1731750484) oleg242-server: Pool lustre.test_206 created Adding targets to pool oleg242-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg242-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg242-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg242-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg242-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (14s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 04:48:18 (1731750498) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0184288 s, 56.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00876936 s, 120 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x3f48b3a0 Mirror 3: 0x3f48b3a0 PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 04:48:21 (1731750501) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 04:48:21 (1731750501) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 04:48:22 (1731750502) Starting client: oleg242-client.virtnet: -o user_xattr,flock 192.168.202.142@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0203705 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1330000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x9c0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x9c0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x9c0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x22b0000 Read this (4 bytes): BBBB 192.168.202.142@tcp:/lustre /mnt/lustre2 lustre rw,checksum,csumfix,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg242-client.virtnet /mnt/lustre2 (opts:) PASS 209a (14s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 04:48:36 (1731750516) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00279793 s, 2.9 MB/s the very first read read with warm cache PASS 209b (1s) == sanity-flr test 210a: handle broken mirrored lovea ==== 04:48:37 (1731750517) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00947118 s, 111 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x850 lmm_fid: [0x200000404:0x850:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x850 lmm_fid: [0x200000404:0x850:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x95d:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322836048 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-11-16 04:48:37.000000000 -0500 Modify: 2024-11-16 04:48:37.000000000 -0500 Change: 2024-11-16 04:48:38.000000000 -0500 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 04:48:39 (1731750519) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test complete, duration 738 sec ============ 04:48:40 (1731750520) === sanity-flr: start cleanup 04:48:40 (1731750520) === === sanity-flr: finish cleanup 04:48:42 (1731750522) ===