-----============= acceptance-small: recovery-small ============----- Wed May 8 18:02:09 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 136 === recovery-small: start setup 18:02:13 (1715205733) === oleg318-client.virtnet: executing check_config_client /mnt/lustre oleg318-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg318-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012a5a2800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012a5a2800.idle_timeout=debug disable quota as required oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all === recovery-small: finish setup 18:02:19 (1715205739) === debug_raw_pointers=Y debug_raw_pointers=Y == recovery-small test 1: create, chmod, stat: drop req, drop rep ========================================================== 18:02:20 (1715205740) fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x122 fail_loc=0 PASS 1 (101s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == recovery-small test 4: open: drop req, drop rep ======= 18:04:02 (1715205842) fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x122 fail_loc=0 PASS 4 (35s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == recovery-small test 5: rename: drop req, drop rep ===== 18:04:38 (1715205878) fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 PASS 5 (35s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == recovery-small test 6: link, unlink: drop req, drop rep ========================================================== 18:05:14 (1715205914) fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 PASS 6 (67s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == recovery-small test 8: touch: drop rep (bug 1423) ===== 18:06:23 (1715205983) fail_loc=0x119 fail_loc=0 PASS 8 (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == recovery-small test 9: pause bulk on OST (bug 1420) === 18:06:42 (1715206002) timeout is 0/ fail_val=0 fail_loc=0x80000214 fail_loc=0 PASS 9 (8s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == recovery-small test 10a: finish request on server after client eviction (bug 1521) ========================================================== 18:06:51 (1715206011) ldlm.namespaces.MGC192.168.203.118@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff88012a5a2800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff88012a5a2800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff88012a5a2800.early_lock_cancel=0 fail_loc=0x305 fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.203.118@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff88012a5a2800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff88012a5a2800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff88012a5a2800.early_lock_cancel=1 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4018: /mnt/lustre/recon: Input/output error Connected clients: oleg318-client.virtnet /mnt/lustre has perms 0777 OK PASS 10a (114s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == recovery-small test 10b: re-send BL AST =============== 18:08:46 (1715206126) ldlm.namespaces.MGC192.168.203.118@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff88012a5a2800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff88012a5a2800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff88012a5a2800.early_lock_cancel=0 fail_loc=0x80000305 fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.203.118@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff88012a5a2800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff88012a5a2800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff88012a5a2800.early_lock_cancel=1 uname: write error: Input/output error uname: write error: Input/output error cat: /mnt/lustre/recon: Cannot send after transport endpoint shutdown Connected clients: ls: cannot access /mnt/lustre/recon: Cannot send after transport endpoint shutdown /mnt/lustre has perms 0777 OK PASS 10b (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == recovery-small test 10c: re-send BL AST vs reconnect race (LU-5569) ========================================================== 18:09:06 (1715206146) ldlm.namespaces.MGC192.168.203.118@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff88012a5a2800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff88012a5a2800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff88012a5a2800.early_lock_cancel=0 fail_loc=0x80000305 mdc.lustre-MDT0000-mdc-ffff88012a5a2800.import=connection=192.168.203.118@tcp fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.203.118@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff88012a5a2800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff88012a5a2800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff88012a5a2800.early_lock_cancel=1 Connected clients: oleg318-client.virtnet oleg318-client.virtnet /mnt/lustre/d10c.recovery-small has perms 0777 OK PASS 10c (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == recovery-small test 10d: test failed blocking ast ===== 18:09:10 (1715206150) 7+0 records in 7+0 records out 7 bytes (7 B) copied, 0.000118423 s, 59.1 kB/s Stopping client oleg318-client.virtnet /mnt/lustre (opts:)