-----============= acceptance-small: recovery-small ============----- Wed Nov 13 12:08:47 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 136 === recovery-small: start setup 12:08:51 (1731517731) === oleg124-client.virtnet: executing check_config_client /mnt/lustre oleg124-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg124-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b58f0800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b58f0800.idle_timeout=debug disable quota as required oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all osd-ldiskfs.track_declares_assert=1 === recovery-small: finish setup 12:09:00 (1731517740) === == recovery-small test 1: create, chmod, stat: drop req, drop rep ========================================================== 12:09:00 (1731517740) fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x122 fail_loc=0 PASS 1 (101s) == recovery-small test 4: open: drop req, drop rep ======= 12:10:41 (1731517841) fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x122 fail_loc=0 PASS 4 (34s) == recovery-small test 5: rename: drop req, drop rep ===== 12:11:15 (1731517875) fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 PASS 5 (35s) == recovery-small test 6: link, unlink: drop req, drop rep ========================================================== 12:11:50 (1731517910) fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 PASS 6 (67s) == recovery-small test 8: touch: drop rep (bug 1423) ===== 12:12:57 (1731517977) fail_loc=0x119 fail_loc=0 PASS 8 (18s) == recovery-small test 9: pause bulk on OST (bug 1420) === 12:13:15 (1731517995) timeout is 0/ fail_val=0 fail_loc=0x80000214 fail_loc=0 PASS 9 (7s) == recovery-small test 10a: finish request on server after client eviction (bug 1521) ========================================================== 12:13:22 (1731518002) ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b58f0800.lru_max_age=3900s ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b58f0800.lru_max_age=3900s console_ratelimit=0 ldlm.namespaces.MGC192.168.201.124@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b58f0800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b58f0800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8800b58f0800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8800b58f0800.early_lock_cancel=0 fail_loc=0x305 fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.201.124@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b58f0800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b58f0800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8800b58f0800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8800b58f0800.early_lock_cancel=1 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4096: /mnt/lustre/recon: Input/output error Connected clients: oleg124-client.virtnet /mnt/lustre has perms 0777 OK [ 413.291213] Lustre: ll_ost00_004: service thread pid 11034 was inactive for 40.062 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [ 413.291065] Lustre: ll_ost00_002: service thread pid 9552 was inactive for 40.062 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 409.963120] Lustre: mdt00_002: service thread pid 7208 was inactive for 40.060 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 481.994338] Lustre: mdt00_002: service thread pid 7208 completed after 112.091s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b58f0800.lru_max_age=600000 ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b58f0800.lru_max_age=600000 console_ratelimit=1 PASS 10a (115s) == recovery-small test 10b: re-send BL AST =============== 12:15:17 (1731518117) ldlm.namespaces.MGC192.168.201.124@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b58f0800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b58f0800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8800b58f0800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8800b58f0800.early_lock_cancel=0 fail_loc=0x80000305 fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.201.124@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b58f0800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b58f0800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8800b58f0800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8800b58f0800.early_lock_cancel=1 Connected clients: oleg124-client.virtnet oleg124-client.virtnet /mnt/lustre has perms 0777 OK PASS 10b (17s) == recovery-small test 10c: re-send BL AST vs reconnect race (LU-5569) ========================================================== 12:15:34 (1731518134) ldlm.namespaces.MGC192.168.201.124@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b58f0800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b58f0800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8800b58f0800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8800b58f0800.early_lock_cancel=0 fail_loc=0x80000305 mdc.lustre-MDT0001-mdc-ffff8800b58f0800.import=connection=192.168.201.124@tcp fail_loc=0 fail_val=0 ldlm.namespaces.MGC192.168.201.124@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b58f0800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b58f0800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8800b58f0800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8800b58f0800.early_lock_cancel=1 Connected clients: oleg124-client.virtnet oleg124-client.virtnet /mnt/lustre/d10c.recovery-small has perms 0777 OK PASS 10c (3s) == recovery-small test 10d: test failed blocking ast ===== 12:15:37 (1731518137) 7+0 records in 7+0 records out 7 bytes (7 B) copied, 0.00013398 s, 52.2 kB/s Stopping client oleg124-client.virtnet /mnt/lustre (opts:)