[ 1921.179262] LustreError: lustre-OST0000-osc-ffff9f95a0747800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1921.192595] Lustre: 2353:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.159@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1930.030311] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 22:41:18 (1788576078)
[ 1932.250935] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1933.716498] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 22:41:21 (1788576081)
[ 1935.636586] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1936.979918] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 22:41:25 (1788576085)
[ 1960.446459] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 192.168.206.159@tcp) was lost; in progress operations using this service will fail
[ 1960.510767] Lustre: Evicted from MGS (at 192.168.206.159@tcp) after server handle changed from 0xf2c1376926ab0dcd to 0xf2c1376926ab254a
[ 1961.795532] Lustre: 15935:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.159@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2066.891536] LustreError: lustre-MDT0000-mdc-ffff9f95a0747800: operation mds_close to node 192.168.206.159@tcp failed: rc = -19
[ 2066.905638] LustreError: Skipped 3 previous similar messages
[ 2087.399988] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 192.168.206.159@tcp) was lost; in progress operations using this service will fail
[ 2087.421573] Lustre: Evicted from MGS (at 192.168.206.159@tcp) after server handle changed from 0xf2c1376926ab254a to 0xf2c1376926acdd5c
[ 2088.754576] Lustre: 15935:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.159@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2115.230050] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 22:44:22 (1788576262)
[ 2115.716773] Lustre: *** cfs_fail_loc=305, val=0***
[ 2115.723124] Lustre: Skipped 2 previous similar messages
[ 2135.180122] LustreError: lustre-OST0000-osc-ffff9f95a0747800: operation ost_connect to node 192.168.206.159@tcp failed: rc = -75
[ 2158.049799] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 192.168.206.159@tcp) was lost; in progress operations using this service will fail
[ 2168.305378] Lustre: lustre-MDT0000-mdc-ffff9f95a0747800: Connection to lustre-MDT0000 (at 192.168.206.159@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2168.305452] Lustre: Evicted from MGS (at 192.168.206.159@tcp) after server handle changed from 0xf2c1376926acdd5c to 0xf2c1376926ace5c1
[ 2168.314254] Lustre: Skipped 11 previous similar messages
[ 2168.332679] Lustre: MGC192.168.206.159@tcp: Connection restored to 192.168.206.159@tcp (at 192.168.206.159@tcp)
[ 2168.332687] Lustre: Skipped 13 previous similar messages
[ 2182.243155] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2183.501988] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2191.392506] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 22:45:39 (1788576339)
[ 2204.150563] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 192.168.206.159@tcp) was lost; in progress operations using this service will fail
[ 2204.203109] Lustre: Evicted from MGS (at 192.168.206.159@tcp) after server handle changed from 0xf2c1376926ace5c1 to 0xf2c1376926ace9d4
[ 2204.296401] LustreError: lustre-MDT0000-mdc-ffff9f95a0747800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2228.198298] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 22:46:16 (1788576376)
[ 2257.711559] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 22:46:45 (1788576405)
[ 2271.168930] LustreError: lustre-MDT0000-mdc-ffff9f95a0747800: operation mds_reint to node 192.168.206.159@tcp failed: rc = -19
[ 2291.176895] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 192.168.206.159@tcp) was lost; in progress operations using this service will fail
[ 2291.201429] Lustre: Evicted from MGS (at 192.168.206.159@tcp) after server handle changed from 0xf2c1376926ace9d4 to 0xf2c1376926ad4273
[ 2308.829420] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2310.630680] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2394.592900] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788576527/real 1788576527]  req@ffff9f95840f0700 x1875455996909568/t0(0) o400->MGC192.168.206.159@tcp@192.168.206.159@tcp:26/25 lens 224/224 e 0 to 1 dl 1788576543 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2394.614946] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
[ 2394.625752] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 192.168.206.159@tcp) was lost; in progress operations using this service will fail
[ 2403.820317] Lustre: Evicted from MGS (at 192.168.206.159@tcp) after server handle changed from 0xf2c1376926ad4273 to 0xf2c1376926af4ecb
[ 2419.564675] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2421.359770] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2505.954655] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 192.168.206.159@tcp) was lost; in progress operations using this service will fail
[ 2516.135036] Lustre: Evicted from MGS (at 192.168.206.159@tcp) after server handle changed from 0xf2c1376926af4ecb to 0xf2c1376926b13d47
[ 2534.927578] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2537.193675] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2566.539600] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 22:51:54 (1788576714)
[ 2570.960475] LustreError: lustre-MDT0000-mdc-ffff9f95a0747800: operation ldlm_enqueue to node 192.168.206.159@tcp failed: rc = -19
[ 2570.970379] LustreError: Skipped 4 previous similar messages
[ 2587.040220] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 192.168.206.159@tcp) was lost; in progress operations using this service will fail
[ 2597.228801] Lustre: Evicted from MGS (at 192.168.206.159@tcp) after server handle changed from 0xf2c1376926b13d47 to 0xf2c1376926b21b72
[ 2598.382540] Lustre: 15935:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.159@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2609.469921] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2633.521436] Lustre: 15935:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.159@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2648.460180] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2691.583247] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2723.310395] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 192.168.206.159@tcp) was lost; in progress operations using this service will fail
[ 2723.324363] LustreError: Skipped 2 previous similar messages
[ 2734.700584] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2783.890930] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2785.713483] Lustre: Evicted from MGS (at 192.168.206.159@tcp) after server handle changed from 0xf2c1376926b2a550 to 0xf2c1376926b3609e
[ 2785.723244] Lustre: Skipped 3 previous similar messages
[ 2785.728919] Lustre: MGC192.168.206.159@tcp: Connection restored to 192.168.206.159@tcp (at 192.168.206.159@tcp)
[ 2785.744245] Lustre: Skipped 17 previous similar messages
[ 2812.263854] Lustre: lustre-MDT0000-mdc-ffff9f95a0747800: Connection to lustre-MDT0000 (at 192.168.206.159@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2812.278856] Lustre: Skipped 10 previous similar messages
[ 2839.118568] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2893.873747] Lustre: 15935:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.159@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2933.889173] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 22:58:01 (1788577081)
[ 2988.756770] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2990.687395] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3278.897965] LustreError: lustre-OST0000-osc-ffff9f95a0747800: operation ldlm_enqueue to node 192.168.206.159@tcp failed: rc = -19
[ 3278.908556] LustreError: Skipped 13 previous similar messages
[ 3321.232647] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3323.941482] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3610.501848] Lustre: lustre-OST0000-osc-ffff9f95a0747800: Connection to lustre-OST0000 (at 192.168.206.159@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3610.530044] Lustre: Skipped 3 previous similar messages
[ 3653.001819] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3655.551096] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3906.039673] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 23:14:13 (1788578053)
[ 3922.912283] Lustre: 50564:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788578056/real 1788578056]  req@ffff9f9586f68e00 x1875456019880576/t0(0) o101->lustre-MDT0000-mdc-ffff9f95a0747800@192.168.206.159@tcp:12/10 lens 576/1152 e 0 to 1 dl 1788578072 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3922.967341] Lustre: 50564:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 3923.030702] Lustre: lustre-MDT0000-mdc-ffff9f95a0747800: Connection restored to 192.168.206.159@tcp (at 192.168.206.159@tcp)
[ 3923.043735] Lustre: Skipped 5 previous similar messages
[ 3930.707985] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 23:14:38 (1788578078)
[ 3955.007890] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 23:15:03 (1788578103)
[ 3979.971511] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 23:15:27 (1788578127)
[ 3980.371192] Lustre: Mounted lustre-client - version 2.17.58_3_g4a08d7b
[ 4010.784259] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788578144/real 1788578144]  req@ffff9f95840a5c00 x1875456019899136/t0(0) o400->MGC192.168.206.159@tcp@192.168.206.159@tcp:26/25 lens 224/224 e 0 to 1 dl 1788578160 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4010.804342] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4010.808767] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 192.168.206.159@tcp) was lost; in progress operations using this service will fail
[ 4010.817699] LustreError: Skipped 3 previous similar messages
[ 4021.227435] Lustre: Evicted from MGS (at 192.168.206.159@tcp) after server handle changed from 0xf2c1376926b4772b to 0xf2c1376926ca4940
[ 4021.239144] Lustre: Skipped 2 previous similar messages
[ 4026.848129] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4028.413195] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4030.406878] Lustre: Unmounted lustre-client
[ 4036.074993] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 23:16:24 (1788578184)
[ 4172.768586] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788578305/real 1788578305]  req@ffff9f9586f6b480 x1875456019931648/t0(0) o4->lustre-OST0000-osc-ffff9f95a0747800@192.168.206.159@tcp:6/4 lens 488/448 e 0 to 1 dl 1788578321 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4172.790959] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 71 previous similar messages
[ 4220.896600] Lustre: lustre-OST0000-osc-ffff9f95a0747800: Connection to lustre-OST0000 (at 192.168.206.159@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4220.919797] Lustre: Skipped 15 previous similar messages
[ 4330.664754] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 23:21:18 (1788578478)
[ 4379.340947] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 23:22:07 (1788578527)
[ 4382.720177] Lustre: Unmounted lustre-client
[ 4403.432207] Lustre: Mounted lustre-client - version 2.17.58_3_g4a08d7b
[ 4408.862561] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 23:22:36 (1788578556)
[ 4409.153676] LustreError: 56488:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4410.211016] LustreError: 56488:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4410.349560] Lustre: *** cfs_fail_loc=305, val=0***
[ 4435.589414] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 23:23:02 (1788578582)
[ 4436.519470] Lustre: Mounted lustre-client - version 2.17.58_3_g4a08d7b
[ 4440.910207] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4451.235580] Lustre: Unmounted lustre-client
[ 4457.586551] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 23:23:25 (1788578605)
[ 4536.335450] LustreError: lustre-MDT0000-mdc-ffff9f958415b800: operation mds_reint to node 192.168.206.159@tcp failed: rc = -19
[ 4536.358822] LustreError: Skipped 1 previous similar message
[ 4555.247633] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 192.168.206.159@tcp) was lost; in progress operations using this service will fail
[ 4555.278476] Lustre: Evicted from MGS (at 192.168.206.159@tcp) after server handle changed from 0xf2c1376926ca6436 to 0xf2c1376926cd1991
[ 4555.291544] Lustre: MGC192.168.206.159@tcp: Connection restored to 192.168.206.159@tcp (at 192.168.206.159@tcp)
[ 4555.314855] Lustre: Skipped 23 previous similar messages
[ 4686.708613] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 23:27:14 (1788578834)
[ 4690.523458] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4703.717864] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 192.168.206.159@tcp) was lost; in progress operations using this service will fail
[ 4703.738614] Lustre: Evicted from MGS (at 192.168.206.159@tcp) after server handle changed from 0xf2c1376926cd1991 to 0xf2c1376926d1e347
[ 4713.082905] LustreError: lustre-MDT0000-mdc-ffff9f958415b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4721.020394] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 23:27:49 (1788578869)
[ 4721.388342] Lustre: Mounted lustre-client - version 2.17.58_3_g4a08d7b
[ 4728.614217] LustreError: lustre-OST0000-osc-ffff9f958415b800: operation ldlm_enqueue to node 192.168.206.159@tcp failed: rc = -107
[ 4728.627541] LustreError: lustre-OST0000-osc-ffff9f958415b800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4738.016239] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788578871/real 1788578871]  req@ffff9f958432aa00 x1875456024586112/t0(0) o103->lustre-OST0000-osc-ffff9f95a070f000@192.168.206.159@tcp:17/18 lens 328/224 e 0 to 1 dl 1788578887 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4738.050822] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 80 previous similar messages
[ 4740.060800] Lustre: Unmounted lustre-client
[ 4746.015478] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 23:28:14 (1788578894)
[ 4753.101098] LustreError: lustre-MDT0000-mdc-ffff9f958415b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4753.113454] LustreError: 60868:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4761.792910] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 23:28:29 (1788578909)
[ 4761.986272] LustreError: 61567:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4765.960580] LustreError: lustre-MDT0000-mdc-ffff9f958415b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4765.981780] LustreError: 61589:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4765.990410] LustreError: 61567:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1004
[ 4766.003870] LustreError: 61567:0:(import.c:761:ptlrpc_connect_import_locked()) already connecting
[ 4766.060467] LustreError: 61594:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4767.176924] LustreError: 61600:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4767.194294] LustreError: 61600:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4769.588585] LustreError: 61623:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4769.601102] LustreError: 61623:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4774.295258] LustreError: 61668:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4774.307803] LustreError: 61668:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4783.613167] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 23:28:51 (1788578931)
[ 4819.299654] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4820.595971] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4828.794274] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 23:29:36 (1788578976)
[ 4832.233898] Lustre: lustre-OST0000-osc-ffff9f958415b800: Connection to lustre-OST0000 (at 192.168.206.159@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4832.243151] Lustre: Skipped 17 previous similar messages
[ 4863.114547] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4864.245080] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4873.125353] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 23:30:21 (1788579021)
[ 4931.964834] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4933.279708] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4940.486085] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 23:31:28 (1788579088)
[ 4973.699760] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4975.286814] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4981.645332] Lustre: Unmounted lustre-client
[ 5031.308854] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5033.905093] Lustre: Mounted lustre-client - version 2.17.58_3_g4a08d7b
[ 5040.572745] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 23:33:08 (1788579188)
[ 5043.271579] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5044.954679] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 23:33:12 (1788579192)
[ 5075.880344] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 23:33:43 (1788579223)
[ 5077.354256] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5079.261894] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 23:33:46 (1788579226)
[ 5080.730613] Lustre: *** cfs_fail_loc=805, val=0***
[ 5080.765235] Lustre: Mounted lustre-client - version 2.17.58_3_g4a08d7b
[ 5086.001177] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5107.680398] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 192.168.206.159@tcp) was lost; in progress operations using this service will fail
[ 5107.715686] Lustre: Evicted from MGS (at 192.168.206.159@tcp) after server handle changed from 0xf2c1376926d1eddc to 0xf2c1376926d1f00c
[ 5116.634527] Lustre: Unmounted lustre-client
[ 5124.367087] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 23:34:32 (1788579272)
[ 5151.283661] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff9f9585faf480 x1875456024667008/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff9f95a130a000@192.168.206.159@tcp:12/10 lens 648/608 e 0 to 0 dl 1788579316 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5161.119661] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5163.294832] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5171.624963] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 23:35:19 (1788579319)
[ 5173.077633] LustreError: lustre-OST0000-osc-ffff9f95a130a000: operation ost_write to node 192.168.206.159@tcp failed: rc = -107
[ 5173.089901] LustreError: Skipped 2 previous similar messages
[ 5173.124480] LustreError: lustre-OST0000-osc-ffff9f95a130a000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5173.148901] Lustre: 2355:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.159@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5173.184287] LustreError: 72765:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9f95a130a000: namespace resource [0x240000400:0x4102:0x0].0x0 (ffff9f9587164100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5182.134376] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 23:35:29 (1788579329)
[ 5183.813578] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5185.506424] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 23:35:33 (1788579333)
[ 5187.334577] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5188.983732] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 23:35:36 (1788579336)
[ 5190.712540] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5192.570150] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 23:35:40 (1788579340)
[ 5193.872788] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5195.698075] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 23:35:43 (1788579343)
[ 5197.081484] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5199.495654] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 23:35:46 (1788579346)
[ 5201.212508] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5203.378477] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 23:35:51 (1788579351)
[ 5204.756595] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5206.389172] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 23:35:54 (1788579354)
[ 5207.953166] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5209.426790] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 23:35:57 (1788579357)
[ 5211.324780] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5213.051763] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 23:36:00 (1788579360)
[ 5214.407955] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5216.070338] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 23:36:03 (1788579363)
[ 5217.612673] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5219.218393] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 23:36:07 (1788579367)
[ 5221.599321] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5222.996222] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 23:36:11 (1788579371)
[ 5243.387666] Lustre: MGC192.168.206.159@tcp: Connection restored to 192.168.206.159@tcp (at 192.168.206.159@tcp)
[ 5243.397276] Lustre: Skipped 17 previous similar messages
[ 5252.582268] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 23:36:40 (1788579400)
[ 5281.290128] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 23:37:09 (1788579429)
[ 5281.895396] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5292.315641] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 23:37:20 (1788579440)
[ 5292.687520] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5302.844301] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 23:37:30 (1788579450)
[ 5303.360449] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5311.521517] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 23:37:39 (1788579459)
[ 5311.712510] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5321.090798] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 23:37:48 (1788579468)
[ 5321.479198] Lustre: *** cfs_fail_loc=510, val=3***
[ 5330.196330] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 23:37:57 (1788579477)
[ 5330.525543] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5341.056614] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 23:38:08 (1788579488)
[ 5341.335222] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5341.600198] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788579474/real 1788579474]  req@ffff9f9584329880 x1875456024706688/t0(0) o400->lustre-OST0000-osc-ffff9f95a130a000@192.168.206.159@tcp:28/4 lens 224/224 e 0 to 1 dl 1788579490 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5341.614678] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 5407.527206] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 23:39:15 (1788579555)
[ 5407.987954] LustreError: 82973:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5410.783296] LustreError: lustre-MDT0000-mdc-ffff9f95a130a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5410.800934] LustreError: 82987:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9f95a130a000: inode [0x20000a042:0x10:0x0] mdc close failed: rc = -108
[ 5410.831965] LustreError: 82987:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9f95a130a000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9f9587164f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5412.064499] LustreError: 82973:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5414.163818] LustreError: 83003:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5417.053245] LustreError: lustre-MDT0000-mdc-ffff9f95a130a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5417.079020] LustreError: 83017:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9f95a130a000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9f9587164400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5417.114606] LustreError: 83017:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5418.240175] LustreError: 83003:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5424.971937] LustreError: lustre-MDT0000-mdc-ffff9f95a130a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5425.310233] LustreError: 83059:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5428.271028] LustreError: lustre-MDT0000-mdc-ffff9f95a130a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5428.364765] LustreError: 83078:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5428.380204] LustreError: 83078:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5429.376200] LustreError: 83059:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5429.391508] LustreError: 83059:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9f95a130a000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5429.480886] LustreError: 83085:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5429.493809] LustreError: 83085:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5431.797474] LustreError: 83107:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5431.805658] LustreError: 83107:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5435.366470] LustreError: 83134:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5438.285449] Lustre: lustre-MDT0000-mdc-ffff9f95a130a000: Connection to lustre-MDT0000 (at 192.168.206.159@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5438.309961] Lustre: Skipped 13 previous similar messages
[ 5438.333096] LustreError: lustre-MDT0000-mdc-ffff9f95a130a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5438.361656] LustreError: 83148:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9f95a130a000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9f9587096800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5438.413207] LustreError: 83148:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5439.448657] LustreError: 83134:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5446.383091] LustreError: lustre-MDT0000-mdc-ffff9f95a130a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5446.778574] LustreError: 83190:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5449.686785] LustreError: lustre-MDT0000-mdc-ffff9f95a130a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5449.708504] LustreError: 83205:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9f95a130a000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9f9587096d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5449.746246] LustreError: 83205:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5449.768820] LustreError: 83210:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5449.784384] LustreError: 83210:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5450.840384] LustreError: 83190:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5453.748208] LustreError: lustre-MDT0000-mdc-ffff9f95a130a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5454.993350] LustreError: 83218:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9f95a130a000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5462.793973] LustreError: 83289:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5462.803347] LustreError: 83289:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5466.437301] LustreError: lustre-MDT0000-mdc-ffff9f95a130a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5466.525958] LustreError: 83312:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5466.539326] LustreError: 83312:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 5466.872547] LustreError: 83289:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5466.884936] LustreError: 83289:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5466.908889] LustreError: 83289:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9f95a130a000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5477.573901] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 23:40:25 (1788579625)
[ 5496.896856] LustreError: 79724:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5496.915907] LustreError: 79724:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5500.985166] LustreError: 79724:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5500.989652] LustreError: 79724:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5506.889975] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 23:40:54 (1788579654)
[ 5555.085941] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 23:41:43 (1788579703)
[ 5619.094179] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 23:42:47 (1788579767)
[ 5652.663552] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 23:43:20 (1788579800)
[ 5653.157996] Lustre: Mounted lustre-client - version 2.17.58_3_g4a08d7b
[ 5777.672863] Lustre: Unmounted lustre-client
[ 5784.509676] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 23:45:32 (1788579932)
[ 5785.244372] LustreError: 87619:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5787.691018] LustreError: lustre-OST0000-osc-ffff9f95a130a000: operation ldlm_enqueue to node 192.168.206.159@tcp failed: rc = -107
[ 5787.699123] LustreError: Skipped 9 previous similar messages
[ 5787.726606] LustreError: lustre-OST0000-osc-ffff9f95a130a000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5787.735648] LustreError: 87702:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9f95a130a000: namespace resource [0x240000400:0x412d:0x0].0x0 (ffff9f9585c1bc00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5787.750756] LustreError: 87702:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5787.992168] LustreError: 87619:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5788.001575] Lustre: 2354:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.159@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5794.644867] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 23:45:42 (1788579942)
[ 5852.140252] Lustre: lustre-MDT0000-mdc-ffff9f95a130a000: Connection restored to 192.168.206.159@tcp (at 192.168.206.159@tcp)
[ 5852.152418] Lustre: Skipped 13 previous similar messages
[ 5858.875300] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 23:46:46 (1788580006)
[ 5860.588621] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5862.366565] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 23:46:50 (1788580010)
[ 5925.949228] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5927.721979] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 23:47:55 (1788580075)
[ 5984.224159] Lustre: 90019:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788580078/real 1788580078]  req@ffff9f9584329880 x1875456024857600/t0(0) o36->lustre-MDT0000-mdc-ffff9f95a130a000@192.168.206.159@tcp:12/10 lens 496/440 e 0 to 1 dl 1788580133 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ln.0' uid:0 gid:0 projid:4294967295
[ 5984.264581] Lustre: 90019:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 5985.893919] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 23:48:53 (1788580133)
[ 5987.842457] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5989.471044] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 23:48:57 (1788580137)
[ 5991.003474] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5992.637876] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 23:49:00 (1788580140)
[ 6024.103952] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 23:49:32 (1788580172)
[ 6033.100496] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6055.904649] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 192.168.206.159@tcp) was lost; in progress operations using this service will fail
[ 6055.919955] LustreError: Skipped 2 previous similar messages
[ 6061.039511] Lustre: lustre-MDT0000-mdc-ffff9f95a130a000: Connection to lustre-MDT0000 (at 192.168.206.159@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6061.055827] Lustre: Skipped 11 previous similar messages
[ 6066.293363] Lustre: Evicted from MGS (at 192.168.206.159@tcp) after server handle changed from 0xf2c1376926d1fc30 to 0xf2c1376926d21703
[ 6066.306832] Lustre: Skipped 2 previous similar messages
[ 6070.890556] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6072.305235] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6081.836420] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 23:50:29 (1788580229)
[ 6084.024956] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6085.545574] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 23:50:33 (1788580233)
[ 6097.403102] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 192.168.206.159@tcp) was lost; in progress operations using this service will fail
[ 6110.822756] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 23:50:58 (1788580258)
[ 6133.249966] Lustre: Evicted from MGS (at 192.168.206.159@tcp) after server handle changed from 0xf2c1376926d21892 to 0xf2c1376926d21bcc
[ 6133.259665] Lustre: Skipped 1 previous similar message
[ 6149.594531] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 23:51:37 (1788580297)
[ 6210.455691] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6214.408239] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6303.008273] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 192.168.206.159@tcp) was lost; in progress operations using this service will fail
[ 6303.028746] LustreError: Skipped 1 previous similar message
[ 6303.057326] Lustre: Evicted from MGS (at 192.168.206.159@tcp) after server handle changed from 0xf2c1376926d21bcc to 0xf2c1376926d2296a
[ 6304.253166] Lustre: 68226:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.159@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6325.316167] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6326.885414] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6351.980559] Lustre: 68226:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.159@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6372.819552] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6374.159229] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6496.143842] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 23:57:24 (1788580644)
[ 6504.128752] LustreError: lustre-OST0000-osc-ffff9f95a130a000: operation ost_setattr to node 192.168.206.159@tcp failed: rc = -107
[ 6504.139630] LustreError: Skipped 19 previous similar messages
[ 6558.406468] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6562.435489] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6802.107854] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 00:02:30 (1788580950)
[ 6922.730140] Lustre: lustre-MDT0000-mdc-ffff9f95a130a000: Connection to lustre-MDT0000 (at 192.168.206.159@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6922.737150] Lustre: Skipped 6 previous similar messages
[ 6932.965777] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 192.168.206.159@tcp) was lost; in progress operations using this service will fail
[ 6932.993314] LustreError: Skipped 1 previous similar message
[ 6933.004322] Lustre: Evicted from MGS (at 192.168.206.159@tcp) after server handle changed from 0xf2c1376926d22c9d to 0xf2c1376926e1d018
[ 6933.015361] Lustre: Skipped 1 previous similar message
[ 6933.036220] Lustre: MGC192.168.206.159@tcp: Connection restored to 192.168.206.159@tcp (at 192.168.206.159@tcp)
[ 6933.046013] Lustre: Skipped 12 previous similar messages
[ 6969.844258] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 00:05:17 (1788581117)
[ 6971.197240] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6972.730407] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 00:05:20 (1788581120)
[ 6974.440558] LustreError: lustre-MDT0000-mdc-ffff9f95a130a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6974.474268] LustreError: 101976:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9f95a130a000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9f95b23f6d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6974.485317] LustreError: 101976:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6980.411157] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 00:05:28 (1788581128)
[ 7146.988128] LustreError: lustre-OST0000-osc-ffff9f95a130a000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7148.456506] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 00:08:16 (1788581296)
[ 7174.624165] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788581303/real 1788581303]  req@ffff9f958f719880 x1875456042543744/t0(0) o4->lustre-OST0000-osc-ffff9f95a130a000@192.168.206.159@tcp:6/4 lens 4584/448 e 0 to 1 dl 1788581323 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7174.658701] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 20 previous similar messages
[ 7190.475488] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 00:08:58 (1788581338)
[ 7191.769989] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7193.164660] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 00:09:01 (1788581341)
[ 7194.516491] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7196.067650] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 00:09:04 (1788581344)
[ 7227.961512] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 00:09:36 (1788581376)
[ 7266.798111] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 192.168.206.159@tcp) was lost; in progress operations using this service will fail
[ 7266.831852] Lustre: Evicted from MGS (at 192.168.206.159@tcp) after server handle changed from 0xf2c1376926e1d018 to 0xf2c1376926e1f8e4
[ 7266.840711] Lustre: Skipped 1 previous similar message
[ 7278.907215] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 00:10:27 (1788581427)
[ 7280.110591] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7281.378467] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 00:10:29 (1788581429)
[ 7282.424896] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7283.776480] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 00:10:31 (1788581431)
[ 7292.407495] Lustre: Unmounted lustre-client
[ 7293.608275] Lustre: Mounted lustre-client - version 2.17.58_3_g4a08d7b
[ 7296.114822] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7316.783669] Lustre: 106748:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.159@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 7335.532754] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2