[ 1861.828518] LustreError: lustre-OST0000-osc-ffff9d2c03f8f000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1861.853567] Lustre: 2346:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.120@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1868.292990] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 22:49:48 (1788835788)
[ 1870.633035] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1872.308483] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 22:49:52 (1788835792)
[ 1874.305783] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1875.984873] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 22:49:56 (1788835796)
[ 1880.045613] LustreError: lustre-MDT0000-mdc-ffff9d2c03f8f000: operation mds_reint to node 192.168.204.120@tcp failed: rc = -19
[ 1880.056374] LustreError: Skipped 1 previous similar message
[ 1901.536305] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 1901.572277] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x7083404e3fca6ae4 to 0x7083404e3fca8309
[ 1903.360897] Lustre: 15940:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.120@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2010.508384] LustreError: lustre-MDT0000-mdc-ffff9d2c03f8f000: operation mds_reint to node 192.168.204.120@tcp failed: rc = -19
[ 2010.519254] LustreError: Skipped 1 previous similar message
[ 2028.512259] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 2037.677175] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x7083404e3fca8309 to 0x7083404e3fcc615c
[ 2052.961684] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 22:52:53 (1788835973)
[ 2053.340861] Lustre: *** cfs_fail_loc=305, val=0***
[ 2053.342627] Lustre: Skipped 4 previous similar messages
[ 2094.048333] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 2104.315664] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x7083404e3fcc615c to 0x7083404e3fcc656f
[ 2116.490338] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2117.734165] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2125.454436] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 22:54:05 (1788836045)
[ 2135.022522] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 2135.038646] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x7083404e3fcc656f to 0x7083404e3fcc6a00
[ 2135.070561] LustreError: lustre-MDT0000-mdc-ffff9d2c03f8f000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2163.464251] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 22:54:42 (1788836082)
[ 2181.108781] LustreError: lustre-OST0000-osc-ffff9d2c03f8f000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2201.176249] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 22:55:21 (1788836121)
[ 2215.977836] LustreError: lustre-MDT0000-mdc-ffff9d2c03f8f000: operation mds_reint to node 192.168.204.120@tcp failed: rc = -19
[ 2215.988593] LustreError: Skipped 1 previous similar message
[ 2233.312358] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 2243.570296] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x7083404e3fcc6a00 to 0x7083404e3fcca16d
[ 2264.475532] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2267.523315] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2349.600604] Lustre: 2349:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788836255/real 1788836255]  req@ffff9d2c085c1c00 x1875728368306432/t0(0) o400->MGC192.168.204.120@tcp@192.168.204.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1788836271 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2349.656957] Lustre: 2349:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 2349.674319] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 2359.916869] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x7083404e3fcca16d to 0x7083404e3fce318c
[ 2361.201925] Lustre: 15940:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.120@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2379.336795] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2381.770368] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2447.673616] LustreError: lustre-MDT0000-mdc-ffff9d2c03f8f000: operation ldlm_enqueue to node 192.168.204.120@tcp failed: rc = -19
[ 2447.693783] LustreError: Skipped 1 previous similar message
[ 2447.701448] Lustre: lustre-MDT0000-mdc-ffff9d2c03f8f000: Connection to lustre-MDT0000 (at 192.168.204.120@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2447.719758] Lustre: Skipped 9 previous similar messages
[ 2466.786878] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 2476.972569] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x7083404e3fce318c to 0x7083404e3fcfdb12
[ 2476.989519] Lustre: MGC192.168.204.120@tcp: Connection restored to 192.168.204.120@tcp (at 192.168.204.120@tcp)
[ 2477.002627] Lustre: Skipped 15 previous similar messages
[ 2493.587876] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2495.717718] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2525.614682] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 23:00:45 (1788836445)
[ 2548.448274] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 2557.628644] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x7083404e3fcfdb12 to 0x7083404e3fd09215
[ 2559.609190] Lustre: 15940:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.120@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2571.172316] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2613.723439] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2649.279155] Lustre: 15940:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.120@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2660.338856] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2694.114089] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 2694.126656] LustreError: Skipped 2 previous similar messages
[ 2694.154971] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x7083404e3fd0f087 to 0x7083404e3fd1496c
[ 2694.167513] Lustre: Skipped 2 previous similar messages
[ 2696.172591] Lustre: 15940:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.120@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2714.401582] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2738.040217] LustreError: lustre-MDT0000-mdc-ffff9d2c03f8f000: operation mds_reint to node 192.168.204.120@tcp failed: rc = -19
[ 2738.053181] LustreError: Skipped 6 previous similar messages
[ 2766.485849] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2836.368171] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2927.871519] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 23:07:27 (1788836847)
[ 2996.478572] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2999.415328] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3273.748646] LustreError: lustre-OST0000-osc-ffff9d2c03f8f000: operation ldlm_cancel to node 192.168.204.120@tcp failed: rc = -19
[ 3273.748910] Lustre: lustre-OST0000-osc-ffff9d2c03f8f000: Connection to lustre-OST0000 (at 192.168.204.120@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3273.758028] LustreError: Skipped 5 previous similar messages
[ 3273.779443] Lustre: Skipped 8 previous similar messages
[ 3302.523651] Lustre: lustre-OST0000-osc-ffff9d2c03f8f000: Connection restored to 192.168.204.120@tcp (at 192.168.204.120@tcp)
[ 3302.530262] Lustre: Skipped 16 previous similar messages
[ 3321.917238] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3324.424365] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3652.396496] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3654.734214] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3902.285869] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 23:23:41 (1788837821)
[ 3919.840225] Lustre: 50591:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788837825/real 1788837826]  req@ffff9d2c087bd500 x1875728386889216/t0(0) o101->lustre-MDT0000-mdc-ffff9d2c03f8f000@192.168.204.120@tcp:12/10 lens 576/1152 e 0 to 1 dl 1788837841 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3919.892228] Lustre: 50591:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 3919.901390] Lustre: lustre-MDT0000-mdc-ffff9d2c03f8f000: Connection to lustre-MDT0000 (at 192.168.204.120@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3919.920431] Lustre: Skipped 1 previous similar message
[ 3919.979507] Lustre: lustre-MDT0000-mdc-ffff9d2c03f8f000: Connection restored to 192.168.204.120@tcp (at 192.168.204.120@tcp)
[ 3919.999805] Lustre: Skipped 1 previous similar message
[ 3929.554091] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 23:24:09 (1788837849)
[ 3956.690575] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 23:24:36 (1788837876)
[ 3983.547783] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 23:25:03 (1788837903)
[ 3984.370453] Lustre: Mounted lustre-client - version 2.17.57_106_g45f4cc0
[ 4016.097953] Lustre: 2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788837921/real 1788837921]  req@ffff9d2c08c31500 x1875728386909056/t0(0) o400->MGC192.168.204.120@tcp@192.168.204.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1788837937 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4016.144476] Lustre: 2346:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4016.156195] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 4016.177498] LustreError: Skipped 3 previous similar messages
[ 4026.367828] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x7083404e3fd2dfff to 0x7083404e3fe3fa39
[ 4026.376197] Lustre: Skipped 3 previous similar messages
[ 4044.686260] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4047.160623] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4049.334276] Lustre: Unmounted lustre-client
[ 4057.365739] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 23:26:17 (1788837977)
[ 4180.960177] Lustre: 2348:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788838086/real 1788838086]  req@ffff9d2c083b2a00 x1875728386945280/t0(0) o4->lustre-OST0000-osc-ffff9d2c03f8f000@192.168.204.120@tcp:6/4 lens 488/448 e 0 to 1 dl 1788838102 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4181.012424] Lustre: 2348:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 69 previous similar messages
[ 4356.340529] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 23:31:16 (1788838276)
[ 4406.572244] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 23:32:06 (1788838326)
[ 4410.200664] Lustre: Unmounted lustre-client
[ 4434.054393] Lustre: Mounted lustre-client - version 2.17.57_106_g45f4cc0
[ 4442.021318] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 23:32:41 (1788838361)
[ 4442.636327] LustreError: 56519:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4443.680536] LustreError: 56519:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4443.927839] Lustre: *** cfs_fail_loc=305, val=0***
[ 4468.711979] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 23:33:08 (1788838388)
[ 4469.462669] Lustre: Mounted lustre-client - version 2.17.57_106_g45f4cc0
[ 4472.592821] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4482.902305] Lustre: Unmounted lustre-client
[ 4493.917430] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 23:33:33 (1788838413)
[ 4603.088171] LustreError: lustre-MDT0000-mdc-ffff9d2c03813000: operation mds_reint to node 192.168.204.120@tcp failed: rc = -19
[ 4603.093650] LustreError: Skipped 4 previous similar messages
[ 4603.116418] Lustre: lustre-MDT0000-mdc-ffff9d2c03813000: Connection to lustre-MDT0000 (at 192.168.204.120@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4603.143337] Lustre: Skipped 23 previous similar messages
[ 4620.218349] Lustre: 2348:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788838525/real 1788838525]  req@ffff9d2c0668ad80 x1875728389624704/t0(0) o400->MGC192.168.204.120@tcp@192.168.204.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1788838541 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4620.266522] Lustre: 2348:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 87 previous similar messages
[ 4620.279603] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 4630.588266] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x7083404e3fe41663 to 0x7083404e3fe6ca52
[ 4630.599114] Lustre: MGC192.168.204.120@tcp: Connection restored to 192.168.204.120@tcp (at 192.168.204.120@tcp)
[ 4630.606231] Lustre: Skipped 24 previous similar messages
[ 4632.629840] Lustre: 55915:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.120@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4822.842903] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 23:39:02 (1788838742)
[ 4828.713644] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4843.519802] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 4843.548527] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x7083404e3fe6ca52 to 0x7083404e3feb953c
[ 4843.574835] LustreError: lustre-MDT0000-mdc-ffff9d2c03813000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4859.707441] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 23:39:39 (1788838779)
[ 4860.419395] Lustre: Mounted lustre-client - version 2.17.57_106_g45f4cc0
[ 4868.008456] LustreError: lustre-OST0000-osc-ffff9d2c03813000: operation ldlm_enqueue to node 192.168.204.120@tcp failed: rc = -107
[ 4868.031301] LustreError: lustre-OST0000-osc-ffff9d2c03813000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4880.850620] Lustre: Unmounted lustre-client
[ 4888.942525] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 23:40:08 (1788838808)
[ 4895.556417] LustreError: lustre-MDT0000-mdc-ffff9d2c03813000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4895.574125] LustreError: 60917:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4903.442745] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 23:40:23 (1788838823)
[ 4903.562805] LustreError: 61616:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4909.024259] LustreError: 61616:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4909.046223] LustreError: 61616:0:(import.c:711:ptlrpc_connect_import_locked()) already connected
[ 4910.486652] LustreError: lustre-MDT0000-mdc-ffff9d2c03813000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4910.506775] LustreError: 61638:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4910.579216] LustreError: 61643:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4911.674599] LustreError: 61650:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4911.688172] LustreError: 61650:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4914.072382] LustreError: 61672:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4914.093752] LustreError: 61672:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4918.723357] LustreError: 61717:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4918.737242] LustreError: 61717:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4930.370245] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 23:40:50 (1788838850)
[ 4970.871224] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4973.093619] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4984.889799] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 23:41:44 (1788838904)
[ 5026.647559] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5028.408976] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5040.059919] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 23:42:39 (1788838959)
[ 5106.424773] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5108.530331] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5117.970402] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 23:43:57 (1788839037)
[ 5159.093117] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5160.992977] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5168.403273] Lustre: Unmounted lustre-client
[ 5227.136526] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5230.226253] Lustre: Mounted lustre-client - version 2.17.57_106_g45f4cc0
[ 5238.091953] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 23:45:57 (1788839157)
[ 5241.086338] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5243.304896] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 23:46:02 (1788839162)
[ 5271.862322] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 23:46:31 (1788839191)
[ 5273.335520] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5275.829163] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 23:46:35 (1788839195)
[ 5277.290329] Lustre: *** cfs_fail_loc=805, val=0***
[ 5277.366974] Lustre: Mounted lustre-client - version 2.17.57_106_g45f4cc0
[ 5283.096358] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5304.288233] Lustre: 2348:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788839209/real 1788839209]  req@ffff9d2c066bad80 x1875728391681408/t0(0) o400->MGC192.168.204.120@tcp@192.168.204.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1788839225 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5304.290784] Lustre: lustre-MDT0000-mdc-ffff9d2c0824b800: Connection to lustre-MDT0000 (at 192.168.204.120@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5304.369078] Lustre: 2348:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5304.369152] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 5304.401355] Lustre: Skipped 12 previous similar messages
[ 5314.034092] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x7083404e3feb9f30 to 0x7083404e3feba1b4
[ 5314.091709] Lustre: MGC192.168.204.120@tcp: Connection restored to 192.168.204.120@tcp (at 192.168.204.120@tcp)
[ 5314.104428] Lustre: Skipped 12 previous similar messages
[ 5326.765661] Lustre: Unmounted lustre-client
[ 5336.118146] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 23:47:35 (1788839255)
[ 5378.517706] LustreError: 2345:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff9d2c066b9500 x1875728391694080/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff9d2c0824b800@192.168.204.120@tcp:12/10 lens 664/608 e 0 to 0 dl 1788839315 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5387.260788] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5389.043231] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5398.512910] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 23:48:38 (1788839318)
[ 5400.166631] LustreError: lustre-OST0000-osc-ffff9d2c0824b800: operation ost_write to node 192.168.204.120@tcp failed: rc = -107
[ 5400.176099] LustreError: Skipped 2 previous similar messages
[ 5400.192943] LustreError: lustre-OST0000-osc-ffff9d2c0824b800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5400.210992] Lustre: 2349:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.120@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5400.244454] LustreError: 72835:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9d2c0824b800: namespace resource [0x240000400:0x37e2:0x0].0x0 (ffff9d2c043ed000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5409.092313] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 23:48:49 (1788839329)
[ 5410.790878] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5412.735485] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 23:48:52 (1788839332)
[ 5414.316615] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5415.827380] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 23:48:56 (1788839336)
[ 5417.579904] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5419.442845] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 23:48:59 (1788839339)
[ 5420.903739] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5422.531850] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 23:49:02 (1788839342)
[ 5424.140826] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5425.847780] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 23:49:05 (1788839345)
[ 5427.607278] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5429.793480] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 23:49:09 (1788839349)
[ 5431.731606] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5433.558679] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 23:49:13 (1788839353)
[ 5435.489976] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5437.573215] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 23:49:17 (1788839357)
[ 5439.240406] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5441.307320] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 23:49:20 (1788839360)
[ 5442.952386] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5445.253883] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 23:49:24 (1788839364)
[ 5447.048810] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5449.175636] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 23:49:28 (1788839368)
[ 5452.212612] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5454.005414] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 23:49:33 (1788839373)
[ 5491.153715] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 23:50:09 (1788839409)
[ 5523.287457] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 23:50:43 (1788839443)
[ 5524.013487] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5534.811579] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 23:50:54 (1788839454)
[ 5535.109721] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5546.399238] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 23:51:06 (1788839466)
[ 5546.897454] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5554.981630] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 23:51:14 (1788839474)
[ 5555.307449] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5564.833327] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 23:51:24 (1788839484)
[ 5565.139704] Lustre: *** cfs_fail_loc=510, val=3***
[ 5575.379486] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 23:51:34 (1788839494)
[ 5575.820040] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5643.970057] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 23:52:43 (1788839563)
[ 5644.457209] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5710.873270] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 23:53:50 (1788839630)
[ 5711.253202] LustreError: 83050:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5714.274440] LustreError: lustre-MDT0000-mdc-ffff9d2c0824b800: operation ldlm_enqueue to node 192.168.204.120@tcp failed: rc = -107
[ 5714.305363] LustreError: lustre-MDT0000-mdc-ffff9d2c0824b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5714.340624] LustreError: 83064:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9d2c0824b800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9d2c08567200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5715.328172] LustreError: 83050:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5717.478038] LustreError: 83080:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5720.803548] LustreError: lustre-MDT0000-mdc-ffff9d2c0824b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5720.831933] LustreError: 83094:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9d2c0824b800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9d2c065dd700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5720.856908] LustreError: 83094:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5721.498376] LustreError: 83080:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5728.712748] LustreError: lustre-MDT0000-mdc-ffff9d2c0824b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5728.735786] LustreError: 83123:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9d2c0824b800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9d2c065ddf00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5728.770804] LustreError: 83123:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5729.235609] LustreError: 83136:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5732.405891] LustreError: lustre-MDT0000-mdc-ffff9d2c0824b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5732.514303] LustreError: 83155:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5732.523123] LustreError: 83155:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5733.320287] LustreError: 83136:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5733.331216] LustreError: 83136:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9d2c0824b800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5733.674562] LustreError: 83162:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5733.687565] LustreError: 83162:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5736.067686] LustreError: 83184:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5736.084702] LustreError: 83184:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5739.735123] LustreError: 83211:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5743.010615] LustreError: lustre-MDT0000-mdc-ffff9d2c0824b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5743.028341] LustreError: 83225:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9d2c0824b800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9d2c066a1b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5743.832270] LustreError: 83211:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5750.935942] LustreError: lustre-MDT0000-mdc-ffff9d2c0824b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5751.552945] LustreError: 83267:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5754.632965] LustreError: lustre-MDT0000-mdc-ffff9d2c0824b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5754.654981] LustreError: 83282:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9d2c0824b800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9d2c065ddd00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5754.689031] LustreError: 83282:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5755.634868] LustreError: 83267:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5758.770762] LustreError: lustre-MDT0000-mdc-ffff9d2c0824b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5758.858974] LustreError: 83317:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5758.868579] LustreError: 83317:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5759.825123] LustreError: 83295:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9d2c0824b800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5767.937325] LustreError: 83369:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5767.946958] LustreError: 83369:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5771.580162] LustreError: lustre-MDT0000-mdc-ffff9d2c0824b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5771.705590] LustreError: 83391:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5771.718142] LustreError: 83391:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5772.024900] LustreError: 83369:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5772.039195] LustreError: 83369:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5772.048925] LustreError: 83369:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9d2c0824b800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5784.007534] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 23:55:03 (1788839703)
[ 5803.762109] LustreError: 79793:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5803.781379] LustreError: 79793:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5807.856163] LustreError: 79793:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5807.864672] LustreError: 79793:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5816.079798] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 23:55:35 (1788839735)
[ 5869.524669] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 23:56:29 (1788839789)
[ 5926.880237] Lustre: 85549:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788839793/real 1788839793]  req@ffff9d2c08450700 x1875728391826176/t0(0) o101->lustre-MDT0000-mdc-ffff9d2c0824b800@192.168.204.120@tcp:12/10 lens 576/1152 e 0 to 1 dl 1788839848 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'stat.0' uid:0 gid:0 projid:0
[ 5926.918032] Lustre: 85549:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 21 previous similar messages
[ 5926.927606] Lustre: lustre-MDT0000-mdc-ffff9d2c0824b800: Connection to lustre-MDT0000 (at 192.168.204.120@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5926.956825] Lustre: Skipped 17 previous similar messages
[ 5926.978665] Lustre: lustre-MDT0000-mdc-ffff9d2c0824b800: Connection restored to 192.168.204.120@tcp (at 192.168.204.120@tcp)
[ 5926.987233] Lustre: Skipped 17 previous similar messages
[ 5935.253549] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 23:57:34 (1788839854)
[ 5972.591906] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 23:58:11 (1788839891)
[ 5973.176867] Lustre: Mounted lustre-client - version 2.17.57_106_g45f4cc0
[ 6096.918689] Lustre: Unmounted lustre-client
[ 6104.141218] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 00:00:24 (1788840024)
[ 6104.903151] LustreError: 87700:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6107.805263] LustreError: lustre-OST0000-osc-ffff9d2c0824b800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6107.816727] LustreError: 87787:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9d2c0824b800: namespace resource [0x240000400:0x380d:0x0].0x0 (ffff9d2c08567500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6107.832670] LustreError: 87787:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6107.952284] LustreError: 87700:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6107.961643] Lustre: 2348:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.120@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6114.332235] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 00:00:34 (1788840034)
[ 6180.192265] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 00:01:39 (1788840099)
[ 6182.008724] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6183.763127] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 00:01:43 (1788840103)
[ 6248.988946] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6251.504564] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 00:02:50 (1788840170)
[ 6310.491400] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 00:03:49 (1788840229)
[ 6313.021982] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6315.074418] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 00:03:54 (1788840234)
[ 6317.021276] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6318.988162] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 00:03:58 (1788840238)
[ 6356.879462] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 00:04:36 (1788840276)
[ 6369.477703] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6394.848473] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 6394.868091] LustreError: Skipped 2 previous similar messages
[ 6405.100715] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x7083404e3febad92 to 0x7083404e3febc5a9
[ 6405.109429] Lustre: Skipped 2 previous similar messages
[ 6406.593518] Lustre: 68294:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.120@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6423.221803] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6425.017298] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6437.239906] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 00:05:56 (1788840356)
[ 6441.370515] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6444.513377] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 00:06:03 (1788840363)
[ 6461.424185] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 6461.466771] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x7083404e3febc5a9 to 0x7083404e3febc9e6
[ 6478.841969] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 00:06:38 (1788840398)
[ 6514.807299] Lustre: 68294:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.120@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6547.607502] Lustre: lustre-MDT0000-mdc-ffff9d2c0824b800: Connection restored to 192.168.204.120@tcp (at 192.168.204.120@tcp)
[ 6547.624514] Lustre: Skipped 9 previous similar messages
[ 6559.982520] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 00:07:59 (1788840479)
[ 6568.626513] LustreError: lustre-OST0000-osc-ffff9d2c0824b800: operation ost_setattr to node 192.168.204.120@tcp failed: rc = -19
[ 6568.655313] LustreError: Skipped 9 previous similar messages
[ 6568.660854] Lustre: lustre-OST0000-osc-ffff9d2c0824b800: Connection to lustre-OST0000 (at 192.168.204.120@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6568.678738] Lustre: Skipped 7 previous similar messages
[ 6631.989920] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6636.256831] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6722.851778] Lustre: 2348:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788840628/real 1788840628]  req@ffff9d2c10349f80 x1875728398400640/t0(0) o400->MGC192.168.204.120@tcp@192.168.204.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1788840644 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6722.893884] Lustre: 2348:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 6722.904973] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 6722.921153] LustreError: Skipped 1 previous similar message
[ 6733.290600] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x7083404e3febcc71 to 0x7083404e3febdac5
[ 6733.309170] Lustre: Skipped 1 previous similar message
[ 6750.615132] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6752.640612] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6793.611591] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6795.398578] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6959.322483] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 00:14:39 (1788840879)
[ 7035.230656] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 7038.058081] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7314.538324] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 00:20:33 (1788841233)
[ 7502.330711] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 7502.340050] LustreError: Skipped 1 previous similar message
[ 7502.345264] Lustre: lustre-MDT0000-mdc-ffff9d2c0824b800: Connection to lustre-MDT0000 (at 192.168.204.120@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7502.349652] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x7083404e3febe14e to 0x7083404e3ffb81f8
[ 7502.360642] Lustre: Skipped 3 previous similar messages
[ 7502.382930] Lustre: Skipped 1 previous similar message
[ 7502.406115] Lustre: MGC192.168.204.120@tcp: Connection restored to 192.168.204.120@tcp (at 192.168.204.120@tcp)
[ 7502.417840] Lustre: Skipped 4 previous similar messages
[ 7535.073653] Lustre: 2348:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788841413/real 1788841413]  req@ffff9d2c10be1c00 x1875728409566720/t0(0) o400->lustre-MDT0000-mdc-ffff9d2c0824b800@192.168.204.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1788841456 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7535.104977] Lustre: 2348:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 7535.964954] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 00:24:16 (1788841456)
[ 7537.546657] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7539.248971] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 00:24:19 (1788841459)
[ 7541.277419] LustreError: lustre-MDT0000-mdc-ffff9d2c0824b800: operation ldlm_enqueue to node 192.168.204.120@tcp failed: rc = -107
[ 7541.295608] LustreError: Skipped 54 previous similar messages
[ 7541.309968] LustreError: lustre-MDT0000-mdc-ffff9d2c0824b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7549.504480] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 00:24:29 (1788841469)
[ 7718.516110] Lustre: Evicted from lustre-OST0000_UUID (at 192.168.204.120@tcp) after server handle changed from 0x7083404e3ffb843d to 0x7083404e3ffb8547
[ 7718.528288] LustreError: lustre-OST0000-osc-ffff9d2c0824b800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7719.993920] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 00:27:20 (1788841640)
[ 7766.260705] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 00:28:05 (1788841685)
[ 7767.975683] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7770.592871] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 00:28:10 (1788841690)
[ 7772.216426] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7774.410731] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 00:28:13 (1788841693)
[ 7810.098723] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 00:28:49 (1788841729)
[ 7852.014654] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 7852.053132] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x7083404e3ffb81f8 to 0x7083404e3ffba894
[ 7867.034303] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 00:29:47 (1788841787)
[ 7869.335729] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7871.769373] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 00:29:51 (1788841791)
[ 7873.967306] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7876.394564] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 00:29:55 (1788841795)
[ 7889.013270] Lustre: Unmounted lustre-client
[ 7891.134856] Lustre: Mounted lustre-client - version 2.17.57_106_g45f4cc0
[ 7894.404958] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7917.034245] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 7917.050454] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x7083404e3ffbb18c to 0x7083404e3ffbb1e0
[ 7918.835796] Lustre: 106862:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.120@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 7938.069094] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2