[ 1856.538211] LustreError: lustre-OST0000-osc-ffff8bdc02b96800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1856.554725] Lustre: 2356:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.122@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1861.988946] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 20:03:52 (1788307432)
[ 1863.963392] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1865.570909] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 20:03:55 (1788307435)
[ 1866.982262] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1868.410576] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 20:03:58 (1788307438)
[ 1871.392346] LustreError: lustre-MDT0000-mdc-ffff8bdc02b96800: operation mds_close to node 192.168.206.122@tcp failed: rc = -19
[ 1871.404551] LustreError: Skipped 1 previous similar message
[ 1890.278939] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 1890.320899] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0xd13f7c8cbe5791a8 to 0xd13f7c8cbe57aa59
[ 1990.963255] LustreError: lustre-MDT0000-mdc-ffff8bdc02b96800: operation mds_close to node 192.168.206.122@tcp failed: rc = -19
[ 1990.970234] LustreError: Skipped 2 previous similar messages
[ 2008.095310] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 2018.287423] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0xd13f7c8cbe57aa59 to 0xd13f7c8cbe59fc10
[ 2034.785469] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 20:06:44 (1788307604)
[ 2035.262745] Lustre: *** cfs_fail_loc=305, val=0***
[ 2035.265960] Lustre: Skipped 2 previous similar messages
[ 2079.719183] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 2079.741736] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0xd13f7c8cbe59fc10 to 0xd13f7c8cbe59ff97
[ 2081.653181] Lustre: 15922:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.122@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2101.261312] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2102.708992] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2110.208740] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 20:08:00 (1788307680)
[ 2120.684992] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 2120.708847] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0xd13f7c8cbe59ff97 to 0xd13f7c8cbe5a0436
[ 2122.124527] LustreError: lustre-MDT0000-mdc-ffff8bdc02b96800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2143.494092] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 20:08:32 (1788307712)
[ 2176.297255] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 20:09:06 (1788307746)
[ 2189.842139] LustreError: lustre-MDT0000-mdc-ffff8bdc02b96800: operation ldlm_enqueue to node 192.168.206.122@tcp failed: rc = -19
[ 2189.842987] Lustre: lustre-MDT0000-mdc-ffff8bdc02b96800: Connection to lustre-MDT0000 (at 192.168.206.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2189.848590] LustreError: Skipped 2 previous similar messages
[ 2189.884947] Lustre: Skipped 13 previous similar messages
[ 2208.479211] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 2208.526093] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0xd13f7c8cbe5a0436 to 0xd13f7c8cbe5a516e
[ 2208.540415] Lustre: MGC192.168.206.122@tcp: Connection restored to 192.168.206.122@tcp (at 192.168.206.122@tcp)
[ 2208.548180] Lustre: Skipped 16 previous similar messages
[ 2233.867064] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2235.842512] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2321.183661] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788307875/real 1788307875]  req@ffff8bdc1818ad80 x1875174399849728/t0(0) o400->MGC192.168.206.122@tcp@192.168.206.122@tcp:26/25 lens 224/224 e 0 to 1 dl 1788307891 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2321.212938] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 2321.219829] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 2321.277255] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0xd13f7c8cbe5a516e to 0xd13f7c8cbe5bf9e3
[ 2347.242610] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2349.273841] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2414.705188] LustreError: lustre-MDT0000-mdc-ffff8bdc02b96800: operation mds_close to node 192.168.206.122@tcp failed: rc = -19
[ 2414.716565] LustreError: Skipped 2 previous similar messages
[ 2431.968665] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 2442.237123] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0xd13f7c8cbe5bf9e3 to 0xd13f7c8cbe5dccf2
[ 2458.889369] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2460.916897] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2488.391853] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 20:14:18 (1788308058)
[ 2508.769414] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 2518.206958] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0xd13f7c8cbe5dccf2 to 0xd13f7c8cbe5ee0ed
[ 2520.628162] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2548.483337] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2582.648533] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2622.075800] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2661.855755] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 2661.877613] LustreError: Skipped 3 previous similar messages
[ 2671.401816] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2672.108348] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0xd13f7c8cbe5fe552 to 0xd13f7c8cbe60b3e7
[ 2672.117571] Lustre: Skipped 3 previous similar messages
[ 2699.074733] LustreError: lustre-MDT0000-mdc-ffff8bdc02b96800: operation ldlm_enqueue to node 192.168.206.122@tcp failed: rc = -19
[ 2699.097267] LustreError: Skipped 8 previous similar messages
[ 2724.713742] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2807.370934] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 20:19:37 (1788308377)
[ 2819.969390] Lustre: lustre-OST0000-osc-ffff8bdc02b96800: Connection to lustre-OST0000 (at 192.168.206.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2819.981421] Lustre: Skipped 9 previous similar messages
[ 2852.727992] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2854.575234] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3172.843054] Lustre: lustre-OST0000-osc-ffff8bdc02b96800: Connection restored to 192.168.206.122@tcp (at 192.168.206.122@tcp)
[ 3172.853486] Lustre: Skipped 19 previous similar messages
[ 3186.595530] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3188.362923] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3483.731487] LustreError: lustre-OST0000-osc-ffff8bdc02b96800: operation ldlm_cancel to node 192.168.206.122@tcp failed: rc = -19
[ 3483.740727] Lustre: lustre-OST0000-osc-ffff8bdc02b96800: Connection to lustre-OST0000 (at 192.168.206.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3483.744285] LustreError: Skipped 4 previous similar messages
[ 3483.775706] Lustre: Skipped 1 previous similar message
[ 3525.779094] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3527.955225] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3778.818468] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 20:35:48 (1788309348)
[ 3796.450489] Lustre: 50565:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788309351/real 1788309351]  req@ffff8bdc12c2f100 x1875174424893056/t0(0) o101->lustre-MDT0000-mdc-ffff8bdc02b96800@192.168.206.122@tcp:12/10 lens 576/1152 e 0 to 1 dl 1788309367 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3796.487326] Lustre: 50565:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 3796.531926] Lustre: lustre-MDT0000-mdc-ffff8bdc02b96800: Connection restored to 192.168.206.122@tcp (at 192.168.206.122@tcp)
[ 3796.543812] Lustre: Skipped 1 previous similar message
[ 3803.962235] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 20:36:13 (1788309373)
[ 3828.103512] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 20:36:38 (1788309398)
[ 3852.486483] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 20:37:02 (1788309422)
[ 3852.989085] Lustre: Mounted lustre-client - version 2.17.57_44_g3acc781
[ 3883.999291] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788309439/real 1788309439]  req@ffff8bdc12c89880 x1875174424911616/t0(0) o400->MGC192.168.206.122@tcp@192.168.206.122@tcp:26/25 lens 224/224 e 0 to 1 dl 1788309455 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3884.037218] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3884.043723] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 3884.064863] LustreError: Skipped 2 previous similar messages
[ 3884.092464] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0xd13f7c8cbe61b478 to 0xd13f7c8cbe7968f3
[ 3884.103915] Lustre: Skipped 2 previous similar messages
[ 3885.231076] Lustre: 15922:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.122@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 3905.023292] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3906.493213] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3908.657355] Lustre: Unmounted lustre-client
[ 3914.316898] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 20:38:04 (1788309484)
[ 4034.336183] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788309589/real 1788309589]  req@ffff8bdc05c25500 x1875174424947072/t0(0) o4->lustre-OST0000-osc-ffff8bdc02b96800@192.168.206.122@tcp:6/4 lens 488/448 e 0 to 1 dl 1788309605 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4034.357985] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 53 previous similar messages
[ 4098.015248] Lustre: lustre-OST0000-osc-ffff8bdc02b96800: Connection to lustre-OST0000 (at 192.168.206.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4098.031885] Lustre: Skipped 15 previous similar messages
[ 4206.234913] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 20:42:56 (1788309776)
[ 4254.770932] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 20:43:44 (1788309824)
[ 4257.859834] Lustre: Unmounted lustre-client
[ 4279.250625] Lustre: Mounted lustre-client - version 2.17.57_44_g3acc781
[ 4284.875909] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 20:44:14 (1788309854)
[ 4285.125192] LustreError: 56487:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4286.184795] LustreError: 56487:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4286.346824] Lustre: *** cfs_fail_loc=305, val=0***
[ 4309.542660] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 20:44:39 (1788309879)
[ 4309.970823] Lustre: Mounted lustre-client - version 2.17.57_44_g3acc781
[ 4312.253992] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4322.575779] Lustre: Unmounted lustre-client
[ 4328.598605] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 20:44:58 (1788309898)
[ 4406.113721] LustreError: lustre-MDT0000-mdc-ffff8bdc1802a000: operation mds_reint to node 192.168.206.122@tcp failed: rc = -19
[ 4406.119078] LustreError: Skipped 1 previous similar message
[ 4423.647201] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788309978/real 1788309978]  req@ffff8bdc07a0ed80 x1875174427623936/t0(0) o400->MGC192.168.206.122@tcp@192.168.206.122@tcp:26/25 lens 224/224 e 0 to 1 dl 1788309994 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4423.685325] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 81 previous similar messages
[ 4423.695431] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 4433.890386] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0xd13f7c8cbe798405 to 0xd13f7c8cbe7c3abe
[ 4433.909254] Lustre: MGC192.168.206.122@tcp: Connection restored to 192.168.206.122@tcp (at 192.168.206.122@tcp)
[ 4433.931090] Lustre: Skipped 24 previous similar messages
[ 4573.013421] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 20:49:02 (1788310142)
[ 4577.889543] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4590.570563] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 4590.593506] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0xd13f7c8cbe7c3abe to 0xd13f7c8cbe810283
[ 4590.679552] LustreError: lustre-MDT0000-mdc-ffff8bdc1802a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4606.173743] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 20:49:36 (1788310176)
[ 4606.688467] Lustre: Mounted lustre-client - version 2.17.57_44_g3acc781
[ 4613.709557] LustreError: lustre-OST0000-osc-ffff8bdc1802a000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4625.045919] Lustre: Unmounted lustre-client
[ 4630.005426] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 20:50:00 (1788310200)
[ 4635.859837] LustreError: lustre-MDT0000-mdc-ffff8bdc1802a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4635.875403] LustreError: 60871:0:(file.c:6166:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4641.241184] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 20:50:11 (1788310211)
[ 4641.363313] LustreError: 61572:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4645.060294] LustreError: lustre-MDT0000-mdc-ffff8bdc1802a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4645.066963] LustreError: 61593:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4645.075663] LustreError: 61572:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1298
[ 4645.078786] LustreError: 61572:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4645.117401] LustreError: 61598:0:(file.c:6166:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4646.176802] LustreError: 61604:0:(file.c:6166:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4646.194029] LustreError: 61604:0:(file.c:6166:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4648.507479] LustreError: 61626:0:(file.c:6166:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4648.513062] LustreError: 61626:0:(file.c:6166:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4653.041228] LustreError: 61671:0:(file.c:6166:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4653.050976] LustreError: 61671:0:(file.c:6166:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4662.872887] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 20:50:32 (1788310232)
[ 4698.906379] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4700.278827] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4710.166618] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 20:51:20 (1788310280)
[ 4717.550476] Lustre: lustre-OST0000-osc-ffff8bdc1802a000: Connection to lustre-OST0000 (at 192.168.206.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4717.566914] Lustre: Skipped 17 previous similar messages
[ 4744.989736] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4746.426464] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4755.860747] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 20:52:05 (1788310325)
[ 4817.266705] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4818.417633] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4825.292908] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 20:53:15 (1788310395)
[ 4857.120390] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4858.507033] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4863.767628] Lustre: Unmounted lustre-client
[ 4908.348740] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4910.394354] Lustre: Mounted lustre-client - version 2.17.57_44_g3acc781
[ 4915.107562] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 20:54:45 (1788310485)
[ 4917.127574] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4918.355766] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 20:54:48 (1788310488)
[ 4941.315676] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 20:55:11 (1788310511)
[ 4942.511621] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4943.787279] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 20:55:13 (1788310513)
[ 4944.731904] Lustre: *** cfs_fail_loc=805, val=0***
[ 4944.762901] Lustre: Mounted lustre-client - version 2.17.57_44_g3acc781
[ 4948.650764] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4970.481292] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 4970.506848] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0xd13f7c8cbe810caf to 0xd13f7c8cbe81104b
[ 4977.203076] Lustre: Unmounted lustre-client
[ 4983.774864] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 20:55:53 (1788310553)
[ 5011.484236] LustreError: 2352:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8bdc074c8a80 x1875174429677568/t94489280517(94489280517) o101->lustre-MDT0000-mdc-ffff8bdc02e16000@192.168.206.122@tcp:12/10 lens 648/608 e 0 to 0 dl 1788310598 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5018.245607] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5019.569767] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5026.958469] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 20:56:36 (1788310596)
[ 5028.083964] LustreError: lustre-OST0000-osc-ffff8bdc02e16000: operation ost_write to node 192.168.206.122@tcp failed: rc = -107
[ 5028.092131] LustreError: Skipped 3 previous similar messages
[ 5028.106744] LustreError: lustre-OST0000-osc-ffff8bdc02e16000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5028.116026] Lustre: 2355:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.122@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5028.139811] LustreError: 72791:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8bdc02e16000: namespace resource [0x240000400:0x4582:0x0].0x0 (ffff8bdc07600b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5034.824172] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 20:56:44 (1788310604)
[ 5036.112850] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5037.680903] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 20:56:47 (1788310607)
[ 5038.956527] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5040.363662] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 20:56:50 (1788310610)
[ 5041.684379] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5043.245118] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 20:56:53 (1788310613)
[ 5044.756193] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5046.329213] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 20:56:56 (1788310616)
[ 5047.809196] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5049.704247] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 20:56:59 (1788310619)
[ 5051.479467] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5052.992914] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 20:57:02 (1788310622)
[ 5054.662263] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5056.315733] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 20:57:06 (1788310626)
[ 5057.840671] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5059.514586] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 20:57:09 (1788310629)
[ 5061.146693] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5062.742848] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 20:57:12 (1788310632)
[ 5064.192920] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5065.579461] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 20:57:15 (1788310635)
[ 5066.972176] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5068.633814] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 20:57:18 (1788310638)
[ 5071.278926] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5073.222476] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 20:57:22 (1788310642)
[ 5093.387632] Lustre: MGC192.168.206.122@tcp: Connection restored to 192.168.206.122@tcp (at 192.168.206.122@tcp)
[ 5093.396684] Lustre: Skipped 17 previous similar messages
[ 5094.367164] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788310649/real 1788310649]  req@ffff8bdc07a01880 x1875174429691904/t0(0) o400->lustre-MDT0000-mdc-ffff8bdc02e16000@192.168.206.122@tcp:12/10 lens 224/224 e 0 to 1 dl 1788310665 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5094.420336] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 5102.818173] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 20:57:52 (1788310672)
[ 5131.476817] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 20:58:21 (1788310701)
[ 5131.849334] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5141.742563] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 20:58:31 (1788310711)
[ 5142.050388] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5151.707650] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 20:58:41 (1788310721)
[ 5151.996445] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5159.572574] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 20:58:49 (1788310729)
[ 5159.836654] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5168.608613] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 20:58:58 (1788310738)
[ 5168.885663] Lustre: *** cfs_fail_loc=510, val=3***
[ 5176.533376] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 20:59:06 (1788310746)
[ 5176.786761] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5185.259611] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 20:59:15 (1788310755)
[ 5185.494192] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5251.015225] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 21:00:20 (1788310820)
[ 5251.249662] LustreError: 82993:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5254.082732] LustreError: lustre-MDT0000-mdc-ffff8bdc02e16000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5254.119138] LustreError: 83008:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8bdc02e16000: inode [0x20000a042:0x8:0x0] mdc close failed: rc = -108
[ 5254.160059] LustreError: 83008:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8bdc02e16000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8bdc07a6e200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5254.201981] LustreError: 83013:0:(file.c:6166:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5254.215671] LustreError: 83013:0:(file.c:6166:ll_inode_revalidate_fini()) Skipped 20 previous similar messages
[ 5255.343238] LustreError: 82993:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5257.459180] LustreError: 83023:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5260.290979] LustreError: lustre-MDT0000-mdc-ffff8bdc02e16000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5260.311079] LustreError: 83038:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8bdc02e16000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8bdc07600e00) refcount nonzero (2) after lock cleanup; forcing cleanup.
[ 5260.330257] LustreError: 83038:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5261.551123] LustreError: 83023:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5268.272918] LustreError: lustre-MDT0000-mdc-ffff8bdc02e16000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5268.647436] LustreError: 83079:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5271.367242] LustreError: lustre-MDT0000-mdc-ffff8bdc02e16000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5271.412376] LustreError: 83098:0:(file.c:6166:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5272.727252] LustreError: 83079:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5272.733776] LustreError: 83079:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8bdc02e16000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5272.740137] LustreError: 83079:0:(file.c:251:ll_close_inode_openhandle()) Skipped 2 previous similar messages
[ 5278.317618] LustreError: 83153:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5281.154689] LustreError: lustre-MDT0000-mdc-ffff8bdc02e16000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5281.170651] LustreError: 83167:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8bdc02e16000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8bdc07a6e000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5281.194913] LustreError: 83167:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5282.383122] LustreError: 83153:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5289.104374] LustreError: lustre-MDT0000-mdc-ffff8bdc02e16000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5289.487420] LustreError: 83209:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5292.189810] LustreError: lustre-MDT0000-mdc-ffff8bdc02e16000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5292.205815] LustreError: 83235:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8bdc02e16000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8bdc07600000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5292.233374] LustreError: 83235:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5293.559125] LustreError: 83209:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5296.365538] LustreError: lustre-MDT0000-mdc-ffff8bdc02e16000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5297.752329] LustreError: 83248:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8bdc02e16000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5305.245792] LustreError: 83319:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5305.253677] LustreError: 83319:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5308.503169] LustreError: lustre-MDT0000-mdc-ffff8bdc02e16000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5308.562712] LustreError: 83341:0:(file.c:6166:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5308.568567] LustreError: 83341:0:(file.c:6166:ll_inode_revalidate_fini()) Skipped 53 previous similar messages
[ 5309.329724] LustreError: 83319:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5309.344171] LustreError: 83319:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5309.359388] LustreError: 83319:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8bdc02e16000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5318.500836] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 21:01:28 (1788310888)
[ 5335.007185] Lustre: lustre-MDT0000-mdc-ffff8bdc02e16000: Connection to lustre-MDT0000 (at 192.168.206.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5335.020049] Lustre: Skipped 18 previous similar messages
[ 5336.657597] LustreError: 79749:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5336.667922] LustreError: 79749:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5340.727256] LustreError: 79749:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5340.730642] LustreError: 79749:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5346.106844] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 21:01:56 (1788310916)
[ 5395.403448] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 21:02:45 (1788310965)
[ 5459.512574] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 21:03:49 (1788311029)
[ 5494.019683] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 21:04:23 (1788311063)
[ 5494.517588] Lustre: Mounted lustre-client - version 2.17.57_44_g3acc781
[ 5617.764576] Lustre: Unmounted lustre-client
[ 5624.036418] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 21:06:34 (1788311194)
[ 5624.627952] LustreError: 87643:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5627.018390] LustreError: lustre-OST0000-osc-ffff8bdc02e16000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5627.027902] LustreError: 87730:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8bdc02e16000: namespace resource [0x240000400:0x45ad:0x0].0x0 (ffff8bdc13636700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5627.043409] LustreError: 87730:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5627.255222] LustreError: 87643:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5627.263490] Lustre: 2356:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.122@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5632.970244] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 21:06:42 (1788311202)
[ 5696.670628] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 21:07:46 (1788311266)
[ 5697.909952] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5699.823963] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 21:07:49 (1788311269)
[ 5756.896076] Lustre: 89307:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788311272/real 1788311272]  req@ffff8bdc07a01500 x1875174429853312/t0(0) o101->lustre-MDT0000-mdc-ffff8bdc02e16000@192.168.206.122@tcp:12/10 lens 648/66320 e 0 to 1 dl 1788311327 ref 2 fl Rpc:XPQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 5756.935607] Lustre: 89307:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 5756.961634] Lustre: lustre-MDT0000-mdc-ffff8bdc02e16000: Connection restored to 192.168.206.122@tcp (at 192.168.206.122@tcp)
[ 5756.969303] Lustre: Skipped 15 previous similar messages
[ 5762.354572] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5763.742717] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 21:08:53 (1788311333)
[ 5820.730883] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 21:09:50 (1788311390)
[ 5822.956351] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5824.605319] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 21:09:54 (1788311394)
[ 5826.005410] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5827.927376] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 21:09:57 (1788311397)
[ 5862.596077] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 21:10:32 (1788311432)
[ 5872.737801] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5896.159444] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 5896.166409] LustreError: Skipped 2 previous similar messages
[ 5906.472787] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0xd13f7c8cbe811b03 to 0xd13f7c8cbe81361c
[ 5906.488050] Lustre: Skipped 2 previous similar messages
[ 5910.917354] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5912.373507] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5921.434041] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 21:11:31 (1788311491)
[ 5923.921405] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 5925.476849] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 21:11:35 (1788311495)
[ 5937.132254] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 5937.143520] Lustre: lustre-MDT0000-mdc-ffff8bdc02e16000: Connection to lustre-MDT0000 (at 192.168.206.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5937.155083] Lustre: Skipped 7 previous similar messages
[ 5947.023587] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 21:11:57 (1788311517)
[ 5968.884865] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0xd13f7c8cbe8137e3 to 0xd13f7c8cbe813a44
[ 5968.905457] Lustre: Skipped 1 previous similar message
[ 5970.279901] Lustre: 68254:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.122@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5995.812955] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 21:12:45 (1788311565)
[ 6002.618970] LustreError: lustre-OST0000-osc-ffff8bdc02e16000: operation ost_setattr to node 192.168.206.122@tcp failed: rc = -19
[ 6002.631596] LustreError: Skipped 9 previous similar messages
[ 6051.890980] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6054.257834] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6140.895534] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 6140.909315] LustreError: Skipped 1 previous similar message
[ 6140.958208] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0xd13f7c8cbe813a44 to 0xd13f7c8cbe814a12
[ 6151.716985] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6153.042658] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6188.629681] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6189.924654] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6310.146889] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 21:18:00 (1788311880)
[ 6370.983562] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6373.943943] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6626.018461] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 21:23:15 (1788312195)
[ 6770.662993] Lustre: lustre-MDT0000-mdc-ffff8bdc02e16000: Connection to lustre-MDT0000 (at 192.168.206.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6770.685698] Lustre: Skipped 5 previous similar messages
[ 6770.696402] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 6770.728314] LustreError: Skipped 1 previous similar message
[ 6770.771705] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0xd13f7c8cbe814e1e to 0xd13f7c8cbe90f096
[ 6770.787682] Lustre: Skipped 1 previous similar message
[ 6770.796943] Lustre: MGC192.168.206.122@tcp: Connection restored to 192.168.206.122@tcp (at 192.168.206.122@tcp)
[ 6770.814266] Lustre: Skipped 11 previous similar messages
[ 6805.026767] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 21:26:14 (1788312374)
[ 6806.535692] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6808.265967] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 21:26:18 (1788312378)
[ 6810.143776] LustreError: lustre-MDT0000-mdc-ffff8bdc02e16000: operation ldlm_enqueue to node 192.168.206.122@tcp failed: rc = -107
[ 6810.156101] LustreError: Skipped 127 previous similar messages
[ 6810.167438] LustreError: lustre-MDT0000-mdc-ffff8bdc02e16000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6818.149803] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 21:26:27 (1788312387)
[ 6985.202426] LustreError: lustre-OST0000-osc-ffff8bdc02e16000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6986.921774] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 21:29:16 (1788312556)
[ 7012.319177] Lustre: 2354:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788312563/real 1788312563]  req@ffff8bdc10f29c00 x1875174447554048/t0(0) o4->lustre-OST0000-osc-ffff8bdc02e16000@192.168.206.122@tcp:6/4 lens 4584/448 e 0 to 1 dl 1788312583 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7012.358152] Lustre: 2354:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 20 previous similar messages
[ 7028.785885] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 21:29:58 (1788312598)
[ 7030.412564] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7031.939817] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 21:30:01 (1788312601)
[ 7033.224282] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7034.670669] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 21:30:04 (1788312604)
[ 7067.037351] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 21:30:36 (1788312636)
[ 7104.500504] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 7104.522329] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0xd13f7c8cbe90f096 to 0xd13f7c8cbe9118dd
[ 7104.537206] Lustre: Skipped 1 previous similar message
[ 7116.336277] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 21:31:26 (1788312686)
[ 7117.556751] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7118.755252] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 21:31:28 (1788312688)
[ 7120.167284] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7121.439618] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 21:31:31 (1788312691)
[ 7131.354152] Lustre: Unmounted lustre-client
[ 7132.541841] Lustre: Mounted lustre-client - version 2.17.57_44_g3acc781
[ 7135.802759] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7152.991810] Lustre: 2353:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788312708/real 1788312708]  req@ffff8bdc069fd180 x1875174447739904/t0(0) o400->MGC192.168.206.122@tcp@192.168.206.122@tcp:26/25 lens 224/224 e 0 to 1 dl 1788312724 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7173.283126] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2