[ 1943.546633] LustreError: lustre-OST0000-osc-ffff9942a0aff800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1943.578524] Lustre: 2364:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.137@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -5)
[ 1950.847144] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 01:04:26 (1787375066)
[ 1953.846296] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1955.594154] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 01:04:31 (1787375071)
[ 1957.362539] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1959.428964] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 01:04:34 (1787375074)
[ 1983.464364] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 192.168.204.137@tcp) was lost; in progress operations using this service will fail
[ 1983.488204] Lustre: Evicted from MGS (at 192.168.204.137@tcp) after server handle changed from 0xec4213410820ccf0 to 0xec4213410820e35c
[ 2090.993852] LustreError: lustre-MDT0000-mdc-ffff9942a0aff800: operation ldlm_enqueue to node 192.168.204.137@tcp failed: rc = -19
[ 2091.009493] LustreError: Skipped 4 previous similar messages
[ 2112.480269] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 192.168.204.137@tcp) was lost; in progress operations using this service will fail
[ 2112.508861] Lustre: Evicted from MGS (at 192.168.204.137@tcp) after server handle changed from 0xec4213410820e35c to 0xec4213410822dbfd
[ 2114.234893] Lustre: 15933:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.137@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2139.267069] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 01:07:34 (1787375254)
[ 2139.788400] Lustre: *** cfs_fail_loc=305, val=0***
[ 2139.791413] Lustre: Skipped 2 previous similar messages
[ 2160.912990] LustreError: lustre-OST0001-osc-ffff9942a0aff800: operation ost_connect to node 192.168.204.137@tcp failed: rc = -75
[ 2182.112279] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 192.168.204.137@tcp) was lost; in progress operations using this service will fail
[ 2192.375077] Lustre: Evicted from MGS (at 192.168.204.137@tcp) after server handle changed from 0xec4213410822dbfd to 0xec4213410822e224
[ 2192.404532] Lustre: MGC192.168.204.137@tcp: Connection restored to 192.168.204.137@tcp (at 192.168.204.137@tcp)
[ 2192.416654] Lustre: Skipped 14 previous similar messages
[ 2193.713841] Lustre: 15933:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.137@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2195.970870] Lustre: lustre-MDT0000-mdc-ffff9942a0aff800: Connection to lustre-MDT0000 (at 192.168.204.137@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2195.997460] Lustre: Skipped 12 previous similar messages
[ 2212.218720] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2214.066769] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2224.859544] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 01:08:59 (1787375339)
[ 2243.560713] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 192.168.204.137@tcp) was lost; in progress operations using this service will fail
[ 2243.586945] Lustre: Evicted from MGS (at 192.168.204.137@tcp) after server handle changed from 0xec4213410822e224 to 0xec4213410822e828
[ 2243.659326] LustreError: lustre-MDT0000-mdc-ffff9942a0aff800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2273.514851] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 01:09:48 (1787375388)
[ 2288.934139] LustreError: lustre-OST0000-osc-ffff9942a0aff800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2309.697919] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 01:10:25 (1787375425)
[ 2323.817894] LustreError: lustre-MDT0000-mdc-ffff9942a0aff800: operation mds_reint to node 192.168.204.137@tcp failed: rc = -19
[ 2323.830269] LustreError: Skipped 2 previous similar messages
[ 2340.833707] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 192.168.204.137@tcp) was lost; in progress operations using this service will fail
[ 2351.103177] Lustre: Evicted from MGS (at 192.168.204.137@tcp) after server handle changed from 0xec4213410822e828 to 0xec421341082337cf
[ 2370.960595] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2373.597084] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2459.616233] Lustre: 2362:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787375560/real 1787375560]  req@ffff99428862a680 x1874196630633728/t0(0) o400->MGC192.168.204.137@tcp@192.168.204.137@tcp:26/25 lens 224/224 e 0 to 1 dl 1787375576 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2459.673221] Lustre: 2362:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 17 previous similar messages
[ 2459.692606] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 192.168.204.137@tcp) was lost; in progress operations using this service will fail
[ 2468.911712] Lustre: Evicted from MGS (at 192.168.204.137@tcp) after server handle changed from 0xec421341082337cf to 0xec4213410824b667
[ 2488.124306] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2490.815689] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2576.353103] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 192.168.204.137@tcp) was lost; in progress operations using this service will fail
[ 2586.607662] Lustre: Evicted from MGS (at 192.168.204.137@tcp) after server handle changed from 0xec4213410824b667 to 0xec42134108267e32
[ 2606.359558] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2609.054662] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2639.233708] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 01:15:54 (1787375754)
[ 2644.154872] LustreError: lustre-MDT0000-mdc-ffff9942a0aff800: operation ldlm_enqueue to node 192.168.204.137@tcp failed: rc = -19
[ 2644.171980] LustreError: Skipped 3 previous similar messages
[ 2662.433188] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 192.168.204.137@tcp) was lost; in progress operations using this service will fail
[ 2672.683193] Lustre: Evicted from MGS (at 192.168.204.137@tcp) after server handle changed from 0xec42134108267e32 to 0xec42134108274072
[ 2674.704595] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2715.238471] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2743.348785] Lustre: 15933:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.137@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2763.326847] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2800.160451] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 192.168.204.137@tcp) was lost; in progress operations using this service will fail
[ 2800.186899] LustreError: Skipped 2 previous similar messages
[ 2800.236720] Lustre: MGC192.168.204.137@tcp: Connection restored to 192.168.204.137@tcp (at 192.168.204.137@tcp)
[ 2800.241884] Lustre: Skipped 14 previous similar messages
[ 2801.409280] Lustre: 15933:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.137@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2816.822469] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2840.483443] Lustre: lustre-MDT0000-mdc-ffff9942a0aff800: Connection to lustre-MDT0000 (at 192.168.204.137@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2840.502766] Lustre: Skipped 7 previous similar messages
[ 2859.514601] Lustre: Evicted from MGS (at 192.168.204.137@tcp) after server handle changed from 0xec4213410827b764 to 0xec42134108283533
[ 2859.525458] Lustre: Skipped 3 previous similar messages
[ 2870.521084] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2931.799939] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3028.515254] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 01:22:24 (1787376144)
[ 3082.794495] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3085.099229] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3375.554508] LustreError: lustre-OST0000-osc-ffff9942a0aff800: operation ldlm_enqueue to node 192.168.204.137@tcp failed: rc = -19
[ 3375.574296] LustreError: Skipped 10 previous similar messages
[ 3415.737590] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3418.598664] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3710.154752] Lustre: lustre-OST0000-osc-ffff9942a0aff800: Connection to lustre-OST0000 (at 192.168.204.137@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3710.173830] Lustre: Skipped 4 previous similar messages
[ 3755.923113] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3758.482632] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4006.084859] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 01:38:41 (1787377121)
[ 4024.288806] Lustre: 50591:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787377125/real 1787377125]  req@ffff9942a03c5500 x1874196652540928/t0(0) o101->lustre-MDT0000-mdc-ffff9942a0aff800@192.168.204.137@tcp:12/10 lens 576/1152 e 0 to 1 dl 1787377141 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4024.339107] Lustre: 50591:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 4024.377813] Lustre: lustre-MDT0000-mdc-ffff9942a0aff800: Connection restored to 192.168.204.137@tcp (at 192.168.204.137@tcp)
[ 4024.388789] Lustre: Skipped 8 previous similar messages
[ 4032.257810] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 01:39:08 (1787377148)
[ 4058.175927] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 01:39:34 (1787377174)
[ 4083.984493] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 01:39:59 (1787377199)
[ 4084.379564] Lustre: Mounted lustre-client - version 2.17.57_46_g5381a3b
[ 4116.448355] Lustre: 2363:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787377217/real 1787377217]  req@ffff994298377800 x1874196652559616/t0(0) o400->MGC192.168.204.137@tcp@192.168.204.137@tcp:26/25 lens 224/224 e 0 to 1 dl 1787377233 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4116.478287] Lustre: 2363:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4116.503199] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 192.168.204.137@tcp) was lost; in progress operations using this service will fail
[ 4116.520701] LustreError: Skipped 3 previous similar messages
[ 4116.542682] Lustre: Evicted from MGS (at 192.168.204.137@tcp) after server handle changed from 0xec42134108297ded to 0xec421341083e6065
[ 4116.552310] Lustre: Skipped 2 previous similar messages
[ 4116.558722] Lustre: MGC192.168.204.137@tcp: Connection restored to 192.168.204.137@tcp (at 192.168.204.137@tcp)
[ 4116.571275] Lustre: Skipped 2 previous similar messages
[ 4117.627491] Lustre: 15933:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.137@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4143.430408] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4144.949223] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4147.396243] Lustre: Unmounted lustre-client
[ 4154.908568] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 01:41:10 (1787377270)
[ 4277.216737] Lustre: 2364:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787377378/real 1787377378]  req@ffff9942a03c7800 x1874196652595328/t0(0) o4->lustre-OST0000-osc-ffff9942a0aff800@192.168.204.137@tcp:6/4 lens 488/448 e 0 to 1 dl 1787377394 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4277.244775] Lustre: lustre-OST0000-osc-ffff9942a0aff800: Connection restored to 192.168.204.137@tcp (at 192.168.204.137@tcp)
[ 4277.248458] Lustre: 2364:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 60 previous similar messages
[ 4277.287479] Lustre: Skipped 7 previous similar messages
[ 4325.345053] Lustre: lustre-OST0000-osc-ffff9942a0aff800: Connection to lustre-OST0000 (at 192.168.204.137@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4325.369624] Lustre: Skipped 14 previous similar messages
[ 4449.217695] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 01:46:04 (1787377564)
[ 4499.257900] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 01:46:54 (1787377614)
[ 4503.893811] Lustre: Unmounted lustre-client
[ 4528.205722] Lustre: Mounted lustre-client - version 2.17.57_46_g5381a3b
[ 4535.468318] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 01:47:31 (1787377651)
[ 4536.129727] LustreError: 56516:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4537.187462] LustreError: 56516:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4537.434404] Lustre: *** cfs_fail_loc=305, val=0***
[ 4561.255454] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 01:47:56 (1787377676)
[ 4561.744960] Lustre: Mounted lustre-client - version 2.17.57_46_g5381a3b
[ 4564.341304] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4574.670887] Lustre: Unmounted lustre-client
[ 4582.177971] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 01:48:17 (1787377697)
[ 4669.286598] LustreError: lustre-MDT0000-mdc-ffff994288688800: operation mds_reint to node 192.168.204.137@tcp failed: rc = -19
[ 4669.292167] LustreError: Skipped 1 previous similar message
[ 4688.867253] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 192.168.204.137@tcp) was lost; in progress operations using this service will fail
[ 4688.881804] Lustre: Evicted from MGS (at 192.168.204.137@tcp) after server handle changed from 0xec421341083e7be7 to 0xec421341084133db
[ 4688.896762] Lustre: MGC192.168.204.137@tcp: Connection restored to 192.168.204.137@tcp (at 192.168.204.137@tcp)
[ 4688.913691] Lustre: Skipped 12 previous similar messages
[ 4834.496305] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 01:52:29 (1787377949)
[ 4839.878323] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4857.839217] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 192.168.204.137@tcp) was lost; in progress operations using this service will fail
[ 4857.870479] Lustre: Evicted from MGS (at 192.168.204.137@tcp) after server handle changed from 0xec421341084133db to 0xec4213410845f97e
[ 4857.938560] LustreError: lustre-MDT0000-mdc-ffff994288688800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4870.811981] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 01:53:06 (1787377986)
[ 4871.348239] Lustre: Mounted lustre-client - version 2.17.57_46_g5381a3b
[ 4878.715225] LustreError: lustre-OST0000-osc-ffff994288688800: operation ldlm_enqueue to node 192.168.204.137@tcp failed: rc = -107
[ 4878.750704] LustreError: lustre-OST0000-osc-ffff994288688800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4888.032852] Lustre: 2362:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787377989/real 1787377989]  req@ffff994298c89180 x1874196657249024/t0(0) o103->lustre-OST0000-osc-ffff9942a0ae6000@192.168.204.137@tcp:17/18 lens 328/224 e 0 to 1 dl 1787378005 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4888.063668] Lustre: 2362:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 86 previous similar messages
[ 4889.984536] Lustre: Unmounted lustre-client
[ 4897.258294] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 01:53:32 (1787378012)
[ 4903.736309] LustreError: lustre-MDT0000-mdc-ffff994288688800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4903.751343] LustreError: 60902:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4911.091250] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 01:53:46 (1787378026)
[ 4911.262332] LustreError: 61601:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4915.389864] LustreError: lustre-MDT0000-mdc-ffff994288688800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4915.410912] LustreError: 61621:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4915.423818] LustreError: 61601:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=842
[ 4915.433028] LustreError: 61601:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4915.459778] LustreError: 61626:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4916.532759] LustreError: 61632:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4916.549759] LustreError: 61632:0:(file.c:6170:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4918.812065] LustreError: 61655:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4918.823877] LustreError: 61655:0:(file.c:6170:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4923.425859] LustreError: 61700:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4923.436382] LustreError: 61700:0:(file.c:6170:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4933.529868] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 01:54:09 (1787378049)
[ 4941.291198] Lustre: lustre-OST0000-osc-ffff994288688800: Connection to lustre-OST0000 (at 192.168.204.137@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4941.305676] Lustre: Skipped 17 previous similar messages
[ 4969.900722] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4971.265336] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4980.263237] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 01:54:56 (1787378096)
[ 5014.393552] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5016.059176] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5025.427235] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 01:55:41 (1787378141)
[ 5087.772182] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5089.177657] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5098.127405] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 01:56:53 (1787378213)
[ 5137.556442] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5139.602842] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5146.134824] Lustre: Unmounted lustre-client
[ 5198.028619] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5201.186352] Lustre: Mounted lustre-client - version 2.17.57_46_g5381a3b
[ 5208.118955] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 01:58:43 (1787378323)
[ 5211.442514] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5213.077743] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 01:58:48 (1787378328)
[ 5247.684901] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 01:59:23 (1787378363)
[ 5249.099265] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5250.556836] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 01:59:26 (1787378366)
[ 5251.788910] Lustre: *** cfs_fail_loc=805, val=0***
[ 5251.841958] Lustre: Mounted lustre-client - version 2.17.57_46_g5381a3b
[ 5257.115900] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5277.687942] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 192.168.204.137@tcp) was lost; in progress operations using this service will fail
[ 5277.709840] Lustre: Evicted from MGS (at 192.168.204.137@tcp) after server handle changed from 0xec42134108460395 to 0xec421341084605b0
[ 5278.760610] Lustre: 68276:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.137@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5295.084379] Lustre: Unmounted lustre-client
[ 5304.193950] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 02:00:19 (1787378419)
[ 5327.868090] Lustre: MGC192.168.204.137@tcp: Connection restored to 192.168.204.137@tcp (at 192.168.204.137@tcp)
[ 5327.876853] Lustre: Skipped 14 previous similar messages
[ 5329.430044] LustreError: 2361:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff99428688c700 x1874196657333632/t85899345924(85899345924) o101->lustre-MDT0000-mdc-ffff9942a0af8800@192.168.204.137@tcp:12/10 lens 648/608 e 0 to 0 dl 1787378462 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5340.409741] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5341.819545] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5350.035523] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 02:01:06 (1787378466)
[ 5351.187788] LustreError: lustre-OST0000-osc-ffff9942a0af8800: operation ost_write to node 192.168.204.137@tcp failed: rc = -107
[ 5351.193124] LustreError: Skipped 2 previous similar messages
[ 5351.202121] LustreError: lustre-OST0000-osc-ffff9942a0af8800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5351.221206] Lustre: 2365:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.137@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5358.828485] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 02:01:14 (1787378474)
[ 5360.187268] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5361.781602] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 02:01:17 (1787378477)
[ 5363.179155] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5365.363471] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 02:01:20 (1787378480)
[ 5367.098760] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5368.633528] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 02:01:24 (1787378484)
[ 5370.186879] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5371.635945] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 02:01:27 (1787378487)
[ 5373.326706] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5374.964272] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 02:01:30 (1787378490)
[ 5376.151345] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5377.823354] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 02:01:33 (1787378493)
[ 5379.447726] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5380.905432] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 02:01:36 (1787378496)
[ 5382.336539] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5384.100259] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 02:01:39 (1787378499)
[ 5385.571493] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5387.180806] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 02:01:42 (1787378502)
[ 5388.564879] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5390.118558] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 02:01:46 (1787378506)
[ 5391.524558] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5393.241125] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 02:01:49 (1787378509)
[ 5395.702741] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5397.326229] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 02:01:53 (1787378513)
[ 5420.010897] Lustre: 68276:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.137@tcp: IR log lustre-cliir failed, not fatal: rc = -5
[ 5431.537124] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 02:02:26 (1787378546)
[ 5462.175893] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 02:02:57 (1787378577)
[ 5462.750241] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5473.672919] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 02:03:09 (1787378589)
[ 5473.966910] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5484.439260] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 02:03:20 (1787378600)
[ 5484.812046] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5492.897994] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 02:03:28 (1787378608)
[ 5493.264871] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5501.312901] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 02:03:37 (1787378617)
[ 5501.616639] Lustre: *** cfs_fail_loc=510, val=3***
[ 5512.012435] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 02:03:47 (1787378627)
[ 5512.426318] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5571.360320] Lustre: 2362:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787378633/real 1787378633]  req@ffff994298da4380 x1874196657375104/t0(0) o3->lustre-OST0001-osc-ffff9942a0af8800@192.168.204.137@tcp:6/4 lens 488/4536 e 0 to 1 dl 1787378688 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 5571.392095] Lustre: 2362:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 5571.404111] Lustre: lustre-OST0001-osc-ffff9942a0af8800: Connection to lustre-OST0001 (at 192.168.204.137@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5571.427851] Lustre: Skipped 8 previous similar messages
[ 5579.622585] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 02:04:54 (1787378694)
[ 5580.181974] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5645.759162] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 02:06:01 (1787378761)
[ 5646.077238] LustreError: 83044:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5649.065435] LustreError: lustre-MDT0000-mdc-ffff9942a0af8800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5649.106447] LustreError: 83058:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9942a0af8800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9942887d2700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5650.176300] LustreError: 83044:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5652.304966] LustreError: 83074:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5655.277598] LustreError: lustre-MDT0000-mdc-ffff9942a0af8800: operation ldlm_enqueue to node 192.168.204.137@tcp failed: rc = -107
[ 5655.287581] LustreError: Skipped 1 previous similar message
[ 5655.303980] LustreError: lustre-MDT0000-mdc-ffff9942a0af8800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5655.321851] LustreError: 83088:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9942a0af8800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff994298dd0600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5655.352727] LustreError: 83088:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5655.363175] LustreError: 83093:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5655.373219] LustreError: 83093:0:(file.c:6170:ll_inode_revalidate_fini()) Skipped 20 previous similar messages
[ 5656.393117] LustreError: 83074:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5663.382856] LustreError: lustre-MDT0000-mdc-ffff9942a0af8800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5663.770990] LustreError: 83130:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5666.615388] LustreError: lustre-MDT0000-mdc-ffff9942a0af8800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5666.710294] LustreError: 83149:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5667.842096] LustreError: 83130:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5667.854472] LustreError: 83130:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9942a0af8800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5668.895436] LustreError: 83167:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5668.904295] LustreError: 83167:0:(file.c:6170:ll_inode_revalidate_fini()) Skipped 12 previous similar messages
[ 5673.629792] LustreError: 83205:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5676.505069] LustreError: lustre-MDT0000-mdc-ffff9942a0af8800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5676.532733] LustreError: 83219:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9942a0af8800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9942a2b2fe00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5676.559045] LustreError: 83219:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5677.696181] LustreError: 83205:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5684.605556] LustreError: lustre-MDT0000-mdc-ffff9942a0af8800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5685.068199] LustreError: 83261:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5688.073197] LustreError: lustre-MDT0000-mdc-ffff9942a0af8800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5688.117529] LustreError: 83276:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9942a0af8800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9942887d2300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5688.148969] LustreError: 83276:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5688.171174] LustreError: 83281:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5688.181369] LustreError: 83281:0:(file.c:6170:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5689.144505] LustreError: 83261:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5692.035991] LustreError: lustre-MDT0000-mdc-ffff9942a0af8800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5693.305183] LustreError: 83289:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9942a0af8800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5701.278569] LustreError: 83360:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5701.289451] LustreError: 83360:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5704.695535] LustreError: lustre-MDT0000-mdc-ffff9942a0af8800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5704.781964] LustreError: 83385:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5704.791470] LustreError: 83385:0:(file.c:6170:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 5705.280155] LustreError: 83360:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5705.282780] LustreError: 83360:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5705.292280] LustreError: 83360:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9942a0af8800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5716.626461] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 02:07:12 (1787378832)
[ 5736.027431] LustreError: 68273:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5736.044745] LustreError: 68273:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5740.120178] LustreError: 68273:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5740.129825] LustreError: 68273:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5746.445462] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 02:07:42 (1787378862)
[ 5799.650349] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 02:08:34 (1787378914)
[ 5866.117905] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 02:09:41 (1787378981)
[ 5901.185639] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 02:10:16 (1787379016)
[ 5901.782303] Lustre: Mounted lustre-client - version 2.17.57_46_g5381a3b
[ 6027.212123] Lustre: Unmounted lustre-client
[ 6036.529979] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 02:12:32 (1787379152)
[ 6037.441711] LustreError: 87690:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6040.068113] LustreError: lustre-OST0000-osc-ffff9942a0af8800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6040.079315] LustreError: 87771:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9942a0af8800: namespace resource [0x240000400:0x3e6d:0x0].0x0 (ffff9942a2b2f000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6040.110425] LustreError: 87771:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6040.617230] LustreError: 87690:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6040.628192] Lustre: 2365:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.137@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6040.647800] Lustre: lustre-OST0000-osc-ffff9942a0af8800: Connection restored to 192.168.204.137@tcp (at 192.168.204.137@tcp)
[ 6040.660770] Lustre: Skipped 16 previous similar messages
[ 6047.931975] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 02:12:43 (1787379163)
[ 6117.041709] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 02:13:52 (1787379232)
[ 6119.021785] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6121.427344] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 02:13:56 (1787379236)
[ 6180.832177] Lustre: 89352:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787379241/real 1787379241]  req@ffff994298da5880 x1874196657516800/t0(0) o101->lustre-MDT0000-mdc-ffff9942a0af8800@192.168.204.137@tcp:12/10 lens 648/66320 e 0 to 1 dl 1787379297 ref 2 fl Rpc:XPQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 6180.846849] Lustre: 89352:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 6180.851282] Lustre: lustre-MDT0000-mdc-ffff9942a0af8800: Connection to lustre-MDT0000 (at 192.168.204.137@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6180.858070] Lustre: Skipped 15 previous similar messages
[ 6191.113340] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6193.683419] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 02:15:09 (1787379309)
[ 6253.702678] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 02:16:09 (1787379369)
[ 6256.652342] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6259.093585] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 02:16:14 (1787379374)
[ 6261.003656] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6263.088943] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 02:16:18 (1787379378)
[ 6303.425451] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 02:16:59 (1787379419)
[ 6316.116962] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6343.662485] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 192.168.204.137@tcp) was lost; in progress operations using this service will fail
[ 6343.674195] LustreError: Skipped 3 previous similar messages
[ 6343.694241] Lustre: Evicted from MGS (at 192.168.204.137@tcp) after server handle changed from 0xec4213410846126e to 0xec42134108462aa8
[ 6343.707409] Lustre: Skipped 3 previous similar messages
[ 6358.529381] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6360.204326] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6373.393907] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 02:18:08 (1787379488)
[ 6376.614316] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6379.306866] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 02:18:14 (1787379494)
[ 6394.853754] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 192.168.204.137@tcp) was lost; in progress operations using this service will fail
[ 6394.876785] Lustre: Evicted from MGS (at 192.168.204.137@tcp) after server handle changed from 0xec42134108462aa8 to 0xec42134108462f4e
[ 6411.860927] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 02:18:47 (1787379527)
[ 6442.346487] Lustre: 68276:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.137@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6464.772795] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 02:19:40 (1787379580)
[ 6472.487301] LustreError: lustre-OST0000-osc-ffff9942a0af8800: operation ost_setattr to node 192.168.204.137@tcp failed: rc = -19
[ 6472.492877] LustreError: Skipped 7 previous similar messages
[ 6527.190838] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6531.925749] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6619.108091] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 192.168.204.137@tcp) was lost; in progress operations using this service will fail
[ 6619.145792] LustreError: Skipped 1 previous similar message
[ 6619.173534] Lustre: Evicted from MGS (at 192.168.204.137@tcp) after server handle changed from 0xec4213410846315b to 0xec42134108464096
[ 6619.181190] Lustre: Skipped 1 previous similar message
[ 6634.458992] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6635.925786] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6659.068403] Lustre: MGC192.168.204.137@tcp: Connection restored to 192.168.204.137@tcp (at 192.168.204.137@tcp)
[ 6659.077448] Lustre: Skipped 11 previous similar messages
[ 6660.333643] Lustre: 68276:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.137@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6686.228793] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6687.799507] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6814.735287] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 02:25:30 (1787379930)
[ 6826.754402] Lustre: lustre-OST0000-osc-ffff9942a0af8800: Connection to lustre-OST0000 (at 192.168.204.137@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6826.779053] Lustre: Skipped 7 previous similar messages
[ 6877.107472] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6880.822831] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7123.307568] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 02:30:39 (1787380239)
[ 7273.451765] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 192.168.204.137@tcp) was lost; in progress operations using this service will fail
[ 7273.471847] LustreError: Skipped 1 previous similar message
[ 7273.498889] Lustre: Evicted from MGS (at 192.168.204.137@tcp) after server handle changed from 0xec4213410846449b to 0xec4213410855e824
[ 7273.513502] Lustre: Skipped 1 previous similar message
[ 7273.520665] Lustre: MGC192.168.204.137@tcp: Connection restored to 192.168.204.137@tcp (at 192.168.204.137@tcp)
[ 7273.538702] Lustre: Skipped 1 previous similar message
[ 7305.236474] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 02:33:41 (1787380421)
[ 7306.707565] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7308.538624] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 02:33:44 (1787380424)
[ 7310.348215] LustreError: lustre-MDT0000-mdc-ffff9942a0af8800: operation ldlm_enqueue to node 192.168.204.137@tcp failed: rc = -107
[ 7310.356893] LustreError: Skipped 254 previous similar messages
[ 7310.378923] LustreError: lustre-MDT0000-mdc-ffff9942a0af8800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7317.230857] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 02:33:53 (1787380433)
[ 7484.391544] LustreError: lustre-OST0000-osc-ffff9942a0af8800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7486.259740] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 02:36:42 (1787380602)
[ 7511.008194] Lustre: 2362:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787380607/real 1787380607]  req@ffff994286f4b800 x1874196675238528/t0(0) o4->lustre-OST0000-osc-ffff9942a0af8800@192.168.204.137@tcp:6/4 lens 4584/448 e 0 to 1 dl 1787380627 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7511.040102] Lustre: 2362:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 17 previous similar messages
[ 7511.046445] Lustre: lustre-OST0000-osc-ffff9942a0af8800: Connection to lustre-OST0000 (at 192.168.204.137@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7511.061101] Lustre: Skipped 3 previous similar messages
[ 7525.916498] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 02:37:22 (1787380642)
[ 7527.092131] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7528.274876] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 02:37:24 (1787380644)
[ 7529.351882] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7530.995508] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 02:37:26 (1787380646)
[ 7562.713574] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 02:37:58 (1787380678)
[ 7598.058705] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 192.168.204.137@tcp) was lost; in progress operations using this service will fail
[ 7598.076210] Lustre: Evicted from MGS (at 192.168.204.137@tcp) after server handle changed from 0xec4213410855e824 to 0xec42134108560fa7
[ 7598.086653] Lustre: Skipped 1 previous similar message
[ 7613.255864] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 02:38:48 (1787380728)
[ 7614.933948] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7617.113893] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 02:38:52 (1787380732)
[ 7618.743178] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7620.177587] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 02:38:56 (1787380736)
[ 7632.065214] Lustre: Unmounted lustre-client
[ 7633.649452] Lustre: Mounted lustre-client - version 2.17.57_46_g5381a3b
[ 7636.190301] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7655.392185] Lustre: 2363:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787380756/real 1787380756]  req@ffff994290392d80 x1874196675424896/t0(0) o400->MGC192.168.204.137@tcp@192.168.204.137@tcp:26/25 lens 224/224 e 0 to 1 dl 1787380772 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7666.090884] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2