[ 1868.176361] LustreError: 34761:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 1872.905513] LustreError: lustre-OST0000-osc-ffff9f0e09334000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1878.183197] LustreError: 34761:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 1878.199948] Lustre: 2356:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.136@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1878.202095] LustreError: 34761:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 1888.247152] LustreError: 34761:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 1888.264821] Lustre: 2356:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.136@tcp:/lustre/fid: [0x200000404:0x36:0x0]// may get corrupted (rc -108)
[ 1895.010877] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 09:06:06 (1790946366)
[ 1896.862818] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1898.857706] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 09:06:09 (1790946369)
[ 1900.317289] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1901.833474] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 09:06:12 (1790946372)
[ 1905.515872] LustreError: lustre-MDT0000-mdc-ffff9f0e09334000: operation ldlm_enqueue to node 192.168.203.136@tcp failed: rc = -19
[ 1922.015390] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 1932.215459] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0x6bafb87c027ed820 to 0x6bafb87c027ef2ec
[ 2027.415304] LustreError: lustre-MDT0000-mdc-ffff9f0e09334000: operation ldlm_enqueue to node 192.168.203.136@tcp failed: rc = -19
[ 2027.433621] LustreError: Skipped 1 previous similar message
[ 2043.871272] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 2054.061573] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0x6bafb87c027ef2ec to 0x6bafb87c0280f32e
[ 2070.204415] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 09:09:01 (1790946541)
[ 2070.532802] Lustre: *** cfs_fail_loc=305, val=0***
[ 2070.536540] Lustre: Skipped 4 previous similar messages
[ 2110.239448] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 2110.264447] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0x6bafb87c0280f32e to 0x6bafb87c0280f58f
[ 2124.141472] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2126.097696] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2135.174949] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 09:10:05 (1790946605)
[ 2151.425564] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 2151.436755] Lustre: lustre-MDT0000-mdc-ffff9f0e09334000: Connection to lustre-MDT0000 (at 192.168.203.136@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2151.457870] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0x6bafb87c0280f58f to 0x6bafb87c0280fc2d
[ 2151.471818] Lustre: Skipped 12 previous similar messages
[ 2151.565708] LustreError: lustre-MDT0000-mdc-ffff9f0e09334000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2174.858264] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 09:10:45 (1790946645)
[ 2190.654318] LustreError: lustre-OST0000-osc-ffff9f0e09334000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2190.702129] Lustre: lustre-OST0000-osc-ffff9f0e09334000: Connection restored to 192.168.203.136@tcp (at 192.168.203.136@tcp)
[ 2190.719833] Lustre: Skipped 16 previous similar messages
[ 2208.673821] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 09:11:19 (1790946679)
[ 2222.263594] LustreError: lustre-MDT0000-mdc-ffff9f0e09334000: operation mds_reint to node 192.168.203.136@tcp failed: rc = -19
[ 2239.455230] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 2249.645104] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0x6bafb87c0280fc2d to 0x6bafb87c02814ce5
[ 2266.979074] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2269.158880] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2334.144230] LustreError: lustre-MDT0000-mdc-ffff9f0e09334000: operation mds_close to node 192.168.203.136@tcp failed: rc = -19
[ 2334.159641] LustreError: Skipped 2 previous similar messages
[ 2351.071135] Lustre: 2357:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790946807/real 1790946807]  req@ffff9f0e097dea00 x1877941493372928/t0(0) o400->MGC192.168.203.136@tcp@192.168.203.136@tcp:26/25 lens 224/224 e 0 to 1 dl 1790946823 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2351.104283] Lustre: 2357:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 2351.120463] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 2361.523970] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0x6bafb87c02814ce5 to 0x6bafb87c028370dd
[ 2368.639146] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2370.691833] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2453.985346] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 2464.165389] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0x6bafb87c028370dd to 0x6bafb87c02856bf4
[ 2481.360105] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2483.297020] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2511.649881] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 09:16:22 (1790946982)
[ 2515.674121] LustreError: lustre-MDT0000-mdc-ffff9f0e09334000: operation mds_reint to node 192.168.203.136@tcp failed: rc = -19
[ 2515.684098] LustreError: Skipped 2 previous similar messages
[ 2534.879461] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 2534.895651] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0x6bafb87c02856bf4 to 0x6bafb87c028651f1
[ 2543.100345] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2567.487809] Lustre: 15927:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.136@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2583.595601] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2619.550764] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2661.228343] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2702.303253] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 2702.325042] LustreError: Skipped 3 previous similar messages
[ 2712.692156] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0x6bafb87c0286b1ac to 0x6bafb87c02873379
[ 2712.713041] Lustre: Skipped 3 previous similar messages
[ 2714.086277] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2771.004763] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2804.245928] LustreError: lustre-MDT0000-mdc-ffff9f0e09334000: operation ldlm_enqueue to node 192.168.203.136@tcp failed: rc = -19
[ 2804.251132] Lustre: lustre-MDT0000-mdc-ffff9f0e09334000: Connection to lustre-MDT0000 (at 192.168.203.136@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2804.263941] LustreError: Skipped 8 previous similar messages
[ 2804.295763] Lustre: Skipped 9 previous similar messages
[ 2830.762709] Lustre: MGC192.168.203.136@tcp: Connection restored to 192.168.203.136@tcp (at 192.168.203.136@tcp)
[ 2830.782310] Lustre: Skipped 17 previous similar messages
[ 2859.116765] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 09:22:10 (1790947330)
[ 2915.307136] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2917.344631] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3248.824958] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3251.027254] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3538.896572] LustreError: lustre-OST0000-osc-ffff9f0e09334000: operation ldlm_enqueue to node 192.168.203.136@tcp failed: rc = -107
[ 3538.910938] LustreError: Skipped 2 previous similar messages
[ 3538.916409] Lustre: lustre-OST0000-osc-ffff9f0e09334000: Connection to lustre-OST0000 (at 192.168.203.136@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3538.926183] Lustre: Skipped 2 previous similar messages
[ 3586.550778] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3589.010640] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3833.301844] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 09:38:24 (1790948304)
[ 3851.231170] Lustre: 50575:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790948307/real 1790948307]  req@ffff9f0e18404e00 x1877941515452800/t0(0) o101->lustre-MDT0000-mdc-ffff9f0e09334000@192.168.203.136@tcp:12/10 lens 576/1152 e 0 to 1 dl 1790948323 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3851.297297] Lustre: 50575:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 3851.344219] Lustre: lustre-MDT0000-mdc-ffff9f0e09334000: Connection restored to 192.168.203.136@tcp (at 192.168.203.136@tcp)
[ 3851.353562] Lustre: Skipped 2 previous similar messages
[ 3859.333730] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 09:38:50 (1790948330)
[ 3887.869295] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 09:39:18 (1790948358)
[ 3914.571480] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 09:39:45 (1790948385)
[ 3914.989207] Lustre: Mounted lustre-client - version 2.17.59_2_gbfaf0e7
[ 3945.887132] Lustre: 2356:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790948402/real 1790948402]  req@ffff9f0e0946df80 x1877941515470592/t0(0) o400->lustre-MDT0000-mdc-ffff9f0e09334000@192.168.203.136@tcp:12/10 lens 224/224 e 0 to 1 dl 1790948418 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3945.900832] Lustre: 2356:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3945.909611] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 3945.918366] LustreError: Skipped 2 previous similar messages
[ 3956.218587] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0x6bafb87c02888a87 to 0x6bafb87c029d3f63
[ 3956.233231] Lustre: Skipped 2 previous similar messages
[ 3961.445267] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3962.960066] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3964.967428] Lustre: Unmounted lustre-client
[ 3970.744806] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 09:40:41 (1790948441)
[ 4108.767240] Lustre: 2357:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790948565/real 1790948565]  req@ffff9f0e0946c700 x1877941515502208/t0(0) o4->lustre-OST0000-osc-ffff9f0e09334000@192.168.203.136@tcp:6/4 lens 488/448 e 0 to 1 dl 1790948581 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4108.806770] Lustre: 2357:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 55 previous similar messages
[ 4141.343279] Lustre: lustre-OST0000-osc-ffff9f0e09334000: Connection to lustre-OST0000 (at 192.168.203.136@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4141.370104] Lustre: Skipped 14 previous similar messages
[ 4267.649306] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 09:45:38 (1790948738)
[ 4316.860940] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 09:46:27 (1790948787)
[ 4320.209801] Lustre: Unmounted lustre-client
[ 4344.630954] Lustre: Mounted lustre-client - version 2.17.59_2_gbfaf0e7
[ 4353.434922] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 09:47:03 (1790948823)
[ 4354.113730] LustreError: 56508:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4355.171920] LustreError: 56508:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4355.531743] Lustre: *** cfs_fail_loc=305, val=0***
[ 4378.299564] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 09:47:29 (1790948849)
[ 4379.111258] Lustre: Mounted lustre-client - version 2.17.59_2_gbfaf0e7
[ 4381.818458] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4392.157647] Lustre: Unmounted lustre-client
[ 4400.284413] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 09:47:51 (1790948871)
[ 4485.496792] LustreError: lustre-MDT0000-mdc-ffff9f0e08dd9000: operation mds_reint to node 192.168.203.136@tcp failed: rc = -19
[ 4503.007585] Lustre: 2356:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790948959/real 1790948959]  req@ffff9f0e04685880 x1877941518180736/t0(0) o400->MGC192.168.203.136@tcp@192.168.203.136@tcp:26/25 lens 224/224 e 0 to 1 dl 1790948975 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4503.034419] Lustre: 2356:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 89 previous similar messages
[ 4503.039163] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 4513.260728] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0x6bafb87c029d5837 to 0x6bafb87c02a00f05
[ 4513.275054] Lustre: MGC192.168.203.136@tcp: Connection restored to 192.168.203.136@tcp (at 192.168.203.136@tcp)
[ 4513.288105] Lustre: Skipped 23 previous similar messages
[ 4671.775886] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 09:52:22 (1790949142)
[ 4676.897642] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4691.445553] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 4691.490346] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0x6bafb87c02a00f05 to 0x6bafb87c02a4d6ed
[ 4691.536055] LustreError: lustre-MDT0000-mdc-ffff9f0e08dd9000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4710.108242] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 09:53:00 (1790949180)
[ 4710.567415] Lustre: Mounted lustre-client - version 2.17.59_2_gbfaf0e7
[ 4718.343983] LustreError: lustre-OST0000-osc-ffff9f0e08dd9000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4729.348657] Lustre: Unmounted lustre-client
[ 4736.073485] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 09:53:26 (1790949206)
[ 4743.204573] Lustre: lustre-MDT0000-mdc-ffff9f0e08dd9000: Connection to lustre-MDT0000 (at 192.168.203.136@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4743.217059] Lustre: Skipped 14 previous similar messages
[ 4743.228777] LustreError: lustre-MDT0000-mdc-ffff9f0e08dd9000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4743.238407] LustreError: 60900:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4743.239587] LustreError: 60984:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9f0e08dd9000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9f0e0718e200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4753.460652] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 09:53:43 (1790949223)
[ 4753.597317] LustreError: 61598:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4759.007132] LustreError: 61598:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4759.023249] LustreError: 61598:0:(import.c:756:ptlrpc_connect_import_locked()) already connected
[ 4759.653720] LustreError: lustre-MDT0000-mdc-ffff9f0e08dd9000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4759.682955] LustreError: 61622:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4759.786697] LustreError: 61627:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4760.898603] LustreError: 61633:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4760.920811] LustreError: 61633:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4763.333847] LustreError: 61655:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4763.344990] LustreError: 61655:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4768.180510] LustreError: 61700:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4768.212969] LustreError: 61700:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4779.171869] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 09:54:09 (1790949249)
[ 4822.283158] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4823.965604] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4834.891497] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 09:55:05 (1790949305)
[ 4877.892628] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4880.379241] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4892.567965] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 09:56:02 (1790949362)
[ 4963.071413] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4965.036509] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4975.105917] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 09:57:25 (1790949445)
[ 5015.869547] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5017.250951] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5023.303367] Lustre: Unmounted lustre-client
[ 5083.215344] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5086.331804] Lustre: Mounted lustre-client - version 2.17.59_2_gbfaf0e7
[ 5093.866941] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 09:59:24 (1790949564)
[ 5096.131688] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5098.468485] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 09:59:29 (1790949569)
[ 5134.440864] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 10:00:04 (1790949604)
[ 5136.631498] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5138.845818] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 10:00:09 (1790949609)
[ 5140.362375] Lustre: *** cfs_fail_loc=805, val=0***
[ 5140.421841] Lustre: Mounted lustre-client - version 2.17.59_2_gbfaf0e7
[ 5146.287186] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5167.071155] Lustre: 2356:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790949623/real 1790949623]  req@ffff9f0e08ace680 x1877941520230400/t0(0) o400->MGC192.168.203.136@tcp@192.168.203.136@tcp:26/25 lens 224/224 e 0 to 1 dl 1790949639 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5167.145556] Lustre: 2356:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5167.162541] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 5177.322900] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0x6bafb87c02a4e127 to 0x6bafb87c02a4e437
[ 5177.338156] Lustre: MGC192.168.203.136@tcp: Connection restored to 192.168.203.136@tcp (at 192.168.203.136@tcp)
[ 5177.349198] Lustre: Skipped 10 previous similar messages
[ 5187.200325] Lustre: Unmounted lustre-client
[ 5197.410265] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 10:01:07 (1790949667)
[ 5225.370621] LustreError: 2353:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff9f0e04420380 x1877941520242944/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffff9f0e11057800@192.168.203.136@tcp:12/10 lens 648/608 e 0 to 0 dl 1790949713 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5235.897450] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5237.450882] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5246.416889] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 10:01:56 (1790949716)
[ 5248.375043] LustreError: lustre-OST0000-osc-ffff9f0e11057800: operation ost_write to node 192.168.203.136@tcp failed: rc = -107
[ 5248.387660] LustreError: Skipped 3 previous similar messages
[ 5248.405638] LustreError: lustre-OST0000-osc-ffff9f0e11057800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5248.427980] Lustre: 2355:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.136@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5248.472750] LustreError: 72822:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0000-osc-ffff9f0e11057800: namespace resource [0x240000400:0x3f42:0x0].0x0 (ffff9f0e0462ee00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5258.712525] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 10:02:09 (1790949729)
[ 5260.313390] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5262.642725] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 10:02:13 (1790949733)
[ 5264.397518] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5266.697662] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 10:02:17 (1790949737)
[ 5268.505615] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5270.601092] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 10:02:21 (1790949741)
[ 5272.149322] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5274.051582] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 10:02:24 (1790949744)
[ 5275.708630] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5277.604697] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 10:02:28 (1790949748)
[ 5279.213854] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5281.193466] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 10:02:31 (1790949751)
[ 5282.584872] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5284.606821] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 10:02:35 (1790949755)
[ 5286.529331] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5288.811745] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 10:02:39 (1790949759)
[ 5290.467513] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5292.442671] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 10:02:43 (1790949763)
[ 5293.942179] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5295.845545] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 10:02:46 (1790949766)
[ 5297.381886] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5299.038384] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 10:02:50 (1790949770)
[ 5301.927147] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5303.741075] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 10:02:54 (1790949774)
[ 5338.617842] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 10:03:29 (1790949809)
[ 5368.141273] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 10:03:59 (1790949839)
[ 5368.572124] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5378.961719] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 10:04:09 (1790949849)
[ 5379.426280] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5390.165710] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 10:04:21 (1790949861)
[ 5390.456186] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5400.456696] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 10:04:31 (1790949871)
[ 5400.753830] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5410.414346] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 10:04:41 (1790949881)
[ 5410.761993] Lustre: *** cfs_fail_loc=510, val=3***
[ 5419.429801] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 10:04:50 (1790949890)
[ 5419.844507] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5479.329054] Lustre: lustre-OST0001-osc-ffff9f0e11057800: Connection to lustre-OST0001 (at 192.168.203.136@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5479.354553] Lustre: Skipped 10 previous similar messages
[ 5487.398275] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 10:05:57 (1790949957)
[ 5487.897934] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5554.295457] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 10:07:05 (1790950025)
[ 5555.168496] LustreError: 83037:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5558.144930] LustreError: lustre-MDT0000-mdc-ffff9f0e11057800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5558.182702] LustreError: 83052:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9f0e11057800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9f0e0718e200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5559.251021] LustreError: 83037:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5561.430330] LustreError: 83068:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5564.228125] LustreError: lustre-MDT0000-mdc-ffff9f0e11057800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5564.254037] LustreError: 83082:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9f0e11057800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9f0e0718e900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5564.278625] LustreError: 83082:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5564.290107] LustreError: 83087:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5564.299703] LustreError: 83087:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5565.431141] LustreError: 83068:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5572.628505] LustreError: lustre-MDT0000-mdc-ffff9f0e11057800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5573.254680] LustreError: 83123:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5576.134488] LustreError: lustre-MDT0000-mdc-ffff9f0e11057800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5576.191338] LustreError: 83143:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5577.337109] LustreError: 83123:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5577.362328] LustreError: 83123:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff9f0e11057800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5578.440754] LustreError: 83160:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5578.454872] LustreError: 83160:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 12 previous similar messages
[ 5583.265570] LustreError: 83198:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5586.231251] LustreError: lustre-MDT0000-mdc-ffff9f0e11057800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5586.246404] LustreError: 83212:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9f0e11057800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9f0e0718e900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5586.277857] LustreError: 83212:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5587.335119] LustreError: 83198:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5594.332784] LustreError: lustre-MDT0000-mdc-ffff9f0e11057800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5594.869272] LustreError: 83254:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5597.936323] LustreError: lustre-MDT0000-mdc-ffff9f0e11057800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5597.957632] LustreError: 83268:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9f0e11057800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9f0e0718ee00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5598.001656] LustreError: 83268:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5598.951137] LustreError: 83254:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5601.948394] LustreError: lustre-MDT0000-mdc-ffff9f0e11057800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5602.035237] LustreError: 83301:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5602.042673] LustreError: 83301:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5603.148097] LustreError: 83281:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff9f0e11057800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5611.120242] LustreError: 83352:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5611.139425] LustreError: 83352:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5614.967635] LustreError: lustre-MDT0000-mdc-ffff9f0e11057800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5615.053926] LustreError: 83374:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5615.059545] LustreError: 83374:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5615.207121] LustreError: 83352:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5615.213130] LustreError: 83352:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5615.226858] LustreError: 83352:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff9f0e11057800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5626.059893] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 10:08:17 (1790950097)
[ 5644.777867] LustreError: 68266:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5644.784400] LustreError: 68266:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5648.847230] LustreError: 68266:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5648.864975] LustreError: 68266:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5655.413460] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 10:08:46 (1790950126)
[ 5704.669339] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 10:09:35 (1790950175)
[ 5768.265190] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 10:10:39 (1790950239)
[ 5803.871777] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 10:11:14 (1790950274)
[ 5804.475744] Lustre: Mounted lustre-client - version 2.17.59_2_gbfaf0e7
[ 5928.672890] Lustre: Unmounted lustre-client
[ 5936.911812] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 10:13:27 (1790950407)
[ 5937.778386] LustreError: 87679:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5940.520903] LustreError: lustre-OST0000-osc-ffff9f0e11057800: operation ost_statfs to node 192.168.203.136@tcp failed: rc = -107
[ 5940.543188] LustreError: Skipped 9 previous similar messages
[ 5940.560706] LustreError: lustre-OST0000-osc-ffff9f0e11057800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5940.569503] LustreError: 87766:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0000-osc-ffff9f0e11057800: namespace resource [0x240000400:0x3f6d:0x0].0x0 (ffff9f0e07050900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5940.587189] LustreError: 87766:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5940.839712] LustreError: 87679:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5940.850887] Lustre: 2354:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.136@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5940.891541] Lustre: lustre-OST0000-osc-ffff9f0e11057800: Connection restored to 192.168.203.136@tcp (at 192.168.203.136@tcp)
[ 5940.903048] Lustre: Skipped 18 previous similar messages
[ 5947.248986] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 10:13:38 (1790950418)
[ 6004.703164] Lustre: 88345:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790950421/real 1790950421]  req@ffff9f0e0952a300 x1877941520418048/t0(0) o101->lustre-MDT0000-mdc-ffff9f0e11057800@192.168.203.136@tcp:12/10 lens 328/344 e 0 to 1 dl 1790950476 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6004.742217] Lustre: 88345:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
[ 6011.338307] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 10:14:42 (1790950482)
[ 6012.704789] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6014.573641] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 10:14:45 (1790950485)
[ 6078.131442] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6080.031940] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 10:15:50 (1790950550)
[ 6135.775297] Lustre: lustre-MDT0000-mdc-ffff9f0e11057800: Connection to lustre-MDT0000 (at 192.168.203.136@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6135.790459] Lustre: Skipped 16 previous similar messages
[ 6137.443398] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 10:16:48 (1790950608)
[ 6139.388603] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6140.997263] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 10:16:52 (1790950612)
[ 6142.266295] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6143.832797] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 10:16:54 (1790950614)
[ 6176.635526] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 10:17:27 (1790950647)
[ 6187.684925] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6213.599229] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 6213.611588] LustreError: Skipped 2 previous similar messages
[ 6213.627838] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0x6bafb87c02a4ef7b to 0x6bafb87c02a507ca
[ 6213.643175] Lustre: Skipped 2 previous similar messages
[ 6215.021122] Lustre: 68269:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.136@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6235.218235] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6236.842996] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6246.359346] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 10:18:37 (1790950717)
[ 6248.654434] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6250.221743] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 10:18:41 (1790950721)
[ 6259.695919] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 6259.708078] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0x6bafb87c02a507ca to 0x6bafb87c02a50b27
[ 6272.786132] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 10:19:03 (1790950743)
[ 6318.051052] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 10:19:48 (1790950788)
[ 6369.336663] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6371.974237] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6456.289084] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 6456.307175] LustreError: Skipped 1 previous similar message
[ 6466.547410] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0x6bafb87c02a50f72 to 0x6bafb87c02a51e59
[ 6466.567489] Lustre: Skipped 1 previous similar message
[ 6477.735872] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6479.031743] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6513.263280] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6514.509078] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6628.708917] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 10:24:59 (1790951099)
[ 6637.990035] LustreError: lustre-OST0000-osc-ffff9f0e11057800: operation ost_setattr to node 192.168.203.136@tcp failed: rc = -19
[ 6638.002858] LustreError: Skipped 19 previous similar messages
[ 6682.551991] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6684.938975] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6940.238640] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 10:30:11 (1790951411)
[ 7071.205740] Lustre: lustre-MDT0000-mdc-ffff9f0e11057800: Connection to lustre-MDT0000 (at 192.168.203.136@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7071.226740] Lustre: Skipped 7 previous similar messages
[ 7081.447034] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 7081.465088] LustreError: Skipped 1 previous similar message
[ 7081.492829] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0x6bafb87c02a5223b to 0x6bafb87c02b4c47b
[ 7081.517228] Lustre: Skipped 1 previous similar message
[ 7081.526760] Lustre: MGC192.168.203.136@tcp: Connection restored to 192.168.203.136@tcp (at 192.168.203.136@tcp)
[ 7081.539205] Lustre: Skipped 13 previous similar messages
[ 7117.710162] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 10:33:08 (1790951588)
[ 7119.039869] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7120.494206] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 10:33:11 (1790951591)
[ 7122.326955] LustreError: lustre-MDT0000-mdc-ffff9f0e11057800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7122.344811] LustreError: 102003:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9f0e11057800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9f0e0898a200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 7122.364313] LustreError: 102003:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
[ 7128.917989] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 10:33:19 (1790951599)
[ 7296.487977] LustreError: lustre-OST0000-osc-ffff9f0e11057800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7298.184970] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 10:36:09 (1790951769)
[ 7322.015145] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790951774/real 1790951774]  req@ffff9f0e298b0380 x1877941538118528/t0(0) o4->lustre-OST0000-osc-ffff9f0e11057800@192.168.203.136@tcp:6/4 lens 4584/448 e 0 to 1 dl 1790951794 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7322.047033] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 27 previous similar messages
[ 7340.496375] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 10:36:51 (1790951811)
[ 7342.056440] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7343.437896] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 10:36:54 (1790951814)
[ 7345.186292] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7347.139732] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 10:36:58 (1790951818)
[ 7388.076978] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 10:37:37 (1790951857)
[ 7429.618152] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 7429.666218] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0x6bafb87c02b4c47b to 0x6bafb87c02b4ee4a
[ 7429.678405] Lustre: Skipped 1 previous similar message
[ 7443.120110] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 10:38:34 (1790951914)
[ 7444.452660] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7446.359929] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 10:38:37 (1790951917)
[ 7447.971217] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7449.830382] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 10:38:40 (1790951920)
[ 7458.216797] Lustre: Unmounted lustre-client
[ 7459.480707] Lustre: Mounted lustre-client - version 2.17.59_2_gbfaf0e7
[ 7462.500614] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7481.311233] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790951937/real 1790951937]  req@ffff9f0e08f44000 x1877941538304896/t0(0) o400->MGC192.168.203.136@tcp@192.168.203.136@tcp:26/25 lens 224/224 e 0 to 1 dl 1790951953 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7504.014897] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2