[ 1423.480983] Lustre: DEBUG MARKER: == recovery-small test 24b: test dirty page discard due to client eviction ========================================================== 23:26:47 (1717558007)
[ 1423.489748] LustreError: 11-0: lustre-OST0000-osc-ffff8800abe8a000: operation ost_sync to node 192.168.204.138@tcp failed: rc = -107
[ 1423.498153] Lustre: lustre-OST0000-osc-ffff8800abe8a000: Connection to lustre-OST0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1423.506961] LustreError: 167-0: lustre-OST0000-osc-ffff8800abe8a000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1424.105665] Lustre: 1999:0:(/home/green/git/lustre-release/lustre/llite/llite_lib.c:4076:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.138@tcp:/lustre/fid: [0x240000403:0xb:0x0]// may get corrupted (rc -108)
[ 1425.813123] Lustre: DEBUG MARKER: recovery-small test_24b: @@@@@@ FAIL: no discarded dirty page found!
[ 1426.511319] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 23:26:50 (1717558010)
[ 1427.119251] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1427.682290] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 23:26:51 (1717558011)
[ 1428.291738] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1430.065227] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 23:26:52 (1717558012)
[ 1430.070778] LustreError: 11-0: lustre-MDT0000-mdc-ffff8800abe8a000: operation mds_reint to node 192.168.204.138@tcp failed: rc = -19
[ 1430.070785] LustreError: 11-0: lustre-MDT0000-mdc-ffff8800abe8a000: operation mds_close to node 192.168.204.138@tcp failed: rc = -19
[ 1443.941873] Lustre: lustre-MDT0000-mdc-ffff8800abe8a000: Connection to lustre-MDT0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1443.960218] LustreError: 166-1: MGC192.168.204.138@tcp: Connection to MGS (at 192.168.204.138@tcp) was lost; in progress operations using this service will fail
[ 1537.596320] Lustre: Evicted from MGS (at 192.168.204.138@tcp) after server handle changed from 0xb6044830d79afc97 to 0xb6044830d79b6ac2
[ 1537.601034] LustreError: 11-0: lustre-MDT0000-mdc-ffff8800abe8a000: operation mds_reint to node 192.168.204.138@tcp failed: rc = -19
[ 1537.604721] Lustre: lustre-MDT0000-mdc-ffff8800abe8a000: Connection to lustre-MDT0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1554.117632] LustreError: 11-0: lustre-MDT0000-mdc-ffff8800abe8a000: operation mds_close to node 192.168.204.138@tcp failed: rc = -19
[ 1554.163173] LustreError: 166-1: MGC192.168.204.138@tcp: Connection to MGS (at 192.168.204.138@tcp) was lost; in progress operations using this service will fail
[ 1557.042627] Lustre: Evicted from MGS (at 192.168.204.138@tcp) after server handle changed from 0xb6044830d79b6ac2 to 0xb6044830d7b3c74b
[ 1574.814702] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 23:29:01 (1717558141)
[ 1584.166027] LustreError: 11-0: lustre-OST0000-osc-ffff8800abe8a000: operation ost_connect to node 192.168.204.138@tcp failed: rc = -75
[ 1594.181557] Lustre: lustre-MDT0000-mdc-ffff8800abe8a000: Connection to lustre-MDT0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1594.185926] LustreError: 166-1: MGC192.168.204.138@tcp: Connection to MGS (at 192.168.204.138@tcp) was lost; in progress operations using this service will fail
[ 1597.869441] Lustre: Evicted from MGS (at 192.168.204.138@tcp) after server handle changed from 0xb6044830d7b3c74b to 0xb6044830d7b3cd33
[ 1598.303672] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1600.938982] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1604.197576] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 23:29:45 (1717558185)
[ 1604.197606] LustreError: 166-1: MGC192.168.204.138@tcp: Connection to MGS (at 192.168.204.138@tcp) was lost; in progress operations using this service will fail
[ 1604.206298] Lustre: lustre-MDT0000-mdc-ffff8800abe8a000: Connection to lustre-MDT0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1623.269101] Lustre: Evicted from MGS (at 192.168.204.138@tcp) after server handle changed from 0xb6044830d7b3cd33 to 0xb6044830d7b3d066
[ 1629.238389] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 23:30:07 (1717558207)
[ 1642.114037] Lustre: lustre-OST0000-osc-ffff8800abe8a000: Connection to lustre-OST0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1652.802623] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 23:30:26 (1717558226)
[ 1652.802689] LustreError: 11-0: lustre-MDT0000-mdc-ffff8800abe8a000: operation mds_close to node 192.168.204.138@tcp failed: rc = -19
[ 1652.802696] LustreError: 11-0: lustre-MDT0000-mdc-ffff8800abe8a000: operation ldlm_enqueue to node 192.168.204.138@tcp failed: rc = -19
[ 1669.302450] Lustre: lustre-MDT0000-mdc-ffff8800abe8a000: Connection to lustre-MDT0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1669.314609] LustreError: 166-1: MGC192.168.204.138@tcp: Connection to MGS (at 192.168.204.138@tcp) was lost; in progress operations using this service will fail
[ 1671.526161] Lustre: Evicted from MGS (at 192.168.204.138@tcp) after server handle changed from 0xb6044830d7b3d066 to 0xb6044830d7b675bb
[ 1672.070772] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1734.110743] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1734.112831] LustreError: 11-0: lustre-MDT0000-mdc-ffff8800abe8a000: operation mds_reint to node 192.168.204.138@tcp failed: rc = -19
[ 1734.112842] LustreError: 11-0: lustre-MDT0000-mdc-ffff8800abe8a000: operation mds_close to node 192.168.204.138@tcp failed: rc = -19
[ 1749.430042] Lustre: lustre-MDT0000-mdc-ffff8800abe8a000: Connection to lustre-MDT0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1749.485508] LustreError: 166-1: MGC192.168.204.138@tcp: Connection to MGS (at 192.168.204.138@tcp) was lost; in progress operations using this service will fail
[ 1752.881695] Lustre: Evicted from MGS (at 192.168.204.138@tcp) after server handle changed from 0xb6044830d7b675bb to 0xb6044830d7c53872
[ 1753.448113] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1815.370832] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1815.373727] LustreError: 11-0: lustre-MDT0000-mdc-ffff8800abe8a000: operation ldlm_enqueue to node 192.168.204.138@tcp failed: rc = -19
[ 1829.557517] Lustre: lustre-MDT0000-mdc-ffff8800abe8a000: Connection to lustre-MDT0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1829.603908] LustreError: 166-1: MGC192.168.204.138@tcp: Connection to MGS (at 192.168.204.138@tcp) was lost; in progress operations using this service will fail
[ 1833.989023] Lustre: Evicted from MGS (at 192.168.204.138@tcp) after server handle changed from 0xb6044830d7c53872 to 0xb6044830d7d4b806
[ 1834.488451] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1857.945386] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1859.983558] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 23:34:02 (1717558442)
[ 1859.988354] LustreError: 11-0: lustre-MDT0000-mdc-ffff8800abe8a000: operation ldlm_enqueue to node 192.168.204.138@tcp failed: rc = -19
[ 1859.989877] Lustre: lustre-MDT0000-mdc-ffff8800abe8a000: Connection to lustre-MDT0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1874.629405] LustreError: 11-0: lustre-MDT0000-mdc-ffff8800abe8a000: operation mds_close to node 192.168.204.138@tcp failed: rc = -19
[ 1874.656296] LustreError: 166-1: MGC192.168.204.138@tcp: Connection to MGS (at 192.168.204.138@tcp) was lost; in progress operations using this service will fail
[ 1875.397731] Lustre: Evicted from MGS (at 192.168.204.138@tcp) after server handle changed from 0xb6044830d7d4b806 to 0xb6044830d7daea9e
[ 1892.419645] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 1894.661972] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 1894.665928] LustreError: 166-1: MGC192.168.204.138@tcp: Connection to MGS (at 192.168.204.138@tcp) was lost; in progress operations using this service will fail
[ 1913.068434] Lustre: Evicted from MGS (at 192.168.204.138@tcp) after server handle changed from 0xb6044830d7daea9e to 0xb6044830d7daecab
[ 1914.694369] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 1914.699035] LustreError: 166-1: MGC192.168.204.138@tcp: Connection to MGS (at 192.168.204.138@tcp) was lost; in progress operations using this service will fail
[ 1929.704747] Lustre: Evicted from MGS (at 192.168.204.138@tcp) after server handle changed from 0xb6044830d7daecab to 0xb6044830d7daeed4
[ 1939.204793] Lustre: 1996:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558459/real 1717558459]  req@ffff8800a9583b80 x1800988669927616/t0(0) o400->lustre-MDT0000-mdc-ffff8800abe8a000@192.168.204.138@tcp:12/10 lens 224/224 e 0 to 1 dl 1717558514 ref 1 fl Rpc:XQr/2c0/fff
[ 1939.734785] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 1939.741662] LustreError: 166-1: MGC192.168.204.138@tcp: Connection to MGS (at 192.168.204.138@tcp) was lost; in progress operations using this service will fail
[ 1959.846203] Lustre: Evicted from MGS (at 192.168.204.138@tcp) after server handle changed from 0xb6044830d7daeed4 to 0xb6044830d7daf0fd
[ 1959.847230] LustreError: 11-0: lustre-MDT0000-mdc-ffff8800abe8a000: operation mds_close to node 192.168.204.138@tcp failed: rc = -19
[ 1959.847239] LustreError: 11-0: lustre-MDT0000-mdc-ffff8800abe8a000: operation mds_reint to node 192.168.204.138@tcp failed: rc = -19
[ 1974.789939] Lustre: lustre-MDT0000-mdc-ffff8800abe8a000: Connection to lustre-MDT0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1974.806335] LustreError: 166-1: MGC192.168.204.138@tcp: Connection to MGS (at 192.168.204.138@tcp) was lost; in progress operations using this service will fail
[ 1975.089814] Lustre: Evicted from MGS (at 192.168.204.138@tcp) after server handle changed from 0xb6044830d7daf0fd to 0xb6044830d7df88da
[ 2000.733965] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2000.737398] LustreError: 11-0: lustre-MDT0000-mdc-ffff8800abe8a000: operation ldlm_enqueue to node 192.168.204.138@tcp failed: rc = -19
[ 2000.745637] Lustre: lustre-MDT0000-mdc-ffff8800abe8a000: Connection to lustre-MDT0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2014.853732] LustreError: 11-0: lustre-MDT0000-mdc-ffff8800abe8a000: operation mds_close to node 192.168.204.138@tcp failed: rc = -19
[ 2014.895523] LustreError: 166-1: MGC192.168.204.138@tcp: Connection to MGS (at 192.168.204.138@tcp) was lost; in progress operations using this service will fail
[ 2015.806173] Lustre: Evicted from MGS (at 192.168.204.138@tcp) after server handle changed from 0xb6044830d7df88da to 0xb6044830d7e53a54
[ 2046.567046] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2046.567079] LustreError: 11-0: lustre-MDT0000-mdc-ffff8800abe8a000: operation mds_reint to node 192.168.204.138@tcp failed: rc = -19
[ 2046.567083] LustreError: 11-0: lustre-MDT0000-mdc-ffff8800abe8a000: operation mds_close to node 192.168.204.138@tcp failed: rc = -19
[ 2059.926040] Lustre: lustre-MDT0000-mdc-ffff8800abe8a000: Connection to lustre-MDT0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2059.970543] LustreError: 166-1: MGC192.168.204.138@tcp: Connection to MGS (at 192.168.204.138@tcp) was lost; in progress operations using this service will fail
[ 2083.576701] Lustre: Evicted from MGS (at 192.168.204.138@tcp) after server handle changed from 0xb6044830d7e53a54 to 0xb6044830d7ec43ae
[ 2094.419135] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 23:37:48 (1717558668)
[ 2094.427828] LustreError: 11-0: lustre-OST0000-osc-ffff8800abe8a000: operation ldlm_enqueue to node 192.168.204.138@tcp failed: rc = -107
[ 2111.658835] Lustre: lustre-OST0000-osc-ffff8800abe8a000: Connection to lustre-OST0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2112.123837] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2424.693831] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2424.700235] LustreError: 11-0: lustre-OST0000-osc-ffff8800abe8a000: operation ldlm_enqueue to node 192.168.204.138@tcp failed: rc = -19
[ 2441.993276] Lustre: lustre-OST0000-osc-ffff8800abe8a000: Connection to lustre-OST0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2442.573516] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2755.657425] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2755.663075] LustreError: 11-0: lustre-OST0000-osc-ffff8800abe8a000: operation ldlm_enqueue to node 192.168.204.138@tcp failed: rc = -19
[ 2773.209124] Lustre: lustre-OST0000-osc-ffff8800abe8a000: Connection to lustre-OST0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2773.715461] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3048.034466] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3064.540737] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 23:53:52 (1717559632)
[ 3064.544238] Lustre: 22854:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559633/real 1717559633]  req@ffff88012dc4bb80 x1800988770331840/t0(0) o101->lustre-MDT0000-mdc-ffff8800abe8a000@192.168.204.138@tcp:12/10 lens 576/1024 e 0 to 1 dl 1717559649 ref 2 fl Rpc:XQr/200/ff
[ 3066.949943] Lustre: lustre-MDT0000-mdc-ffff8800abe8a000: Connection to lustre-MDT0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3083.423772] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 23:54:11 (1717559651)
[ 3083.428400] Lustre: 23603:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559652/real 1717559652]  req@ffff8800a9193b80 x1800988770341056/t0(0) o101->lustre-MDT0000-mdc-ffff8800abe8a000@192.168.204.138@tcp:12/10 lens 576/1024 e 0 to 1 dl 1717559668 ref 2 fl Rpc:XQr/200/ff
[ 3085.941029] Lustre: lustre-MDT0000-mdc-ffff8800abe8a000: Connection to lustre-MDT0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3102.440781] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 23:54:30 (1717559670)
[ 3102.445706] Lustre: 24354:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559671/real 1717559671]  req@ffff8800a9190700 x1800988770343104/t0(0) o101->lustre-MDT0000-mdc-ffff8800abe8a000@192.168.204.138@tcp:12/10 lens 664/66320 e 0 to 1 dl 1717559687 ref 2 fl Rpc:XPQr/200/
[ 3104.966760] Lustre: lustre-MDT0000-mdc-ffff8800abe8a000: Connection to lustre-MDT0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3105.078208] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 23:54:49 (1717559689)
[ 3120.102552] Lustre: Mounted lustre-client
[ 3120.102611] Lustre: lustre-MDT0000-mdc-ffff8800abe8a000: Connection to lustre-MDT0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3130.117909] Lustre: lustre-MDT0000-mdc-ffff8800b6338000: Connection to lustre-MDT0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3130.130849] LustreError: 166-1: MGC192.168.204.138@tcp: Connection to MGS (at 192.168.204.138@tcp) was lost; in progress operations using this service will fail
[ 3134.710180] Lustre: Evicted from MGS (at 192.168.204.138@tcp) after server handle changed from 0xb6044830d7ec43ae to 0xb6044830d8cbe718
[ 3135.281480] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3136.644748] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3138.719811] Lustre: Unmounted lustre-client
[ 3156.147709] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 23:55:23 (1717559723)
[ 3156.158514] Lustre: 2000:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559725/real 1717559725]  req@ffff8800a9665c00 x1800988770361024/t0(0) o4->lustre-OST0000-osc-ffff8800abe8a000@192.168.204.138@tcp:6/4 lens 488/448 e 0 to 1 dl 1717559741 ref 2 fl Rpc:XQr/200/fffffff
[ 3156.158567] Lustre: lustre-OST0000-osc-ffff8800abe8a000: Connection to lustre-OST0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3156.180661] Lustre: 1999:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559725/real 1717559725]  req@ffff8800a9664380 x1800988770360960/t0(0) o4->lustre-OST0000-osc-ffff8800abe8a000@192.168.204.138@tcp:6/4 lens 488/448 e 0 to 1 dl 1717559741 ref 2 fl Rpc:XQr/200/fffffff
[ 3156.180669] Lustre: 2000:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559725/real 1717559725]  req@ffff8800aa2e4000 x1800988770360832/t0(0) o4->lustre-OST0000-osc-ffff8800abe8a000@192.168.204.138@tcp:6/4 lens 488/448 e 0 to 1 dl 1717559741 ref 2 fl Rpc:XQr/200/fffffff
[ 3156.180717] Lustre: 1999:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559725/real 1717559725]  req@ffff8800aa2e7b80 x1800988770360704/t0(0) o4->lustre-OST0000-osc-ffff8800abe8a000@192.168.204.138@tcp:6/4 lens 488/448 e 0 to 1 dl 1717559741 ref 2 fl Rpc:XQr/200/fffffff
[ 3156.180753] Lustre: 1999:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559725/real 1717559725]  req@ffff8800a9630380 x1800988770360448/t0(0) o4->lustre-OST0000-osc-ffff8800abe8a000@192.168.204.138@tcp:6/4 lens 488/448 e 0 to 1 dl 1717559741 ref 2 fl Rpc:XQr/200/fffffff
[ 3156.255656] Lustre: 1999:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559725/real 1717559725]  req@ffff8800a9630000 x1800988770360192/t0(0) o4->lustre-OST0000-osc-ffff8800abe8a000@192.168.204.138@tcp:6/4 lens 488/448 e 0 to 1 dl 1717559741 ref 2 fl Rpc:XQr/200/fffffff
[ 3172.189706] Lustre: 2000:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1717559725/real 1717559725]  req@ffff8800aa2e4000 x1800988770360832/t0(0) o4->lustre-OST0000-osc-ffff8800abe8a000@192.168.204.138@tcp:6/4 lens 488/448 e 0 to 1 dl 1717559741 ref 2 fl Rpc:eXSQU/200/ff
[ 3172.203695] Lustre: 1999:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559741/real 1717559741]  req@ffff8800a9664380 x1800988770360960/t0(0) o4->lustre-OST0000-osc-ffff8800abe8a000@192.168.204.138@tcp:6/4 lens 488/448 e 0 to 1 dl 1717559757 ref 2 fl Rpc:XQr/202/fffffff
[ 3172.203745] Lustre: 1997:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559741/real 1717559741]  req@ffff88012b656680 x1800988770360064/t0(0) o4->lustre-OST0000-osc-ffff8800abe8a000@192.168.204.138@tcp:6/4 lens 488/448 e 0 to 1 dl 1717559757 ref 2 fl Rpc:XQr/202/fffffff
[ 3172.232710] Lustre: lustre-OST0000-osc-ffff8800abe8a000: Connection to lustre-OST0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3188.219795] Lustre: 1999:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1717559741/real 1717559741]  req@ffff8800a9664380 x1800988770360960/t0(0) o4->lustre-OST0000-osc-ffff8800abe8a000@192.168.204.138@tcp:6/4 lens 488/448 e 0 to 1 dl 1717559757 ref 2 fl Rpc:eXSQU/202/ff
[ 3188.233725] Lustre: 1997:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559757/real 1717559757]  req@ffff88012b656680 x1800988770360064/t0(0) o4->lustre-OST0000-osc-ffff8800abe8a000@192.168.204.138@tcp:6/4 lens 488/448 e 0 to 1 dl 1717559773 ref 2 fl Rpc:XQr/202/fffffff
[ 3188.235737] Lustre: lustre-OST0000-osc-ffff8800abe8a000: Connection to lustre-OST0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3188.235746] Lustre: 1999:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559757/real 1717559757]  req@ffff8800a9664380 x1800988770360960/t0(0) o4->lustre-OST0000-osc-ffff8800abe8a000@192.168.204.138@tcp:6/4 lens 488/448 e 0 to 1 dl 1717559773 ref 2 fl Rpc:XQr/202/fffffff
[ 3188.264512] Lustre: 2000:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559757/real 1717559757]  req@ffff8800a9665c00 x1800988770361024/t0(0) o4->lustre-OST0000-osc-ffff8800abe8a000@192.168.204.138@tcp:6/4 lens 488/448 e 0 to 1 dl 1717559773 ref 2 fl Rpc:XQr/202/fffffff
[ 3188.264520] Lustre: 2000:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559757/real 1717559757]  req@ffff8800aa2e4000 x1800988770360832/t0(0) o4->lustre-OST0000-osc-ffff8800abe8a000@192.168.204.138@tcp:6/4 lens 488/448 e 0 to 1 dl 1717559773 ref 2 fl Rpc:XQr/202/fffffff
[ 3188.264762] Lustre: 1999:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559757/real 1717559757]  req@ffff8800aa2e7b80 x1800988770360704/t0(0) o4->lustre-OST0000-osc-ffff8800abe8a000@192.168.204.138@tcp:6/4 lens 488/448 e 0 to 1 dl 1717559773 ref 2 fl Rpc:XQr/202/fffffff
[ 3188.264798] Lustre: 1999:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559757/real 1717559757]  req@ffff8800a9630380 x1800988770360448/t0(0) o4->lustre-OST0000-osc-ffff8800abe8a000@192.168.204.138@tcp:6/4 lens 488/448 e 0 to 1 dl 1717559773 ref 2 fl Rpc:XQr/202/fffffff
[ 3188.318721] Lustre: 1999:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559757/real 1717559757]  req@ffff8800a9630000 x1800988770360192/t0(0) o4->lustre-OST0000-osc-ffff8800abe8a000@192.168.204.138@tcp:6/4 lens 488/448 e 0 to 1 dl 1717559773 ref 2 fl Rpc:XQr/202/fffffff
[ 3204.299766] Lustre: 2000:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1717559757/real 1717559757]  req@ffff8800aa2e4000 x1800988770360832/t0(0) o4->lustre-OST0000-osc-ffff8800abe8a000@192.168.204.138@tcp:6/4 lens 488/448 e 0 to 1 dl 1717559773 ref 2 fl Rpc:eXSQU/202/ff
[ 3204.299775] Lustre: 2000:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559773/real 1717559773]  req@ffff8800a9665c00 x1800988770361024/t0(0) o4->lustre-OST0000-osc-ffff8800abe8a000@192.168.204.138@tcp:6/4 lens 488/448 e 0 to 1 dl 1717559789 ref 2 fl Rpc:XQr/202/fffffff
[ 3204.299833] Lustre: 1999:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559773/real 1717559773]  req@ffff8800a9664380 x1800988770360960/t0(0) o4->lustre-OST0000-osc-ffff8800abe8a000@192.168.204.138@tcp:6/4 lens 488/448 e 0 to 1 dl 1717559789 ref 2 fl Rpc:XQr/202/fffffff
[ 3204.299885] Lustre: lustre-OST0000-osc-ffff8800abe8a000: Connection to lustre-OST0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3204.299940] Lustre: 1999:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559773/real 1717559773]  req@ffff8800aa2e7b80 x1800988770360704/t0(0) o4->lustre-OST0000-osc-ffff8800abe8a000@192.168.204.138@tcp:6/4 lens 488/448 e 0 to 1 dl 1717559789 ref 2 fl Rpc:XQr/202/fffffff
[ 3204.299991] Lustre: 1999:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559773/real 1717559773]  req@ffff8800a9630380 x1800988770360448/t0(0) o4->lustre-OST0000-osc-ffff8800abe8a000@192.168.204.138@tcp:6/4 lens 488/448 e 0 to 1 dl 1717559789 ref 2 fl Rpc:XQr/202/fffffff
[ 3204.349831] Lustre: 1999:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559773/real 1717559773]  req@ffff8800a9630000 x1800988770360192/t0(0) o4->lustre-OST0000-osc-ffff8800abe8a000@192.168.204.138@tcp:6/4 lens 488/448 e 0 to 1 dl 1717559789 ref 2 fl Rpc:XQr/202/fffffff
[ 3220.308712] Lustre: 2000:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1717559773/real 1717559773]  req@ffff8800a9665c00 x1800988770361024/t0(0) o4->lustre-OST0000-osc-ffff8800abe8a000@192.168.204.138@tcp:6/4 lens 488/448 e 0 to 1 dl 1717559789 ref 2 fl Rpc:eXSQU/202/ff
[ 3220.314772] Lustre: 1997:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559789/real 1717559789]  req@ffff88012b655180 x1800988770366720/t0(0) o4->lustre-OST0000-osc-ffff8800abe8a000@192.168.204.138@tcp:6/4 lens 488/448 e 0 to 1 dl 1717559805 ref 2 fl Rpc:XQr/202/fffffff
[ 3220.326008] Lustre: lustre-OST0000-osc-ffff8800abe8a000: Connection to lustre-OST0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3222.334505] Lustre: 1997:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559789/real 1717559789]  req@ffff88012b656680 x1800988770360064/t0(0) o4->lustre-OST0000-osc-ffff8800abe8a000@192.168.204.138@tcp:6/4 lens 488/448 e 0 to 1 dl 1717559805 ref 2 fl Rpc:XQr/202/fffffff
[ 3238.589741] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 23:56:47 (1717559807)
[ 3238.597220] Lustre: 27357:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559807/real 1717559807]  req@ffff8800a9191f80 x1800988770375744/t0(0) o101->lustre-MDT0000-mdc-ffff8800abe8a000@192.168.204.138@tcp:12/10 lens 576/1024 e 0 to 1 dl 1717559823 ref 2 fl Rpc:XQr/200/ff
[ 3254.624746] Lustre: lustre-MDT0000-mdc-ffff8800abe8a000: Connection to lustre-MDT0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3254.632851] Lustre: 27357:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559823/real 1717559823]  req@ffff8800a9191f80 x1800988770375744/t0(0) o101->lustre-MDT0000-mdc-ffff8800abe8a000@192.168.204.138@tcp:12/10 lens 576/1024 e 0 to 1 dl 1717559839 ref 2 fl Rpc:XQr/202/ff
[ 3264.390690] Lustre: lustre-MDT0000-mdc-ffff8800abe8a000: Connection to lustre-MDT0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3265.657573] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 23:57:29 (1717559849)
[ 3279.399071] Lustre: Unmounted lustre-client
[ 3281.040187] Lustre: Mounted lustre-client
[ 3281.096630] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 23:57:45 (1717559865)
[ 3282.103702] LustreError: 29225:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 3299.988642] LustreError: 29225:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:148:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted
[ 3300.081063] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 23:58:04 (1717559884)
[ 3300.293456] Lustre: Mounted lustre-client
[ 3300.296198] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 3300.299587] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request
[ 3300.301739] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request
[ 3310.328610] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request
[ 3312.185188] Lustre: Unmounted lustre-client
[ 3336.429613] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 23:58:16 (1717559896)
[ 3336.433209] LustreError: 11-0: lustre-MDT0000-mdc-ffff8800a89c2000: operation mds_reint to node 192.168.204.138@tcp failed: rc = -19
[ 3350.164992] Lustre: lustre-MDT0000-mdc-ffff8800a89c2000: Connection to lustre-MDT0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3350.173065] LustreError: 166-1: MGC192.168.204.138@tcp: Connection to MGS (at 192.168.204.138@tcp) was lost; in progress operations using this service will fail
[ 3365.772749] Lustre: Evicted from MGS (at 192.168.204.138@tcp) after server handle changed from 0xb6044830d8cbf03a to 0xb6044830d8cef130
[ 3367.895988] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 23:59:10 (1717559950)
[ 3370.197023] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 3375.205157] Lustre: lustre-MDT0000-mdc-ffff8800a89c2000: Connection to lustre-MDT0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3375.206458] LustreError: 166-1: MGC192.168.204.138@tcp: Connection to MGS (at 192.168.204.138@tcp) was lost; in progress operations using this service will fail
[ 3375.219249] LustreError: 167-0: lustre-MDT0000-mdc-ffff8800a89c2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3383.552885] Lustre: Evicted from MGS (at 192.168.204.138@tcp) after server handle changed from 0xb6044830d8cef130 to 0xb6044830d8d24482
[ 3383.611053] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 23:59:28 (1717559968)
[ 3388.613441] Lustre: Mounted lustre-client
[ 3388.615370] Lustre: lustre-OST0000-osc-ffff8800a89c2000: Connection to lustre-OST0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3399.627678] LustreError: 167-0: lustre-OST0000-osc-ffff8800a89c2000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 3399.631787] Lustre: 1997:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559968/real 1717559968]  req@ffff88012b654000 x1800988772490048/t0(0) o103->lustre-OST0000-osc-ffff8800b63d3800@192.168.204.138@tcp:17/18 lens 328/224 e 0 to 1 dl 1717559984 ref 1 fl Rpc:XQr/200/fff
[ 3399.799622] Lustre: lustre-OST0000-osc-ffff8800b63d3800: Connection to lustre-OST0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3401.166921] Lustre: Unmounted lustre-client
[ 3403.516710] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 23:59:45 (1717559985)
[ 3405.748334] Lustre: lustre-MDT0000-mdc-ffff8800a89c2000: Connection to lustre-MDT0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3405.753456] LustreError: 11-0: lustre-MDT0000-mdc-ffff8800a89c2000: operation ldlm_enqueue to node 192.168.204.138@tcp failed: rc = -107
[ 3405.757848] Lustre: lustre-MDT0000-mdc-ffff8800a89c2000: Connection to lustre-MDT0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3405.761376] LustreError: 167-0: lustre-MDT0000-mdc-ffff8800a89c2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3407.355639] LustreError: 1823:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 3407.379155] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 23:59:52 (1717559992)
[ 3407.382008] Lustre: lustre-MDT0000-mdc-ffff8800a89c2000: Connection to lustre-MDT0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3410.612682] LustreError: 2579:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 531 sleeping
[ 3410.615316] LustreError: 11-0: lustre-MDT0000-mdc-ffff8800a89c2000: operation ldlm_enqueue to node 192.168.204.138@tcp failed: rc = -107
[ 3410.618439] Lustre: lustre-MDT0000-mdc-ffff8800a89c2000: Connection to lustre-MDT0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3410.622003] LustreError: 167-0: lustre-MDT0000-mdc-ffff8800a89c2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3410.624710] LustreError: 2606:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:210:cfs_race_loc()) cfs_fail_race id 531 waking
[ 3410.627192] LustreError: 2579:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 531 awake: rc=1760
[ 3410.633313] LustreError: 2579:0:(/home/green/git/lustre-release/lustre/ptlrpc/import.c:705:ptlrpc_connect_import_locked()) already connecting
[ 3410.636669] LustreError: 2611:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3410.645163] LustreError: 2612:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3410.651002] LustreError: 2613:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3410.655947] LustreError: 2614:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3410.659288] LustreError: 2615:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3411.666362] LustreError: 2615:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3411.682947] LustreError: 2617:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3411.689644] LustreError: 2622:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3411.698498] LustreError: 2623:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3411.704714] LustreError: 2625:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3411.709779] LustreError: 2626:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3411.712902] LustreError: 2627:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3412.722427] LustreError: 2627:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3412.736803] LustreError: 2629:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3412.741230] LustreError: 2634:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3412.751680] LustreError: 2635:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3412.758683] LustreError: 2636:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3412.764052] LustreError: 2637:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3412.767015] LustreError: 2638:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3413.773691] LustreError: 2638:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3413.786672] LustreError: 2641:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3413.790354] LustreError: 2646:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3413.797699] LustreError: 2647:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3413.803441] LustreError: 2648:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3413.811308] LustreError: 2649:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3413.815939] LustreError: 2650:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3414.822811] LustreError: 2650:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3414.834885] LustreError: 2653:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3414.838726] LustreError: 2658:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3414.847200] LustreError: 2659:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3414.853703] LustreError: 2660:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3414.862682] LustreError: 2661:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3414.865685] LustreError: 2662:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3415.872094] LustreError: 2662:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3415.884856] LustreError: 2665:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3415.888855] LustreError: 2670:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3415.896996] LustreError: 2671:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3415.903353] LustreError: 2672:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3415.912086] LustreError: 2673:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3415.915120] LustreError: 2674:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3416.922180] LustreError: 2674:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3416.934512] LustreError: 2677:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3416.938542] LustreError: 2682:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3416.946196] LustreError: 2683:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3416.951857] LustreError: 2684:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3416.956460] LustreError: 2685:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3416.959450] LustreError: 2686:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3417.966224] LustreError: 2686:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3417.981936] LustreError: 2689:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3417.985829] LustreError: 2694:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3417.993694] LustreError: 2695:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3417.999924] LustreError: 2696:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3418.005009] LustreError: 2697:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3418.008009] LustreError: 2698:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3419.014345] LustreError: 2698:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3419.027646] LustreError: 2700:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3419.031665] LustreError: 2705:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3419.040693] LustreError: 2706:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3419.046750] LustreError: 2707:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3419.052403] LustreError: 2708:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3419.059016] LustreError: 2709:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3420.069736] LustreError: 2709:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3420.082145] LustreError: 2712:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3420.085948] LustreError: 2717:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3420.093757] LustreError: 2718:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3420.099727] LustreError: 2719:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3420.104833] LustreError: 2720:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3420.107662] LustreError: 2721:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3422.493933] LustreError: 2721:0:(/home/green/git/lustre-release/lustre/llite/file.c:5683:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3425.685337] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 00:00:07 (1717560007)
[ 3440.961444] Lustre: lustre-OST0000-osc-ffff8800a89c2000: Connection to lustre-OST0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3441.332303] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3444.302751] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3445.717325] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 00:00:29 (1717560029)
[ 3460.217494] Lustre: lustre-OST0000-osc-ffff8800a89c2000: Connection to lustre-OST0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3460.741417] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3464.592990] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3470.757264] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 00:00:49 (1717560049)
[ 3506.619280] Lustre: lustre-OST0000-osc-ffff8800a89c2000: Connection to lustre-OST0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3506.956259] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3509.483570] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3510.821040] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 00:01:34 (1717560094)
[ 3525.549219] Lustre: lustre-OST0000-osc-ffff8800a89c2000: Connection to lustre-OST0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3525.888765] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3527.710610] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3546.844778] Lustre: Unmounted lustre-client
[ 3548.134143] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3549.476424] Lustre: Mounted lustre-client
[ 3550.037457] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 00:02:14 (1717560134)
[ 3550.400075] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 3553.141335] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 00:02:15 (1717560135)
[ 3557.959094] Lustre: lustre-OST0000-osc-ffff8800b6ccd000: Connection to lustre-OST0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3558.271628] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 00:02:22 (1717560142)
[ 3558.615516] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 3558.915600] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 00:02:23 (1717560143)
[ 3560.974072] Lustre: Mounted lustre-client
[ 3563.925465] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 3563.926899] Lustre: lustre-MDT0000-mdc-ffff8800aa225800: Connection to lustre-MDT0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3578.949610] Lustre: lustre-MDT0000-mdc-ffff8800b6ccd000: Connection to lustre-MDT0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3578.950307] LustreError: 166-1: MGC192.168.204.138@tcp: Connection to MGS (at 192.168.204.138@tcp) was lost; in progress operations using this service will fail
[ 3578.951492] LustreError: 1996:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_resource.c:1128:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800aa225800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff88012dc16300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 3578.961974] LustreError: 1996:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_resource.c:1128:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800aa225800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff88012dc16300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 3579.264475] Lustre: Evicted from MGS (at 192.168.204.138@tcp) after server handle changed from 0xb6044830d8d24fc6 to 0xb6044830d8d25426
[ 3580.431571] Lustre: DEBUG MARKER: recovery-small test_106: @@@@@@ FAIL: lightweight client was not recovered
[ 3580.769288] Lustre: Unmounted lustre-client
[ 3583.957251] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 00:02:45 (1717560165)
[ 3593.972895] Lustre: lustre-MDT0000-mdc-ffff8800b6ccd000: Connection to lustre-MDT0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3593.979213] LustreError: 166-1: MGC192.168.204.138@tcp: Connection to MGS (at 192.168.204.138@tcp) was lost; in progress operations using this service will fail
[ 3593.987723] LustreError: 1996:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:3293:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8800a9448000 x1800988772525824/t81604378626(81604378626) o101->lustre-MDT0000-mdc-ffff8800b6ccd000@192.168.204.138@tcp:12/10 lens 536/664 e 0 to 0 dl 1717560239 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'multiop.0' ui
[ 3599.753261] Lustre: Evicted from MGS (at 192.168.204.138@tcp) after server handle changed from 0xb6044830d8d25426 to 0xb6044830d8d25824
[ 3600.304705] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3603.666923] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3605.573039] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 00:03:08 (1717560188)
[ 3605.574828] LustreError: 11-0: lustre-OST0000-osc-ffff8800b6ccd000: operation ost_statfs to node 192.168.204.138@tcp failed: rc = -107
[ 3605.577836] Lustre: lustre-OST0000-osc-ffff8800b6ccd000: Connection to lustre-OST0000 (at 192.168.204.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3605.581038] LustreError: 1996:0:(/home/green/git/lustre-release/lustre/osc/osc_request.c:1037:osc_init_grant()) lustre-OST0000-osc-ffff8800b6ccd000: granted 8437760 but already consumed 135102464
[ 3605.584224] LustreError: 167-0: lustre-OST0000-osc-ffff8800b6ccd000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 3605.965138] Lustre: 1998:0:(/home/green/git/lustre-release/lustre/llite/llite_lib.c:4076:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.138@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)