[ 1534.941766] Lustre: Failing over lustre-MDT0000
[ 1535.115500] Lustre: server umount lustre-MDT0000 complete
[ 1539.411645] Lustre: lustre-MDT0000: Aborting client recovery
[ 1539.414769] LustreError: 21122:0:(ldlm_lib.c:2898:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 1539.418987] Lustre: 21266:0:(ldlm_lib.c:2281:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 1539.424569] Lustre: 21266:0:(ldlm_lib.c:2281:target_recovery_overseer()) Skipped 2 previous similar messages
[ 1539.429343] Lustre: 21266:0:(genops.c:1529:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 9cdfd1aa-de4f-44c6-b6dd-d33822cabee4@<unknown>
[ 1539.435603] Lustre: lustre-MDT0000: disconnecting 1 stale clients
[ 1539.466359] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0]
[ 1539.545712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1543 to 0x240000400:1633)
[ 1539.554130] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1571 to 0x280000400:1633)
[ 1541.007367] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1547.776968] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 13:33:51 (1726853631)
[ 1564.972712] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1565.752119] Lustre: Failing over lustre-MDT0000
[ 1565.915990] Lustre: server umount lustre-MDT0000 complete
[ 1579.964670] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1580.458250] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 1580.464219] Lustre: Skipped 17 previous similar messages
[ 1580.537154] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 1580.541621] Lustre: Skipped 17 previous similar messages
[ 1580.564969] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2034 to 0x240000400:2049)
[ 1580.573605] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2034 to 0x280000400:2049)
[ 1583.499764] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1584.389654] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1597.012034] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 13:34:40 (1726853680)
[ 1609.614825] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1615.017159] Lustre: Failing over lustre-MDT0000
[ 1615.197035] Lustre: server umount lustre-MDT0000 complete
[ 1628.674330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up)
[ 1630.335308] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1632.467425] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2450 to 0x280000400:2465)
[ 1632.467566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2450 to 0x240000400:2465)
[ 1634.250271] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1634.996652] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1646.762259] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 13:35:30 (1726853730)
[ 1647.933864] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request
[ 1648.523070] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 1648.528555] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
[ 1651.681181] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 13:35:35 (1726853735)
[ 1663.749647] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 1669.048694] Lustre: Failing over lustre-OST0000
[ 1669.081838] Lustre: server umount lustre-OST0000 complete
[ 1670.603410] LustreError: lustre-OST0000: not available for connect from 192.168.202.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1670.612949] LustreError: Skipped 1 previous similar message
[ 1672.404115] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107
[ 1672.407971] LustreError: Skipped 5 previous similar messages
[ 1672.408428] LustreError: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1673.814016] LustreError: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1678.820848] LustreError: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1678.826380] LustreError: Skipped 1 previous similar message
[ 1683.419269] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1731.985714] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 13:36:55 (1726853815)
[ 1733.737083] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1734.743643] Lustre: Failing over lustre-MDT0000
[ 1734.773303] LustreError: 3303:0:(client.c:1300:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff8800a602d500 x1810735694991360/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0
[ 1734.894134] Lustre: server umount lustre-MDT0000 complete
[ 1748.639821] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1750.737949] Lustre: *** cfs_fail_loc=204, val=2147483648***
[ 1750.739081] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2866 to 0x280000400:2881)
[ 1751.974216] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1752.507662] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1766.237283] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 13:37:29 (1726853849)
[ 1766.740925] LustreError: 607:0:(osp_precreate.c:992:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11
[ 1766.750102] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 1767.746023] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2867 to 0x240000400:2913)
[ 1768.403118] LustreError: 558:0:(ldlm_lib.c:1078:target_handle_connect()) cfs_race id 701 sleeping
[ 1773.402586] LustreError: 558:0:(ldlm_lib.c:1078:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 1773.405842] Lustre: lustre-MDT0000: Client 9cdfd1aa-de4f-44c6-b6dd-d33822cabee4 (at 192.168.202.9@tcp) reconnecting
[ 1774.122907] LustreError: 2137:0:(ldlm_lib.c:1078:target_handle_connect()) cfs_race id 701 sleeping
[ 1779.126434] LustreError: 2137:0:(ldlm_lib.c:1078:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 1779.130805] Lustre: lustre-MDT0000: Client 9cdfd1aa-de4f-44c6-b6dd-d33822cabee4 (at 192.168.202.9@tcp) reconnecting
[ 1779.788913] LustreError: 2137:0:(ldlm_lib.c:1078:target_handle_connect()) cfs_race id 701 sleeping
[ 1784.793862] LustreError: 2137:0:(ldlm_lib.c:1078:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 1784.797254] Lustre: lustre-MDT0000: Client 9cdfd1aa-de4f-44c6-b6dd-d33822cabee4 (at 192.168.202.9@tcp) reconnecting
[ 1785.457246] LustreError: 2433:0:(ldlm_lib.c:1078:target_handle_connect()) cfs_race id 701 sleeping
[ 1790.460346] LustreError: 2433:0:(ldlm_lib.c:1078:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 1791.092001] LustreError: 2433:0:(ldlm_lib.c:1078:target_handle_connect()) cfs_race id 701 sleeping
[ 1796.097316] LustreError: 2433:0:(ldlm_lib.c:1078:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 1796.100834] Lustre: lustre-MDT0000: Client 9cdfd1aa-de4f-44c6-b6dd-d33822cabee4 (at 192.168.202.9@tcp) reconnecting
[ 1796.104698] Lustre: Skipped 1 previous similar message
[ 1802.462293] LustreError: 558:0:(ldlm_lib.c:1078:target_handle_connect()) cfs_race id 701 sleeping
[ 1802.465586] LustreError: 558:0:(ldlm_lib.c:1078:target_handle_connect()) Skipped 1 previous similar message
[ 1807.469284] LustreError: 558:0:(ldlm_lib.c:1078:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 1807.473283] LustreError: 558:0:(ldlm_lib.c:1078:target_handle_connect()) Skipped 1 previous similar message
[ 1813.252485] Lustre: lustre-MDT0000: Client 9cdfd1aa-de4f-44c6-b6dd-d33822cabee4 (at 192.168.202.9@tcp) reconnecting
[ 1813.257313] Lustre: Skipped 2 previous similar messages
[ 1819.850038] LustreError: 556:0:(ldlm_lib.c:1078:target_handle_connect()) cfs_race id 701 sleeping
[ 1819.852503] LustreError: 556:0:(ldlm_lib.c:1078:target_handle_connect()) Skipped 2 previous similar messages
[ 1824.855373] LustreError: 556:0:(ldlm_lib.c:1078:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 1824.859039] LustreError: 556:0:(ldlm_lib.c:1078:target_handle_connect()) Skipped 2 previous similar messages
[ 1828.037342] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 13:38:31 (1726853911)
[ 1828.829806] LustreError: 2433:0:(ldlm_lib.c:1330:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 1838.825755] Lustre: lustre-MDT0000: Export ffff8800b1ebe000 already connecting from 192.168.202.9@tcp
[ 1843.831595] Lustre: lustre-MDT0000: Export ffff8800b1ebe000 already connecting from 192.168.202.9@tcp
[ 1848.837795] Lustre: lustre-MDT0000: Export ffff8800b1ebe000 already connecting from 192.168.202.9@tcp
[ 1853.846482] Lustre: lustre-MDT0000: Export ffff8800b1ebe000 already connecting from 192.168.202.9@tcp
[ 1858.856369] Lustre: lustre-MDT0000: Export ffff8800b1ebe000 already connecting from 192.168.202.9@tcp
[ 1868.871666] Lustre: lustre-MDT0000: Export ffff8800b1ebe000 already connecting from 192.168.202.9@tcp
[ 1868.876511] Lustre: Skipped 1 previous similar message
[ 1868.888335] LustreError: 2433:0:(ldlm_lib.c:1330:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 1868.892069] Lustre: 2433:0:(service.c:2368:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff88009be89500 x1810735692206720/t0(0) o38->9cdfd1aa-de4f-44c6-b6dd-d33822cabee4@192.168.202.9@tcp:0/0 lens 520/416 e 0 to 0 dl 1726853931 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0
[ 1873.879832] Lustre: lustre-MDT0000: Client 9cdfd1aa-de4f-44c6-b6dd-d33822cabee4 (at 192.168.202.9@tcp) reconnecting
[ 1873.888187] Lustre: Skipped 3 previous similar messages
[ 1874.716118] LustreError: 1873:0:(ldlm_lib.c:1330:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 1889.734143] Lustre: lustre-MDT0000: Export ffff8800b1ebe000 already connecting from 192.168.202.9@tcp
[ 1889.738533] Lustre: Skipped 1 previous similar message
[ 1914.767596] LustreError: 1873:0:(ldlm_lib.c:1330:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 1914.773794] Lustre: 1873:0:(service.c:2368:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff88009f79c700 x1810735692211072/t0(0) o38->9cdfd1aa-de4f-44c6-b6dd-d33822cabee4@192.168.202.9@tcp:0/0 lens 520/416 e 0 to 0 dl 1726853977 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0
[ 1914.789691] LustreError: 2433:0:(ldlm_lib.c:1330:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 1939.782707] Lustre: lustre-MDT0000: Export ffff8800b1ebe000 already connecting from 192.168.202.9@tcp
[ 1939.788857] Lustre: Skipped 5 previous similar messages
[ 1954.861707] LustreError: 2433:0:(ldlm_lib.c:1330:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 1954.867820] Lustre: 2433:0:(service.c:2368:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff8800a4865880 x1810735692213376/t0(0) o38->9cdfd1aa-de4f-44c6-b6dd-d33822cabee4@192.168.202.9@tcp:0/0 lens 520/416 e 0 to 0 dl 1726854017 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0
[ 1957.709350] Lustre: lustre-MDT0000: Client 9cdfd1aa-de4f-44c6-b6dd-d33822cabee4 (at 192.168.202.9@tcp) reconnecting
[ 1957.717613] Lustre: Skipped 2 previous similar messages
[ 1957.720294] LustreError: 2137:0:(ldlm_lib.c:1330:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 1997.721313] LustreError: 2137:0:(ldlm_lib.c:1330:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 1997.724595] Lustre: 2137:0:(service.c:2368:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff880090807b80 x1810735692215424/t0(0) o38->9cdfd1aa-de4f-44c6-b6dd-d33822cabee4@192.168.202.9@tcp:0/0 lens 520/416 e 0 to 0 dl 1726854060 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0
[ 1997.734663] LustreError: 2137:0:(ldlm_lib.c:1330:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 2022.742640] Lustre: lustre-MDT0000: Export ffff8800b1ebe000 already connecting from 192.168.202.9@tcp
[ 2022.750410] Lustre: Skipped 6 previous similar messages
[ 2037.791361] LustreError: 2137:0:(ldlm_lib.c:1330:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 2037.794700] Lustre: 2137:0:(service.c:2368:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff8800a4aa1f80 x1810735692217216/t0(0) o38->9cdfd1aa-de4f-44c6-b6dd-d33822cabee4@192.168.202.9@tcp:0/0 lens 520/416 e 0 to 0 dl 1726854100 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0
[ 2040.715448] LustreError: 2137:0:(ldlm_lib.c:1330:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 2061.456724] LustreError: 2137:0:(ldlm_lib.c:1330:target_handle_connect()) cfs_fail_timeout interrupted
[ 2061.468964] Lustre: 2137:0:(service.c:2368:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/1s); client may timeout  req@ffff880092e97b80 x1810735692219264/t0(0) o38->9cdfd1aa-de4f-44c6-b6dd-d33822cabee4@192.168.202.9@tcp:0/0 lens 520/416 e 0 to 0 dl 1726854143 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0
[ 2063.815031] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 13:42:27 (1726854147)
[ 2065.548352] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2067.669308] Lustre: Failing over lustre-MDT0000
[ 2067.826043] Lustre: server umount lustre-MDT0000 complete
[ 2071.295706] Lustre: *** cfs_fail_loc=712, val=0***
[ 2071.298809] LustreError: 19420:0:(service.c:1237:ptlrpc_check_req()) @@@ Invalid replay without recovery  req@ffff880130150e00 x1810735695124480/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0
[ 2071.379974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2071.383350] Lustre: Skipped 12 previous similar messages
[ 2071.420380] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 2071.420522] Lustre: lustre-MDT0000: Aborting client recovery
[ 2071.420526] LustreError: 6710:0:(ldlm_lib.c:2898:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 2071.429328] Lustre: Skipped 23 previous similar messages
[ 2071.431417] Lustre: 6847:0:(ldlm_lib.c:2281:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 2071.435032] Lustre: 6847:0:(ldlm_lib.c:2281:target_recovery_overseer()) Skipped 2 previous similar messages
[ 2071.444631] Lustre: 6847:0:(genops.c:1529:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 9cdfd1aa-de4f-44c6-b6dd-d33822cabee4@<unknown>
[ 2071.450437] Lustre: lustre-MDT0000: disconnecting 1 stale clients
[ 2071.488639] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0]
[ 2071.588310] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2867 to 0x240000400:2945)
[ 2071.591365] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2866 to 0x280000400:2913)
[ 2073.093967] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2077.838111] Lustre: Failing over lustre-MDT0000
[ 2078.061857] Lustre: server umount lustre-MDT0000 complete
[ 2091.105111] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2091.118065] Lustre: Skipped 20 previous similar messages
[ 2092.536749] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2092.922133] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2866 to 0x280000400:2945)
[ 2092.922226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2867 to 0x240000400:2977)
[ 2095.910785] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2096.163789] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to  (at 0@lo)
[ 2096.168100] Lustre: Skipped 20 previous similar messages
[ 2096.479642] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2097.162351] Lustre: 3304:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1726854164/real 1726854164]  req@ffff88009aa27b80 x1810735695132672/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1726854180 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 2097.173159] Lustre: 3304:0:(client.c:2363:ptlrpc_expire_one_request()) Skipped 44 previous similar messages
[ 2100.162808] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 13:43:03 (1726854183)
[ 2100.205749] Lustre: lustre-MDT0000: Client 9cdfd1aa-de4f-44c6-b6dd-d33822cabee4 (at 192.168.202.9@tcp) reconnecting
[ 2100.213970] Lustre: Skipped 2 previous similar messages
[ 2103.661141] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 13:43:07 (1726854187)
[ 2104.002885] Lustre: *** cfs_fail_loc=122, val=2147483648***
[ 2104.005282] LustreError: 8168:0:(ldlm_lib.c:3242:target_send_reply_msg()) @@@ dropping reply  req@ffff8800a4867100 x1810735692269952/t0(0) o700->9cdfd1aa-de4f-44c6-b6dd-d33822cabee4@192.168.202.9@tcp:77/0 lens 264/248 e 0 to 0 dl 1726854197 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0
[ 2121.084706] Lustre: Failing over lustre-MDT0000
[ 2121.257982] Lustre: server umount lustre-MDT0000 complete
[ 2133.866756] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2133.871105] LustreError: Skipped 7 previous similar messages
[ 2134.153461] mount.lustre (10518) used greatest stack depth: 9744 bytes left
[ 2134.519279] hrtimer: interrupt took 5074675 ns
[ 2135.468853] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2137.929087] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2947 to 0x280000400:2977)
[ 2137.929124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2979 to 0x240000400:3009)
[ 2139.425906] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2140.145840] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2144.704661] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 13:43:48 (1726854228)
[ 2145.879443] Lustre: Failing over lustre-OST0000
[ 2145.906455] Lustre: server umount lustre-OST0000 complete
[ 2148.077125] LustreError: lustre-OST0000: not available for connect from 192.168.202.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2148.083389] LustreError: Skipped 1 previous similar message
[ 2149.042288] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 2149.047916] LustreError: Skipped 2 previous similar messages
[ 2158.070971] LustreError: lustre-OST0000: not available for connect from 192.168.202.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2158.076366] LustreError: Skipped 3 previous similar messages
[ 2161.456357] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2165.844299] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2166.814928] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2232.810787] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 13:45:16 (1726854316)
[ 2234.079382] LustreError: 13688:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 2234.082630] LustreError: 13688:0:(osd_handler.c:698:osd_ro()) Skipped 7 previous similar messages
[ 2234.521985] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2235.570459] Lustre: Failing over lustre-MDT0000
[ 2235.709047] Lustre: server umount lustre-MDT0000 complete
[ 2249.865915] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2252.354642] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 2252.358133] Lustre: Skipped 6 previous similar messages
[ 2252.658492] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 2252.661400] Lustre: Skipped 6 previous similar messages
[ 2252.693753] Lustre: *** cfs_fail_loc=216, val=0***
[ 2252.695109] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2998 to 0x280000400:3041)
[ 2252.704945] LustreError: 14558:0:(osp_precreate.c:992:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -30
[ 2253.714137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3030 to 0x240000400:3073)
[ 2316.283266] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 13:46:39 (1726854399)
[ 2325.086711] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 13:46:48 (1726854408)
[ 2326.518914] Lustre: Failing over lustre-MDT0000
[ 2326.693027] Lustre: server umount lustre-MDT0000 complete
[ 2340.918425] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2342.989378] Lustre: *** cfs_fail_loc=157, val=2147483648***
[ 2342.992646] LustreError: 17155:0:(ldlm_lib.c:3242:target_send_reply_msg()) @@@ dropping reply  req@ffff88009d342d80 x1810735692381952/t0(0) o101->9cdfd1aa-de4f-44c6-b6dd-d33822cabee4@192.168.202.9@tcp:316/0 lens 328/344 e 0 to 0 dl 1726854436 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0
[ 2359.375011] Lustre: lustre-MDT0000: Client 9cdfd1aa-de4f-44c6-b6dd-d33822cabee4 (at 192.168.202.9@tcp) reconnected, waiting for 1 clients in recovery for 0:53
[ 2359.432791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3085 to 0x240000400:3105)
[ 2359.432968] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3052 to 0x280000400:3073)
[ 2360.725913] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2361.277159] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2365.252426] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 13:47:28 (1726854448)
[ 2366.644807] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 2368.307943] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2368.956907] Lustre: Failing over lustre-MDT0000
[ 2369.063790] Lustre: server umount lustre-MDT0000 complete
[ 2382.486523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3085 to 0x240000400:3137)
[ 2382.486773] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3075 to 0x280000400:3105)
[ 2382.839838] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2385.905407] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2386.485506] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2390.145874] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 13:47:53 (1726854473)
[ 2390.678683] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 2393.424322] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2394.157918] Lustre: Failing over lustre-MDT0000
[ 2394.309167] Lustre: server umount lustre-MDT0000 complete
[ 2406.717595] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3075 to 0x280000400:3137)
[ 2406.717798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3139 to 0x240000400:3169)
[ 2407.597263] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2410.419786] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2410.856794] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2414.538707] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 13:48:18 (1726854498)
[ 2414.975383] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 2417.487828] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2418.209287] Lustre: Failing over lustre-MDT0000
[ 2418.372749] Lustre: server umount lustre-MDT0000 complete
[ 2432.197765] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2432.963550] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3139 to 0x280000400:3169)
[ 2432.963558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3139 to 0x240000400:3201)
[ 2438.043520] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 13:48:41 (1726854521)
[ 2439.519283] Lustre: *** cfs_fail_loc=13b, val=315***
[ 2439.521726] Lustre: *** cfs_fail_loc=13b, val=2147483648***
[ 2439.525038] LustreError: 23887:0:(ldlm_lib.c:3242:target_send_reply_msg()) @@@ dropping reply  req@ffff880094621180 x1810735692424576/t257698037777(0) o35->9cdfd1aa-de4f-44c6-b6dd-d33822cabee4@192.168.202.9@tcp:413/0 lens 392/456 e 0 to 0 dl 1726854533 ref 1 fl Interpret:/200/0 rc 0/0 job:'multiop.0' uid:0 gid:0
[ 2440.810077] Lustre: Failing over lustre-MDT0000
[ 2440.992523] Lustre: server umount lustre-MDT0000 complete
[ 2455.216766] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2455.557956] Lustre: 25743:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno  req@ffff880093e51f80 x1810735692424576/t257698037777(0) o35->9cdfd1aa-de4f-44c6-b6dd-d33822cabee4@192.168.202.9@tcp:429/0 lens 392/456 e 0 to 0 dl 1726854549 ref 1 fl Interpret:/202/0 rc 0/0 job:'multiop.0' uid:0 gid:0
[ 2455.567291] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3171 to 0x280000400:3201)
[ 2455.567295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3139 to 0x240000400:3233)
[ 2458.236484] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2458.706670] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2461.850252] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 13:49:05 (1726854545)
[ 2462.212099] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 2462.214320] LustreError: 26543:0:(ldlm_lib.c:3242:target_send_reply_msg()) @@@ dropping reply  req@ffff88009bac5500 x1810735692436480/t261993005072(0) o36->9cdfd1aa-de4f-44c6-b6dd-d33822cabee4@192.168.202.9@tcp:436/0 lens 504/448 e 0 to 0 dl 1726854556 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0
[ 2464.768391] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2465.424108] Lustre: Failing over lustre-MDT0000
[ 2465.555860] Lustre: server umount lustre-MDT0000 complete
[ 2478.240518] Lustre: 28020:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno  req@ffff88009bac5500 x1810735692436480/t261993005072(0) o36->9cdfd1aa-de4f-44c6-b6dd-d33822cabee4@192.168.202.9@tcp:452/0 lens 504/2880 e 0 to 0 dl 1726854572 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0
[ 2478.248876] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3203 to 0x280000400:3233)
[ 2478.248891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3139 to 0x240000400:3265)
[ 2479.206751] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2482.101712] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2482.539888] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2485.682466] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 13:49:29 (1726854569)
[ 2486.002117] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 2486.004269] LustreError: 28022:0:(ldlm_lib.c:3242:target_send_reply_msg()) @@@ dropping reply  req@ffff8800a602d500 x1810735692448384/t266287972368(0) o36->9cdfd1aa-de4f-44c6-b6dd-d33822cabee4@192.168.202.9@tcp:459/0 lens 504/448 e 0 to 0 dl 1726854579 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0
[ 2487.286010] Lustre: *** cfs_fail_loc=13b, val=315***
[ 2488.400885] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2489.014401] Lustre: Failing over lustre-MDT0000
[ 2489.142280] Lustre: server umount lustre-MDT0000 complete
[ 2502.064483] Lustre: 30242:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno  req@ffff880094341180 x1810735692448384/t266287972368(0) o36->9cdfd1aa-de4f-44c6-b6dd-d33822cabee4@192.168.202.9@tcp:475/0 lens 504/2880 e 0 to 0 dl 1726854595 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0
[ 2502.065906] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3235 to 0x280000400:3265)
[ 2502.069403] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3139 to 0x240000400:3297)
[ 2502.083692] Lustre: 30242:0:(mdt_recovery.c:148:mdt_req_from_lrd()) Skipped 1 previous similar message
[ 2503.317114] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2509.245405] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 13:49:52 (1726854592)
[ 2509.701320] LustreError: 30243:0:(ldlm_lib.c:3242:target_send_reply_msg()) @@@ dropping reply  req@ffff880093e51c00 x1810735692459648/t270582939664(0) o36->9cdfd1aa-de4f-44c6-b6dd-d33822cabee4@192.168.202.9@tcp:483/0 lens 504/448 e 0 to 0 dl 1726854603 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0
[ 2509.710752] LustreError: 30243:0:(ldlm_lib.c:3242:target_send_reply_msg()) Skipped 1 previous similar message
[ 2511.031092] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 2511.034594] Lustre: Skipped 1 previous similar message
[ 2512.786545] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2513.469952] Lustre: Failing over lustre-MDT0000
[ 2513.605493] Lustre: server umount lustre-MDT0000 complete
[ 2527.228304] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2535.736904] Lustre: 32426:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno  req@ffff88012d3f0a80 x1810735692459648/t270582939664(0) o36->9cdfd1aa-de4f-44c6-b6dd-d33822cabee4@192.168.202.9@tcp:509/0 lens 504/2880 e 0 to 0 dl 1726854629 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0
[ 2535.739276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3139 to 0x240000400:3329)
[ 2535.739580] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3267 to 0x280000400:3297)
[ 2539.844620] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 13:50:23 (1726854623)
[ 2540.164667] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 2541.430603] Lustre: *** cfs_fail_loc=13b, val=315***
[ 2541.433341] Lustre: *** cfs_fail_loc=13b, val=2147483648***
[ 2541.436091] Lustre: Skipped 2 previous similar messages
[ 2543.468607] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2544.039964] Lustre: Failing over lustre-MDT0000
[ 2544.157994] Lustre: server umount lustre-MDT0000 complete
[ 2557.582902] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2566.182247] Lustre: 2073:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno  req@ffff8800a5d13100 x1810735692471168/t274877906960(0) o35->9cdfd1aa-de4f-44c6-b6dd-d33822cabee4@192.168.202.9@tcp:540/0 lens 392/456 e 0 to 0 dl 1726854660 ref 1 fl Interpret:/202/0 rc 0/0 job:'multiop.0' uid:0 gid:0
[ 2566.190255] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3299 to 0x280000400:3329)
[ 2566.190257] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3139 to 0x240000400:3361)
[ 2570.222749] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 13:50:53 (1726854653)
[ 2570.499974] Lustre: *** cfs_fail_loc=12b, val=2147483991***
[ 2570.503012] LustreError: 2643:0:(ldlm_lib.c:3242:target_send_reply_msg()) @@@ dropping reply  req@ffff88009c316680 x1810735692480128/t279172874255(0) o101->9cdfd1aa-de4f-44c6-b6dd-d33822cabee4@192.168.202.9@tcp:544/0 lens 664/608 e 0 to 0 dl 1726854664 ref 1 fl Interpret:/200/0 rc 301/0 job:'touch.0' uid:0 gid:0
[ 2570.511735] LustreError: 2643:0:(ldlm_lib.c:3242:target_send_reply_msg()) Skipped 1 previous similar message
[ 2586.500391] Lustre: lustre-MDT0000: Client 9cdfd1aa-de4f-44c6-b6dd-d33822cabee4 (at 192.168.202.9@tcp) reconnecting
[ 2586.503874] Lustre: Skipped 4 previous similar messages
[ 2586.507865] Lustre: 2071:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno  req@ffff88009c742a00 x1810735692480128/t279172874255(0) o101->9cdfd1aa-de4f-44c6-b6dd-d33822cabee4@192.168.202.9@tcp:560/0 lens 664/3488 e 0 to 0 dl 1726854680 ref 1 fl Interpret:/202/0 rc 0/0 job:'touch.0' uid:0 gid:0
[ 2588.356161] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 13:51:12 (1726854672)
[ 2589.797164] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2590.511752] Lustre: Failing over lustre-MDT0000
[ 2590.636351] Lustre: server umount lustre-MDT0000 complete
[ 2603.663399] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2605.545697] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3299 to 0x280000400:3361)
[ 2605.545703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3363 to 0x240000400:3393)
[ 2606.520080] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2606.978622] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2620.123680] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 13:51:43 (1726854703)
[ 2621.592547] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2622.163154] Lustre: Failing over lustre-MDT0000
[ 2622.295240] Lustre: server umount lustre-MDT0000 complete
[ 2635.351013] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2636.622899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3363 to 0x240000400:3425)
[ 2636.623161] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3363 to 0x280000400:3393)
[ 2638.195851] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2638.650866] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2640.587974] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 2645.554988] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 13:52:09 (1726854729)
[ 2659.881658] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2660.558612] Lustre: Failing over lustre-MDT0000
[ 2660.753138] Lustre: server umount lustre-MDT0000 complete
[ 2673.358010] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2673.360891] Lustre: Skipped 15 previous similar messages
[ 2673.387677] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 2673.390789] Lustre: Skipped 17 previous similar messages
[ 2674.537838] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2676.650513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4676 to 0x240000400:4705)
[ 2676.650653] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4644 to 0x280000400:4673)
[ 2677.755577] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2678.235596] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2707.448816] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 13:53:11 (1726854791)
[ 2709.207056] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2710.072131] Lustre: Failing over lustre-MDT0000
[ 2710.235706] Lustre: server umount lustre-MDT0000 complete
[ 2722.749778] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2722.757372] Lustre: Skipped 32 previous similar messages
[ 2723.719450] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2727.664129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4707 to 0x240000400:4737)
[ 2727.668807] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4644 to 0x280000400:4705)
[ 2727.778450] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to  (at 0@lo)
[ 2727.780336] Lustre: Skipped 32 previous similar messages
[ 2728.576604] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2729.037687] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2729.777347] Lustre: 3305:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1726854796/real 1726854796]  req@ffff88008f941880 x1810735696013056/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1726854812 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 2729.786609] Lustre: 3305:0:(client.c:2363:ptlrpc_expire_one_request()) Skipped 72 previous similar messages
[ 2731.572846] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid
[ 2732.021802] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec
[ 2733.657410] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 13:53:37 (1726854817)
[ 2739.228684] Lustre: *** cfs_fail_loc=123, val=2147483648***
[ 2755.888978] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 2755.891531] Lustre: Skipped 1 previous similar message
[ 2755.894822] LustreError: 13086:0:(ldlm_lib.c:3242:target_send_reply_msg()) @@@ dropping reply  req@ffff8800a6efd850 x1810735695417856/t296352743433(0) o36->9cdfd1aa-de4f-44c6-b6dd-d33822cabee4@192.168.202.9@tcp:729/0 lens 66040/440 e 0 to 0 dl 1726854849 ref 1 fl Interpret:/200/0 rc 0/0 job:'setfattr.0' uid:0 gid:0
[ 2771.889753] Lustre: 12455:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno  req@ffff88009bac7480 x1810735695417856/t296352743433(0) o36->9cdfd1aa-de4f-44c6-b6dd-d33822cabee4@192.168.202.9@tcp:745/0 lens 66040/440 e 0 to 0 dl 1726854865 ref 1 fl Interpret:/202/0 rc 0/0 job:'setfattr.0' uid:0 gid:0
[ 2774.036148] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59
[ 2774.446086] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 13:54:18 (1726854858)
[ 2776.965352] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2777.676270] Lustre: Failing over lustre-MDT0000
[ 2777.786242] Lustre: server umount lustre-MDT0000 complete
[ 2789.635345] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2789.639357] LustreError: Skipped 14 previous similar messages
[ 2790.762697] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2792.171673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4839 to 0x240000400:4865)
[ 2792.171675] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4806 to 0x280000400:4833)
[ 2793.502964] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2793.923906] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2797.311382] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 13:54:41 (1726854881)
[ 2802.014294] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 2805.418003] Lustre: Failing over lustre-OST0000
[ 2805.451271] Lustre: server umount lustre-OST0000 complete
[ 2806.954553] LustreError: lustre-OST0000: not available for connect from 192.168.202.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2807.170727] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107
[ 2807.174489] LustreError: Skipped 6 previous similar messages
[ 2809.794078] LustreError: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2809.801636] LustreError: Skipped 1 previous similar message
[ 2814.803089] LustreError: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2814.808889] LustreError: Skipped 1 previous similar message
[ 2819.415285] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2830.636001] Lustre: Failing over lustre-OST0000
[ 2830.649922] Lustre: server umount lustre-OST0000 complete
[ 2831.992722] LustreError: lustre-OST0000: not available for connect from 192.168.202.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2831.998872] LustreError: Skipped 1 previous similar message
[ 2844.949923] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2847.959743] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2848.412653] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2882.086293] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 13:56:05 (1726854965)
[ 2883.041670] Lustre: Failing over lustre-MDT0000
[ 2883.187149] Lustre: server umount lustre-MDT0000 complete
[ 2896.673727] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2897.101634] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 2897.105042] Lustre: Skipped 16 previous similar messages
[ 2897.131073] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 2897.134097] Lustre: Skipped 16 previous similar messages
[ 2897.147657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5266 to 0x240000400:5281)
[ 2897.147765] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5234 to 0x280000400:5249)
[ 2908.017750] Lustre: Failing over lustre-MDT0000
[ 2908.139245] Lustre: server umount lustre-MDT0000 complete
[ 2922.208759] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5234 to 0x280000400:5281)
[ 2922.208904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5266 to 0x240000400:5313)
[ 2922.368105] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2925.588851] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2926.205175] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2929.650349] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 13:56:53 (1726855013)
[ 2940.770669] Lustre: Failing over lustre-OST0000
[ 2940.781601] Lustre: server umount lustre-OST0000 complete
[ 2941.089635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 2941.092637] LustreError: Skipped 1 previous similar message
[ 2941.094388] LustreError: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2941.099351] LustreError: Skipped 5 previous similar messages
[ 2954.479802] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2957.654066] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2958.141588] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2961.574962] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 13:57:25 (1726855045)
[ 2962.141932] Lustre: Failing over lustre-MDT0000
[ 2962.201058] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.9@tcp (stopping)
[ 2962.206829] Lustre: Skipped 1 previous similar message
[ 2962.270897] Lustre: server umount lustre-MDT0000 complete
[ 2964.945872] Lustre: *** cfs_fail_loc=605, val=0***
[ 2964.947547] LustreError: 27123:0:(llog_obd.c:207:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffa054ef30 failed: rc = -95
[ 2964.951062] LustreError: 27123:0:(obd_config.c:797:class_setup()) setup MGS failed (-95)
[ 2964.953831] LustreError: 27123:0:(obd_mount.c:214:lustre_start_simple()) MGS setup error -95
[ 2964.956719] LustreError: 27123:0:(tgt_mount.c:136:server_deregister_mount()) MGS not registered
[ 2964.959566] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded?
[ 2964.961944] LustreError: 27123:0:(tgt_mount.c:1740:server_put_super()) no obd lustre-MDT0000
[ 2964.971884] Lustre: server umount lustre-MDT0000 complete
[ 2964.973589] LustreError: 27123:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -95
[ 2967.241497] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5283 to 0x280000400:5313)
[ 2967.241564] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5315 to 0x240000400:5345)
[ 2967.724633] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2970.224274] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 13:57:33 (1726855053)
[ 2971.256036] LustreError: 28515:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 2971.259570] LustreError: 28515:0:(osd_handler.c:698:osd_ro()) Skipped 13 previous similar messages
[ 2971.574858] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2972.846722] Lustre: Failing over lustre-MDT0000
[ 2973.013616] Lustre: server umount lustre-MDT0000 complete
[ 2987.142715] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2987.249905] Lustre: *** cfs_fail_loc=707, val=0***
[ 3003.277567] Lustre: lustre-MDT0000: Client 9cdfd1aa-de4f-44c6-b6dd-d33822cabee4 (at 192.168.202.9@tcp) reconnected, waiting for 1 clients in recovery for 0:53
[ 3003.526667] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5326 to 0x280000400:5345)
[ 3003.526748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5359 to 0x240000400:5377)
[ 3004.874746] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3005.528928] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3009.781610] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 13:58:13 (1726855093)
[ 3033.601727] LustreError: 17873:0:(service.c:2347:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms
[ 3034.636785] Lustre: DEBUG MARKER: replay-single test_65a: @@@@@@ FAIL: No early reply