[ 1806.737790] LustreError: lustre-OST0000-osc-ffff8dc1d2a33000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1806.748461] Lustre: 2360:0:(llite_lib.c:4352:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.136@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1813.278829] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 10:38:06 (1787841486)
[ 1815.410862] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1817.091691] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 10:38:09 (1787841489)
[ 1818.989244] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1820.846986] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 10:38:13 (1787841493)
[ 1824.154224] LustreError: lustre-MDT0000-mdc-ffff8dc1d2a33000: operation ldlm_enqueue to node 192.168.206.136@tcp failed: rc = -19
[ 1824.171045] LustreError: Skipped 1 previous similar message
[ 1843.699370] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 192.168.206.136@tcp) was lost; in progress operations using this service will fail
[ 1843.721838] Lustre: Evicted from MGS (at 192.168.206.136@tcp) after server handle changed from 0x331488887394c477 to 0x331488887394dcfe
[ 1845.230797] Lustre: 15916:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.136@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 1952.798916] LustreError: lustre-MDT0000-mdc-ffff8dc1d2a33000: operation ldlm_enqueue to node 192.168.206.136@tcp failed: rc = -19
[ 1971.680284] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 192.168.206.136@tcp) was lost; in progress operations using this service will fail
[ 1971.721063] Lustre: Evicted from MGS (at 192.168.206.136@tcp) after server handle changed from 0x331488887394dcfe to 0x331488887396b3ef
[ 1997.252635] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 10:41:10 (1787841670)
[ 1997.670146] Lustre: *** cfs_fail_loc=305, val=0***
[ 1997.673361] Lustre: Skipped 2 previous similar messages
[ 2017.884899] LustreError: lustre-OST0001-osc-ffff8dc1d2a33000: operation ost_connect to node 192.168.206.136@tcp failed: rc = -75
[ 2042.335649] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 192.168.206.136@tcp) was lost; in progress operations using this service will fail
[ 2042.372222] Lustre: Evicted from MGS (at 192.168.206.136@tcp) after server handle changed from 0x331488887396b3ef to 0x331488887396b86b
[ 2043.887961] Lustre: 15916:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.136@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2065.513329] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2067.016273] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2074.901478] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 10:42:27 (1787841747)
[ 2087.401046] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 192.168.206.136@tcp) was lost; in progress operations using this service will fail
[ 2087.412557] Lustre: lustre-MDT0000-mdc-ffff8dc1d2a33000: Connection to lustre-MDT0000 (at 192.168.206.136@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2087.429489] Lustre: Evicted from MGS (at 192.168.206.136@tcp) after server handle changed from 0x331488887396b86b to 0x331488887396bdea
[ 2087.440792] Lustre: Skipped 12 previous similar messages
[ 2087.494685] Lustre: MGC192.168.206.136@tcp: Connection restored to 192.168.206.136@tcp (at 192.168.206.136@tcp)
[ 2087.512132] Lustre: Skipped 14 previous similar messages
[ 2087.560848] LustreError: lustre-MDT0000-mdc-ffff8dc1d2a33000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2111.858846] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 10:43:04 (1787841784)
[ 2128.368208] LustreError: lustre-OST0000-osc-ffff8dc1d2a33000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2146.316767] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 10:43:39 (1787841819)
[ 2160.587241] LustreError: lustre-MDT0000-mdc-ffff8dc1d2a33000: operation ldlm_enqueue to node 192.168.206.136@tcp failed: rc = -19
[ 2180.575795] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 192.168.206.136@tcp) was lost; in progress operations using this service will fail
[ 2190.771948] Lustre: Evicted from MGS (at 192.168.206.136@tcp) after server handle changed from 0x331488887396bdea to 0x3314888873970ca3
[ 2208.232643] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2210.993464] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2296.297942] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 192.168.206.136@tcp) was lost; in progress operations using this service will fail
[ 2296.338333] Lustre: Evicted from MGS (at 192.168.206.136@tcp) after server handle changed from 0x3314888873970ca3 to 0x331488887398c45a
[ 2314.391797] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2317.007295] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2402.794664] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 192.168.206.136@tcp) was lost; in progress operations using this service will fail
[ 2402.821922] Lustre: Evicted from MGS (at 192.168.206.136@tcp) after server handle changed from 0x331488887398c45a to 0x33148888739a9ca9
[ 2418.746586] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2421.401772] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2450.804522] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 10:48:43 (1787842123)
[ 2454.691430] LustreError: lustre-MDT0000-mdc-ffff8dc1d2a33000: operation ldlm_enqueue to node 192.168.206.136@tcp failed: rc = -19
[ 2454.702651] LustreError: Skipped 4 previous similar messages
[ 2474.491773] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 192.168.206.136@tcp) was lost; in progress operations using this service will fail
[ 2474.514801] Lustre: Evicted from MGS (at 192.168.206.136@tcp) after server handle changed from 0x33148888739a9ca9 to 0x33148888739b92f2
[ 2482.292900] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2513.396067] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2549.358547] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2580.943189] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787842239/real 1787842239]  req@ffff8dc1d01ed2c0 x1874685828304256/t0(0) o400->MGC192.168.206.136@tcp@192.168.206.136@tcp:26/25 lens 224/224 e 0 to 1 dl 1787842255 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2580.970361] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
[ 2588.154681] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2631.142509] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 192.168.206.136@tcp) was lost; in progress operations using this service will fail
[ 2631.161697] LustreError: Skipped 3 previous similar messages
[ 2631.179818] Lustre: Evicted from MGS (at 192.168.206.136@tcp) after server handle changed from 0x33148888739cb7b7 to 0x33148888739d00ab
[ 2631.187211] Lustre: Skipped 3 previous similar messages
[ 2639.269726] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2687.578048] Lustre: lustre-MDT0000-mdc-ffff8dc1d2a33000: Connection restored to 192.168.206.136@tcp (at 192.168.206.136@tcp)
[ 2687.588286] Lustre: Skipped 19 previous similar messages
[ 2695.041614] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2727.971261] Lustre: lustre-MDT0000-mdc-ffff8dc1d2a33000: Connection to lustre-MDT0000 (at 192.168.206.136@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2727.991290] Lustre: Skipped 10 previous similar messages
[ 2780.328487] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 10:54:13 (1787842453)
[ 2843.160275] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2844.949130] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3125.743651] LustreError: lustre-OST0000-osc-ffff8dc1d2a33000: operation ldlm_enqueue to node 192.168.206.136@tcp failed: rc = -19
[ 3125.761014] LustreError: Skipped 12 previous similar messages
[ 3166.660893] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3169.052402] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3456.902122] Lustre: lustre-OST0000-osc-ffff8dc1d2a33000: Connection to lustre-OST0000 (at 192.168.206.136@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3456.914543] Lustre: Skipped 2 previous similar messages
[ 3480.007605] Lustre: lustre-OST0000-osc-ffff8dc1d2a33000: Connection restored to 192.168.206.136@tcp (at 192.168.206.136@tcp)
[ 3480.019374] Lustre: Skipped 4 previous similar messages
[ 3494.521604] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3496.412966] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3750.849421] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 11:10:23 (1787843423)
[ 3767.776129] Lustre: 50564:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787843426/real 1787843426]  req@ffff8dc1d11aad00 x1874685847015296/t0(0) o101->lustre-MDT0000-mdc-ffff8dc1d2a33000@192.168.206.136@tcp:12/10 lens 576/1152 e 0 to 1 dl 1787843442 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3767.801870] Lustre: 50564:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 3774.926515] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 11:10:47 (1787843447)
[ 3799.708831] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 11:11:12 (1787843472)
[ 3825.795506] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 11:11:38 (1787843498)
[ 3826.335099] Lustre: Mounted lustre-client - version 2.17.57_95_g5b39979
[ 3857.388776] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 192.168.206.136@tcp) was lost; in progress operations using this service will fail
[ 3857.402033] LustreError: Skipped 2 previous similar messages
[ 3857.419268] Lustre: Evicted from MGS (at 192.168.206.136@tcp) after server handle changed from 0x33148888739eca91 to 0x3314888873b2a5fd
[ 3857.424807] Lustre: Skipped 2 previous similar messages
[ 3859.250214] Lustre: 15916:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.136@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 3880.924785] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3882.774285] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3884.971613] Lustre: Unmounted lustre-client
[ 3891.055735] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 11:12:43 (1787843563)
[ 4060.127951] Lustre: lustre-OST0000-osc-ffff8dc1d2a33000: Connection to lustre-OST0000 (at 192.168.206.136@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4060.145535] Lustre: Skipped 14 previous similar messages
[ 4091.886581] Lustre: lustre-OST0000-osc-ffff8dc1d2a33000: Connection restored to 192.168.206.136@tcp (at 192.168.206.136@tcp)
[ 4091.898989] Lustre: Skipped 17 previous similar messages
[ 4184.797121] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 11:17:37 (1787843857)
[ 4234.404966] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 11:18:27 (1787843907)
[ 4237.739534] Lustre: Unmounted lustre-client
[ 4261.007822] Lustre: Mounted lustre-client - version 2.17.57_95_g5b39979
[ 4269.072614] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 11:19:01 (1787843941)
[ 4269.429570] LustreError: 56496:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4270.480417] LustreError: 56496:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4270.672471] Lustre: *** cfs_fail_loc=305, val=0***
[ 4294.480698] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 11:19:27 (1787843967)
[ 4295.044370] Lustre: Mounted lustre-client - version 2.17.57_95_g5b39979
[ 4297.824140] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4308.130856] Lustre: Unmounted lustre-client
[ 4316.565049] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 11:19:49 (1787843989)
[ 4410.272850] LustreError: lustre-MDT0000-mdc-ffff8dc1c7e80000: operation mds_reint to node 192.168.206.136@tcp failed: rc = -19
[ 4410.295511] LustreError: Skipped 1 previous similar message
[ 4429.282863] Lustre: 2360:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787844087/real 1787844087]  req@ffff8dc1c435d2c0 x1874685849750016/t0(0) o400->MGC192.168.206.136@tcp@192.168.206.136@tcp:26/25 lens 224/224 e 0 to 1 dl 1787844103 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4429.310453] Lustre: 2360:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 151 previous similar messages
[ 4429.319250] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 192.168.206.136@tcp) was lost; in progress operations using this service will fail
[ 4439.525191] Lustre: Evicted from MGS (at 192.168.206.136@tcp) after server handle changed from 0x3314888873b2c2c1 to 0x3314888873b577e4
[ 4595.681115] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 11:24:28 (1787844268)
[ 4599.904774] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4612.596575] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 192.168.206.136@tcp) was lost; in progress operations using this service will fail
[ 4612.619936] Lustre: Evicted from MGS (at 192.168.206.136@tcp) after server handle changed from 0x3314888873b577e4 to 0x3314888873ba403c
[ 4621.888909] LustreError: lustre-MDT0000-mdc-ffff8dc1c7e80000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4630.026848] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 11:25:02 (1787844302)
[ 4630.298634] Lustre: Mounted lustre-client - version 2.17.57_95_g5b39979
[ 4637.155481] LustreError: lustre-OST0000-osc-ffff8dc1c7e80000: operation ldlm_enqueue to node 192.168.206.136@tcp failed: rc = -107
[ 4637.167583] LustreError: lustre-OST0000-osc-ffff8dc1c7e80000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4637.178340] LustreError: 60217:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8dc1c7e80000: namespace resource [0x240000400:0x3ee3:0x0].0x0 (ffff8dc1c54c8d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4649.184063] Lustre: Unmounted lustre-client
[ 4655.415653] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 11:25:28 (1787844328)
[ 4661.574686] Lustre: lustre-MDT0000-mdc-ffff8dc1c7e80000: Connection to lustre-MDT0000 (at 192.168.206.136@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4661.584816] Lustre: Skipped 15 previous similar messages
[ 4661.594128] LustreError: lustre-MDT0000-mdc-ffff8dc1c7e80000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4661.604904] LustreError: 60888:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4661.605959] LustreError: 60972:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8dc1c7e80000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8dc1c54c8e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4667.561771] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 11:25:40 (1787844340)
[ 4667.678088] LustreError: 61585:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4671.638954] LustreError: lustre-MDT0000-mdc-ffff8dc1c7e80000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4671.655960] LustreError: 61607:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4671.660969] LustreError: 61585:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1020
[ 4671.668501] LustreError: 61585:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4671.678663] LustreError: 61612:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4672.795789] LustreError: 61619:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4672.826430] LustreError: 61619:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4675.330802] LustreError: 61642:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4675.341032] LustreError: 61642:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4680.114886] LustreError: 61686:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4680.132987] LustreError: 61686:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4689.521485] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 11:26:02 (1787844362)
[ 4722.937028] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4724.461659] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4733.390858] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 11:26:46 (1787844406)
[ 4756.069413] Lustre: lustre-OST0000-osc-ffff8dc1c7e80000: Connection restored to 192.168.206.136@tcp (at 192.168.206.136@tcp)
[ 4756.074164] Lustre: Skipped 17 previous similar messages
[ 4768.552304] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4770.067254] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4779.139786] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 11:27:32 (1787844452)
[ 4839.127482] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4841.055670] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4849.499690] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 11:28:42 (1787844522)
[ 4885.141667] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4886.469307] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4892.235795] Lustre: Unmounted lustre-client
[ 4946.111378] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4948.567428] Lustre: Mounted lustre-client - version 2.17.57_95_g5b39979
[ 4955.066899] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 11:30:27 (1787844627)
[ 4957.893331] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4959.983820] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 11:30:32 (1787844632)
[ 4988.532222] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 11:31:01 (1787844661)
[ 4989.735781] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4991.541389] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 11:31:04 (1787844664)
[ 4992.613683] Lustre: *** cfs_fail_loc=805, val=0***
[ 4992.659122] Lustre: Mounted lustre-client - version 2.17.57_95_g5b39979
[ 4997.268506] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5018.605541] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 192.168.206.136@tcp) was lost; in progress operations using this service will fail
[ 5018.628752] Lustre: Evicted from MGS (at 192.168.206.136@tcp) after server handle changed from 0x3314888873ba4a30 to 0x3314888873ba4dda
[ 5024.966048] Lustre: Unmounted lustre-client
[ 5031.317044] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 11:31:44 (1787844704)
[ 5047.775964] Lustre: 71409:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787844706/real 1787844706]  req@ffff8dc1c431e1c0 x1874685851805312/t0(0) o36->lustre-MDT0000-mdc-ffff8dc1c7bdb800@192.168.206.136@tcp:12/10 lens 504/576 e 0 to 1 dl 1787844722 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'mkdir.0' uid:0 gid:0 projid:4294967295
[ 5047.802825] Lustre: 71409:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 5058.010928] LustreError: 2356:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8dc1c431c780 x1874685851804928/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff8dc1c7bdb800@192.168.206.136@tcp:12/10 lens 664/608 e 0 to 0 dl 1787844748 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5065.463857] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5067.237502] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5074.992090] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 11:32:28 (1787844748)
[ 5076.323949] LustreError: lustre-OST0000-osc-ffff8dc1c7bdb800: operation ost_write to node 192.168.206.136@tcp failed: rc = -107
[ 5076.331267] LustreError: Skipped 2 previous similar messages
[ 5076.355125] LustreError: lustre-OST0000-osc-ffff8dc1c7bdb800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5076.372660] Lustre: 2360:0:(llite_lib.c:4352:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.136@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5083.634183] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 11:32:36 (1787844756)
[ 5084.880820] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5086.308241] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 11:32:39 (1787844759)
[ 5087.724059] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5089.344217] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 11:32:42 (1787844762)
[ 5090.888114] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5092.541396] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 11:32:45 (1787844765)
[ 5094.001590] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5095.394340] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 11:32:48 (1787844768)
[ 5096.923922] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5098.438749] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 11:32:51 (1787844771)
[ 5099.669951] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5101.331742] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 11:32:54 (1787844774)
[ 5102.760655] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5104.196769] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 11:32:57 (1787844777)
[ 5105.438681] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5106.855667] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 11:33:00 (1787844780)
[ 5108.359985] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5109.808518] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 11:33:02 (1787844782)
[ 5111.030183] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5112.501418] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 11:33:05 (1787844785)
[ 5113.714491] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5115.003911] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 11:33:08 (1787844788)
[ 5116.879401] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5118.284181] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 11:33:11 (1787844791)
[ 5148.930879] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 11:33:41 (1787844821)
[ 5184.883109] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 11:34:17 (1787844857)
[ 5185.702694] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5197.424956] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 11:34:30 (1787844870)
[ 5197.775484] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5208.260121] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 11:34:41 (1787844881)
[ 5208.910705] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5218.042978] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 11:34:50 (1787844890)
[ 5218.368717] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5227.019723] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 11:34:59 (1787844899)
[ 5227.312480] Lustre: *** cfs_fail_loc=510, val=3***
[ 5235.066912] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 11:35:07 (1787844907)
[ 5235.589145] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5294.563352] Lustre: lustre-OST0000-osc-ffff8dc1c7bdb800: Connection to lustre-OST0000 (at 192.168.206.136@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5294.588567] Lustre: Skipped 12 previous similar messages
[ 5301.185688] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 11:36:14 (1787844974)
[ 5301.858277] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5367.993598] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 11:37:20 (1787845040)
[ 5368.362667] LustreError: 83002:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5371.337067] LustreError: lustre-MDT0000-mdc-ffff8dc1c7bdb800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5371.379958] LustreError: 83017:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8dc1c7bdb800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8dc1c54c8400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5371.407416] Lustre: lustre-MDT0000-mdc-ffff8dc1c7bdb800: Connection restored to 192.168.206.136@tcp (at 192.168.206.136@tcp)
[ 5371.431451] Lustre: Skipped 10 previous similar messages
[ 5372.367626] LustreError: 83002:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5374.515923] LustreError: 83033:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5377.681803] LustreError: lustre-MDT0000-mdc-ffff8dc1c7bdb800: operation ldlm_enqueue to node 192.168.206.136@tcp failed: rc = -107
[ 5377.699307] LustreError: Skipped 1 previous similar message
[ 5377.727634] LustreError: lustre-MDT0000-mdc-ffff8dc1c7bdb800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5377.763709] LustreError: 83047:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8dc1c7bdb800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8dc1c5549d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5377.807524] LustreError: 83047:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5377.814741] LustreError: 83052:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5377.830858] LustreError: 83052:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5378.601607] LustreError: 83033:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5385.464651] LustreError: lustre-MDT0000-mdc-ffff8dc1c7bdb800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5385.968463] LustreError: 83089:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5389.134279] LustreError: lustre-MDT0000-mdc-ffff8dc1c7bdb800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5389.228521] LustreError: 83108:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5390.071517] LustreError: 83089:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5390.081329] LustreError: 83089:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8dc1c7bdb800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5391.539530] LustreError: 83126:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5391.549741] LustreError: 83126:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 12 previous similar messages
[ 5396.360321] LustreError: 83164:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5399.461958] LustreError: lustre-MDT0000-mdc-ffff8dc1c7bdb800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5399.515958] LustreError: 83178:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8dc1c7bdb800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8dc1c5549c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5399.552896] LustreError: 83178:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5400.448872] LustreError: 83164:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5407.393433] LustreError: lustre-MDT0000-mdc-ffff8dc1c7bdb800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5407.403877] LustreError: 83207:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8dc1c7bdb800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8dc1d07bba00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5407.414797] LustreError: 83207:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5407.827855] LustreError: 83220:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5410.864514] LustreError: lustre-MDT0000-mdc-ffff8dc1c7bdb800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5411.903484] LustreError: 83220:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5414.915964] LustreError: lustre-MDT0000-mdc-ffff8dc1c7bdb800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5415.031490] LustreError: 83267:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5415.038664] LustreError: 83267:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5416.112134] LustreError: 83248:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8dc1c7bdb800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5424.310556] LustreError: 83319:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5424.327511] LustreError: 83319:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5427.957577] LustreError: lustre-MDT0000-mdc-ffff8dc1c7bdb800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5428.044188] LustreError: 83342:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5428.050635] LustreError: 83342:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5428.391282] LustreError: 83319:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5428.397841] LustreError: 83319:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5428.406725] LustreError: 83319:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8dc1c7bdb800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5440.676197] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 11:38:33 (1787845113)
[ 5460.113346] LustreError: 70766:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5460.122361] LustreError: 70766:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5464.183134] LustreError: 70766:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5464.204544] LustreError: 70766:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5472.153676] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 11:39:04 (1787845144)
[ 5523.325542] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 11:39:55 (1787845195)
[ 5586.846764] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 11:40:59 (1787845259)
[ 5620.079257] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 11:41:32 (1787845292)
[ 5620.674339] Lustre: Mounted lustre-client - version 2.17.57_95_g5b39979
[ 5744.682453] Lustre: Unmounted lustre-client
[ 5752.303624] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 11:43:45 (1787845425)
[ 5752.911651] LustreError: 87648:0:(osc_request.c:3192:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5755.508615] LustreError: lustre-OST0000-osc-ffff8dc1c7bdb800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5755.519255] LustreError: 87730:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8dc1c7bdb800: namespace resource [0x240000400:0x3f6d:0x0].0x0 (ffff8dc1c54c8100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5755.531772] LustreError: 87730:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 3 previous similar messages
[ 5755.855293] LustreError: 87648:0:(osc_request.c:3192:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5755.865489] Lustre: 2359:0:(llite_lib.c:4352:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.136@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5762.168427] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 11:43:55 (1787845435)
[ 5819.360350] Lustre: 88314:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787845438/real 1787845438]  req@ffff8dc1d2f32940 x1874685851982592/t0(0) o101->lustre-MDT0000-mdc-ffff8dc1c7bdb800@192.168.206.136@tcp:12/10 lens 328/344 e 0 to 1 dl 1787845493 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 5819.402017] Lustre: 88314:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
[ 5825.882725] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 11:44:58 (1787845498)
[ 5827.242325] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5828.923102] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 11:45:01 (1787845501)
[ 5891.578320] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5893.334398] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 11:46:06 (1787845566)
[ 5949.407548] Lustre: lustre-MDT0000-mdc-ffff8dc1c7bdb800: Connection to lustre-MDT0000 (at 192.168.206.136@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5949.422085] Lustre: Skipped 16 previous similar messages
[ 5951.395319] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 11:47:03 (1787845623)
[ 5953.198309] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5954.733992] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 11:47:07 (1787845627)
[ 5956.436600] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5957.930941] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 11:47:10 (1787845630)
[ 5988.029179] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 11:47:41 (1787845661)
[ 5998.817861] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6026.218092] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 192.168.206.136@tcp) was lost; in progress operations using this service will fail
[ 6026.234149] LustreError: Skipped 2 previous similar messages
[ 6026.257500] Lustre: Evicted from MGS (at 192.168.206.136@tcp) after server handle changed from 0x3314888873ba5868 to 0x3314888873ba70a2
[ 6026.263826] Lustre: Skipped 2 previous similar messages
[ 6026.270518] Lustre: MGC192.168.206.136@tcp: Connection restored to 192.168.206.136@tcp (at 192.168.206.136@tcp)
[ 6026.277387] Lustre: Skipped 14 previous similar messages
[ 6027.764533] Lustre: 68252:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.136@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6047.986908] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6050.062884] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6063.502980] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 11:48:55 (1787845735)
[ 6066.658317] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6068.366485] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 11:49:01 (1787845741)
[ 6082.551450] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 192.168.206.136@tcp) was lost; in progress operations using this service will fail
[ 6082.590411] Lustre: Evicted from MGS (at 192.168.206.136@tcp) after server handle changed from 0x3314888873ba70a2 to 0x3314888873ba755d
[ 6097.942624] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 11:49:30 (1787845770)
[ 6148.902362] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 11:50:21 (1787845821)
[ 6156.104276] LustreError: lustre-OST0000-osc-ffff8dc1c7bdb800: operation ost_setattr to node 192.168.206.136@tcp failed: rc = -107
[ 6156.111290] LustreError: Skipped 8 previous similar messages
[ 6209.245691] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6213.206910] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6299.615619] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 192.168.206.136@tcp) was lost; in progress operations using this service will fail
[ 6299.652019] LustreError: Skipped 1 previous similar message
[ 6308.869286] Lustre: Evicted from MGS (at 192.168.206.136@tcp) after server handle changed from 0x3314888873ba78e4 to 0x3314888873ba85b7
[ 6308.878468] Lustre: Skipped 1 previous similar message
[ 6325.330690] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6326.940689] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6367.922367] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6369.944514] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6513.445874] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 11:56:26 (1787846186)
[ 6576.878656] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6580.711850] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6835.914784] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 12:01:48 (1787846508)
[ 7035.882674] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 192.168.206.136@tcp) was lost; in progress operations using this service will fail
[ 7035.891791] LustreError: Skipped 1 previous similar message
[ 7035.897281] Lustre: lustre-MDT0000-mdc-ffff8dc1c7bdb800: Connection to lustre-MDT0000 (at 192.168.206.136@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7035.904970] Lustre: Skipped 7 previous similar messages
[ 7035.928587] Lustre: Evicted from MGS (at 192.168.206.136@tcp) after server handle changed from 0x3314888873ba8d19 to 0x3314888873c9da79
[ 7035.936557] Lustre: Skipped 1 previous similar message
[ 7035.955696] Lustre: MGC192.168.206.136@tcp: Connection restored to 192.168.206.136@tcp (at 192.168.206.136@tcp)
[ 7035.968741] Lustre: Skipped 9 previous similar messages
[ 7066.536716] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846699/real 1787846699]  req@ffff8dc1dec6d2c0 x1874685869292416/t0(0) o400->lustre-MDT0000-mdc-ffff8dc1c7bdb800@192.168.206.136@tcp:12/10 lens 224/224 e 0 to 1 dl 1787846740 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7066.598991] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 29 previous similar messages
[ 7069.562428] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 12:05:42 (1787846742)
[ 7071.104762] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7072.648231] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 12:05:45 (1787846745)
[ 7074.689742] LustreError: lustre-MDT0000-mdc-ffff8dc1c7bdb800: operation ldlm_enqueue to node 192.168.206.136@tcp failed: rc = -107
[ 7074.697444] LustreError: Skipped 113 previous similar messages
[ 7074.714769] LustreError: lustre-MDT0000-mdc-ffff8dc1c7bdb800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7083.113527] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 12:05:55 (1787846755)
[ 7249.896830] LustreError: lustre-OST0000-osc-ffff8dc1c7bdb800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7252.196559] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 12:08:44 (1787846924)
[ 7278.049428] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846932/real 1787846932]  req@ffff8dc1ede97840 x1874685869313536/t0(0) o4->lustre-OST0000-osc-ffff8dc1c7bdb800@192.168.206.136@tcp:6/4 lens 4584/448 e 0 to 1 dl 1787846952 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7278.085101] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 7297.063309] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 12:09:29 (1787846969)
[ 7298.899845] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7300.613592] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 12:09:33 (1787846973)
[ 7302.223692] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7303.741198] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 12:09:36 (1787846976)
[ 7341.127289] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 12:10:13 (1787847013)
[ 7385.572624] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 192.168.206.136@tcp) was lost; in progress operations using this service will fail
[ 7385.591925] Lustre: Evicted from MGS (at 192.168.206.136@tcp) after server handle changed from 0x3314888873c9da79 to 0x3314888873ca06a9
[ 7385.598654] Lustre: Skipped 1 previous similar message
[ 7397.273627] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 12:11:10 (1787847070)
[ 7399.136694] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7401.394072] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 12:11:13 (1787847073)
[ 7403.282907] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7405.589981] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 12:11:17 (1787847077)
[ 7416.814441] Lustre: Unmounted lustre-client
[ 7418.920155] Lustre: Mounted lustre-client - version 2.17.57_95_g5b39979
[ 7421.464912] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7440.223855] Lustre: 2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787847098/real 1787847098]  req@ffff8dc1ede961c0 x1874685869501056/t0(0) o400->MGC192.168.206.136@tcp@192.168.206.136@tcp:26/25 lens 224/224 e 0 to 1 dl 1787847114 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7440.283524] Lustre: 2357:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 7452.420364] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2