[ 1669.500345] LustreError: lustre-OST0000-osc-ffff8e234353a000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1669.511692] Lustre: 2360:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.130@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1673.117992] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 05:20:33 (1786353633)
[ 1674.220837] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1675.106263] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 05:20:35 (1786353635)
[ 1676.013938] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1676.880523] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 05:20:36 (1786353636)
[ 1695.717404] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 192.168.201.130@tcp) was lost; in progress operations using this service will fail
[ 1695.737767] Lustre: Evicted from MGS (at 192.168.201.130@tcp) after server handle changed from 0x3a95f9991159a034 to 0x3a95f9991159bdae
[ 1790.875789] LustreError: lustre-MDT0000-mdc-ffff8e234353a000: operation ldlm_enqueue to node 192.168.201.130@tcp failed: rc = -19
[ 1790.880052] LustreError: Skipped 4 previous similar messages
[ 1807.337494] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 192.168.201.130@tcp) was lost; in progress operations using this service will fail
[ 1807.358872] Lustre: Evicted from MGS (at 192.168.201.130@tcp) after server handle changed from 0x3a95f9991159bdae to 0x3a95f999115d81ee
[ 1811.318807] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 05:22:51 (1786353771)
[ 1811.530840] Lustre: *** cfs_fail_loc=305, val=0***
[ 1811.532928] Lustre: Skipped 2 previous similar messages
[ 1848.287590] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 192.168.201.130@tcp) was lost; in progress operations using this service will fail
[ 1848.301081] Lustre: Evicted from MGS (at 192.168.201.130@tcp) after server handle changed from 0x3a95f999115d81ee to 0x3a95f999115d94da
[ 1853.231405] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 1854.105305] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1858.946526] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 05:23:39 (1786353819)
[ 1868.778980] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 192.168.201.130@tcp) was lost; in progress operations using this service will fail
[ 1868.794277] Lustre: Evicted from MGS (at 192.168.201.130@tcp) after server handle changed from 0x3a95f999115d94da to 0x3a95f999115d99db
[ 1868.808490] LustreError: lustre-MDT0000-mdc-ffff8e234353a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 1879.468690] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 05:23:59 (1786353839)
[ 1887.535942] LustreError: lustre-OST0000-osc-ffff8e234353a000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1896.271192] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 05:24:16 (1786353856)
[ 1907.735332] LustreError: lustre-MDT0000-mdc-ffff8e234353a000: operation ldlm_enqueue to node 192.168.201.130@tcp failed: rc = -19
[ 1925.091220] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 192.168.201.130@tcp) was lost; in progress operations using this service will fail
[ 1925.106423] Lustre: Evicted from MGS (at 192.168.201.130@tcp) after server handle changed from 0x3a95f999115d99db to 0x3a95f999115e17cd
[ 1928.927783] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 1929.751124] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1993.067804] LustreError: lustre-MDT0000-mdc-ffff8e234353a000: operation mds_reint to node 192.168.201.130@tcp failed: rc = -19
[ 1993.073821] LustreError: Skipped 1 previous similar message
[ 2012.136780] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 192.168.201.130@tcp) was lost; in progress operations using this service will fail
[ 2012.145612] Lustre: Evicted from MGS (at 192.168.201.130@tcp) after server handle changed from 0x3a95f999115e17cd to 0x3a95f9991160f0b4
[ 2015.511706] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2016.580531] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2094.050918] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 192.168.201.130@tcp) was lost; in progress operations using this service will fail
[ 2094.064390] Lustre: Evicted from MGS (at 192.168.201.130@tcp) after server handle changed from 0x3a95f9991160f0b4 to 0x3a95f9991163ed5f
[ 2102.561754] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2103.728828] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2128.875472] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 05:28:08 (1786354088)
[ 2131.365526] LustreError: lustre-MDT0000-mdc-ffff8e234353a000: operation mds_reint to node 192.168.201.130@tcp failed: rc = -19
[ 2131.368561] LustreError: Skipped 2 previous similar messages
[ 2145.263519] Lustre: MGC192.168.201.130@tcp: Connection restored to 192.168.201.130@tcp (at 192.168.201.130@tcp)
[ 2145.267298] Lustre: Skipped 22 previous similar messages
[ 2148.757232] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2150.760652] Lustre: lustre-MDT0000-mdc-ffff8e234353a000: Connection to lustre-MDT0000 (at 192.168.201.130@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2150.770204] Lustre: Skipped 16 previous similar messages
[ 2169.404827] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2169.827704] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 192.168.201.130@tcp) was lost; in progress operations using this service will fail
[ 2169.834087] LustreError: Skipped 1 previous similar message
[ 2169.851878] Lustre: Evicted from MGS (at 192.168.201.130@tcp) after server handle changed from 0x3a95f99911652667 to 0x3a95f99911653d58
[ 2169.856692] Lustre: Skipped 1 previous similar message
[ 2194.794362] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2224.303074] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2262.625711] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2305.992534] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2307.044936] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 192.168.201.130@tcp) was lost; in progress operations using this service will fail
[ 2307.049991] LustreError: Skipped 3 previous similar messages
[ 2307.060731] Lustre: Evicted from MGS (at 192.168.201.130@tcp) after server handle changed from 0x3a95f9991166fd0b to 0x3a95f999116800eb
[ 2307.064364] Lustre: Skipped 3 previous similar messages
[ 2376.883335] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 05:32:17 (1786354337)
[ 2388.372697] LustreError: lustre-OST0000-osc-ffff8e234353a000: operation ldlm_enqueue to node 192.168.201.130@tcp failed: rc = -107
[ 2388.379972] LustreError: Skipped 9 previous similar messages
[ 2412.629914] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2413.397841] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2740.779184] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2741.483158] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3049.921442] LustreError: lustre-OST0000-osc-ffff8e234353a000: operation ldlm_enqueue to node 192.168.201.130@tcp failed: rc = -107
[ 3049.925915] LustreError: Skipped 1 previous similar message
[ 3049.927466] Lustre: lustre-OST0000-osc-ffff8e234353a000: Connection to lustre-OST0000 (at 192.168.201.130@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3049.935178] Lustre: Skipped 7 previous similar messages
[ 3072.032739] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3072.817642] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3342.095746] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 05:48:22 (1786355302)
[ 3359.199233] Lustre: 50622:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786355303/real 1786355303]  req@ffff8e2351229c00 x1873125943094912/t0(0) o101->lustre-MDT0000-mdc-ffff8e234353a000@192.168.201.130@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786355319 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3359.210055] Lustre: 50622:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 3359.225995] Lustre: lustre-MDT0000-mdc-ffff8e234353a000: Connection restored to 192.168.201.130@tcp (at 192.168.201.130@tcp)
[ 3359.228888] Lustre: Skipped 14 previous similar messages
[ 3362.103908] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 05:48:42 (1786355322)
[ 3382.065624] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 05:49:02 (1786355342)
[ 3402.356566] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 05:49:22 (1786355362)
[ 3402.497404] Lustre: Mounted lustre-client - version 2.17.56_50_g4393166
[ 3428.324291] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 192.168.201.130@tcp) was lost; in progress operations using this service will fail
[ 3428.329190] LustreError: Skipped 1 previous similar message
[ 3428.337538] Lustre: Evicted from MGS (at 192.168.201.130@tcp) after server handle changed from 0x3a95f999116988c8 to 0x3a95f999119c3a82
[ 3428.342555] Lustre: Skipped 1 previous similar message
[ 3432.801621] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3433.384773] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3434.767220] Lustre: Unmounted lustre-client
[ 3437.133383] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 05:49:57 (1786355397)
[ 3440.287182] Lustre: 2359:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786355383/real 1786355383]  req@ffff8e23444af100 x1873125943112064/t0(0) o400->lustre-MDT0000-mdc-ffff8e234353a000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 1 dl 1786355399 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3440.297623] Lustre: 2359:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 3472.356035] Lustre: lustre-OST0000-osc-ffff8e234353a000: Connection restored to 192.168.201.130@tcp (at 192.168.201.130@tcp)
[ 3472.359384] Lustre: Skipped 5 previous similar messages
[ 3601.375144] Lustre: 2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786355546/real 1786355546]  req@ffff8e2361779c00 x1873125943141376/t0(0) o4->lustre-OST0000-osc-ffff8e234353a000@192.168.201.130@tcp:6/4 lens 488/448 e 0 to 1 dl 1786355562 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 3601.385295] Lustre: 2358:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 84 previous similar messages
[ 3634.148149] Lustre: lustre-OST0000-osc-ffff8e234353a000: Connection restored to 192.168.201.130@tcp (at 192.168.201.130@tcp)
[ 3634.151530] Lustre: Skipped 9 previous similar messages
[ 3665.887285] Lustre: lustre-OST0000-osc-ffff8e234353a000: Connection to lustre-OST0000 (at 192.168.201.130@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3665.892624] Lustre: Skipped 18 previous similar messages
[ 3718.428834] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 05:54:38 (1786355678)
[ 3761.189024] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 05:55:21 (1786355721)
[ 3762.516778] Lustre: Unmounted lustre-client
[ 3770.878778] Lustre: Mounted lustre-client - version 2.17.56_50_g4393166
[ 3772.979840] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 05:55:33 (1786355733)
[ 3773.055493] LustreError: 56528:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 3774.119566] Lustre: *** cfs_fail_loc=305, val=0***
[ 3793.127130] LustreError: 56528:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 awake
[ 3795.432825] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 05:55:55 (1786355755)
[ 3795.575581] Lustre: Mounted lustre-client - version 2.17.56_50_g4393166
[ 3795.950583] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 3806.195190] Lustre: Unmounted lustre-client
[ 3808.359434] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 05:56:08 (1786355768)
[ 3830.340272] LustreError: lustre-MDT0000-mdc-ffff8e2349bc9000: operation mds_reint to node 192.168.201.130@tcp failed: rc = -19
[ 3846.625237] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 192.168.201.130@tcp) was lost; in progress operations using this service will fail
[ 3846.634111] Lustre: Evicted from MGS (at 192.168.201.130@tcp) after server handle changed from 0x3a95f999119c53f0 to 0x3a95f999119f48ad
[ 3884.471306] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 05:57:24 (1786355844)
[ 3886.114221] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 3892.723328] LustreError: lustre-MDT0000-mdc-ffff8e2349bc9000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3896.659710] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 05:57:36 (1786355856)
[ 3896.773262] Lustre: Mounted lustre-client - version 2.17.56_50_g4393166
[ 3901.925141] LustreError: lustre-OST0000-osc-ffff8e2349bc9000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 3901.928899] LustreError: 60207:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8e2349bc9000: namespace resource [0x240000400:0x7b83:0x0].0x0 (ffff8e2346a61700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 3913.183165] Lustre: 2357:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786355857/real 1786355857]  req@ffff8e23446bbb80 x1873125947725056/t0(0) o103->lustre-OST0000-osc-ffff8e2347df0000@192.168.201.130@tcp:17/18 lens 328/224 e 0 to 1 dl 1786355873 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 3913.193035] Lustre: 2357:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 66 previous similar messages
[ 3913.524234] Lustre: Unmounted lustre-client
[ 3915.648480] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 05:57:55 (1786355875)
[ 3920.441785] LustreError: lustre-MDT0000-mdc-ffff8e2349bc9000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3920.446527] LustreError: 60877:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 3922.875592] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 05:58:03 (1786355883)
[ 3922.936953] LustreError: 61577:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 3926.268475] LustreError: lustre-MDT0000-mdc-ffff8e2349bc9000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3926.273441] LustreError: 61595:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 3926.276521] LustreError: 61577:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1663
[ 3926.279934] LustreError: 61577:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 3926.292736] LustreError: 61600:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3927.322390] LustreError: 61606:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3927.325017] LustreError: 61606:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 3929.412465] LustreError: 61629:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3929.415582] LustreError: 61629:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 3933.590586] LustreError: 61674:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3933.593745] LustreError: 61674:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 3936.737499] Lustre: lustre-MDT0000-mdc-ffff8e2349bc9000: Connection restored to 192.168.201.130@tcp (at 192.168.201.130@tcp)
[ 3936.740535] Lustre: Skipped 15 previous similar messages
[ 3940.109325] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 05:58:20 (1786355900)
[ 3961.342831] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3961.815404] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3965.976155] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 05:58:46 (1786355926)
[ 3985.361543] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3985.890779] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3990.082813] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 05:59:10 (1786355950)
[ 4035.658127] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4036.096554] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4039.379292] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 05:59:59 (1786355999)
[ 4058.588552] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4059.093492] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4061.355093] Lustre: Unmounted lustre-client
[ 4085.500498] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4086.978687] Lustre: Mounted lustre-client - version 2.17.56_50_g4393166
[ 4089.177801] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 06:00:49 (1786356049)
[ 4090.038376] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4090.639824] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 06:00:50 (1786356050)
[ 4101.093871] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 06:01:01 (1786356061)
[ 4101.616123] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4102.232550] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 06:01:02 (1786356062)
[ 4102.698492] Lustre: *** cfs_fail_loc=805, val=0***
[ 4104.561934] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4120.125570] LustreError: 2356:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8e23485ec800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8e2344442f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4120.132140] LustreError: 2356:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4120.459557] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 192.168.201.130@tcp) was lost; in progress operations using this service will fail
[ 4120.463740] LustreError: Skipped 1 previous similar message
[ 4120.467613] Lustre: Evicted from MGS (at 192.168.201.130@tcp) after server handle changed from 0x3a95f99911a3ab7e to 0x3a95f99911a3b16d
[ 4120.471132] Lustre: Skipped 1 previous similar message
[ 4120.472659] Lustre: Unmounted lustre-client
[ 4122.875347] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 06:01:23 (1786356083)
[ 4138.476789] LustreError: 2356:0:(client.c:3435:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8e23446b8000 x1873125947790592/t94489280515(94489280515) o101->lustre-MDT0000-mdc-ffff8e2349bde800@192.168.201.130@tcp:12/10 lens 664/608 e 0 to 0 dl 1786356115 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 4142.206100] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4142.739354] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4146.215836] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 06:01:46 (1786356106)
[ 4146.715384] LustreError: lustre-OST0000-osc-ffff8e2349bde800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4146.720119] Lustre: 2360:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.130@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 4146.754976] LustreError: 72774:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8e2349bde800: namespace resource [0x240000400:0x7be2:0x0].0x0 (ffff8e2346922500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4146.761749] LustreError: 72774:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4150.024425] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 06:01:50 (1786356110)
[ 4150.510939] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 4151.043491] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 06:01:51 (1786356111)
[ 4151.511450] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 4152.007393] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 06:01:52 (1786356112)
[ 4152.446626] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 4152.944199] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 06:01:53 (1786356113)
[ 4153.407750] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 4153.895669] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 06:01:54 (1786356114)
[ 4154.345284] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 4154.874915] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 06:01:55 (1786356115)
[ 4155.406772] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 4155.926713] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 06:01:56 (1786356116)
[ 4156.465369] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 4157.016191] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 06:01:57 (1786356117)
[ 4157.514964] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 4158.053933] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 06:01:58 (1786356118)
[ 4158.535698] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 4159.093132] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 06:01:59 (1786356119)
[ 4159.612874] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 4160.188514] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 06:02:00 (1786356120)
[ 4160.677134] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 4161.238681] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 06:02:01 (1786356121)
[ 4162.107483] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 4162.694633] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 06:02:02 (1786356122)
[ 4172.902100] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 06:02:13 (1786356133)
[ 4196.013546] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 06:02:36 (1786356156)
[ 4196.117358] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4202.181174] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 06:02:42 (1786356162)
[ 4202.265104] Lustre: *** cfs_fail_loc=51b, val=4***
[ 4208.330178] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 06:02:48 (1786356168)
[ 4208.420551] Lustre: *** cfs_fail_loc=50f, val=3***
[ 4211.861736] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 06:02:52 (1786356172)
[ 4211.930617] Lustre: *** cfs_fail_loc=50f, val=4***
[ 4215.505701] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 06:02:55 (1786356175)
[ 4215.579633] Lustre: *** cfs_fail_loc=510, val=3***
[ 4218.966033] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 06:02:59 (1786356179)
[ 4219.054852] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4278.751221] Lustre: lustre-OST0000-osc-ffff8e2349bde800: Connection to lustre-OST0000 (at 192.168.201.130@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4278.755922] Lustre: Skipped 23 previous similar messages
[ 4280.925877] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 06:04:01 (1786356241)
[ 4281.027117] Lustre: *** cfs_fail_loc=51c, val=3***
[ 4343.406911] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 06:05:03 (1786356303)
[ 4343.505543] LustreError: 82971:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4345.817504] LustreError: lustre-MDT0000-mdc-ffff8e2349bde800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4345.823694] LustreError: 82986:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8e2349bde800: inode [0x20000a042:0x8:0x0] mdc close failed: rc = -108
[ 4345.832873] LustreError: 82986:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8e2349bde800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8e23608b2b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4345.839401] LustreError: 82991:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4345.842315] LustreError: 82991:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 23 previous similar messages
[ 4347.567139] LustreError: 82971:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4349.604990] LustreError: 83002:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4351.954798] LustreError: lustre-MDT0000-mdc-ffff8e2349bde800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4351.960716] LustreError: 83017:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8e2349bde800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8e2346a61a00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4351.966679] LustreError: 83017:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4353.663096] LustreError: 83002:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4359.968649] LustreError: lustre-MDT0000-mdc-ffff8e2349bde800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4360.102358] LustreError: 83058:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4362.462351] LustreError: lustre-MDT0000-mdc-ffff8e2349bde800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4362.482746] LustreError: 83077:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4364.159075] LustreError: 83058:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4364.162155] LustreError: 83058:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8e2349bde800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4364.166565] LustreError: 83058:0:(file.c:251:ll_close_inode_openhandle()) Skipped 4 previous similar messages
[ 4368.790934] LustreError: 83132:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4371.067781] LustreError: lustre-MDT0000-mdc-ffff8e2349bde800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4371.072831] LustreError: 83147:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8e2349bde800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8e2346a61f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4371.078530] LustreError: 83147:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4372.847115] LustreError: 83132:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4379.111978] LustreError: lustre-MDT0000-mdc-ffff8e2349bde800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4379.231874] LustreError: 83189:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 4381.509741] LustreError: lustre-MDT0000-mdc-ffff8e2349bde800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4383.287154] LustreError: 83189:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 4385.613458] LustreError: lustre-MDT0000-mdc-ffff8e2349bde800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4387.375697] LustreError: 83217:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8e2349bde800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4396.448168] LustreError: lustre-MDT0000-mdc-ffff8e2349bde800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4396.453299] LustreError: 83306:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4396.456275] LustreError: 83306:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 2 previous similar messages
[ 4396.472246] LustreError: 83311:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4396.475485] LustreError: 83311:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 53 previous similar messages
[ 4397.975572] LustreError: 83289:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8e2349bde800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4400.519138] LustreError: 83306:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4400.521673] LustreError: 83306:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 2 previous similar messages
[ 4402.732219] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 06:06:03 (1786356363)
[ 4425.651172] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 06:06:25 (1786356385)
[ 4470.897185] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 06:07:11 (1786356431)
[ 4528.095131] Lustre: 85465:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786356432/real 1786356432]  req@ffff8e23446b8000 x1873125947908096/t0(0) o101->lustre-MDT0000-mdc-ffff8e2349bde800@192.168.201.130@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786356488 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'stat.0' uid:0 gid:0 projid:0
[ 4528.102360] Lustre: 85465:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
[ 4530.223120] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 06:08:10 (1786356490)
[ 4556.483702] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 06:08:36 (1786356516)
[ 4556.672716] Lustre: Mounted lustre-client - version 2.17.56_50_g4393166
[ 4556.676900] Lustre: Skipped 1 previous similar message
[ 4678.298209] Lustre: Unmounted lustre-client
[ 4682.404828] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 06:10:42 (1786356642)
[ 4682.842501] LustreError: 87608:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 4682.852357] LustreError: 87608:0:(osc_request.c:2989:osc_build_rpc()) Skipped 1 previous similar message
[ 4684.318281] LustreError: lustre-OST0000-osc-ffff8e2349bde800: operation ldlm_enqueue to node 192.168.201.130@tcp failed: rc = -107
[ 4684.323243] LustreError: Skipped 14 previous similar messages
[ 4684.338203] LustreError: lustre-OST0000-osc-ffff8e2349bde800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4684.346913] LustreError: 87690:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8e2349bde800: namespace resource [0x240000400:0x7c0d:0x0].0x0 (ffff8e2346a61500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4684.363214] LustreError: 87690:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 3 previous similar messages
[ 4684.535786] LustreError: 87608:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 4684.543168] Lustre: 2359:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.130@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 4688.134892] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 06:10:48 (1786356648)
[ 4746.213937] Lustre: lustre-MDT0000-mdc-ffff8e2349bde800: Connection restored to 192.168.201.130@tcp (at 192.168.201.130@tcp)
[ 4746.217599] Lustre: Skipped 25 previous similar messages
[ 4749.860589] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 06:11:49 (1786356709)
[ 4750.563994] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 4751.570836] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 06:11:51 (1786356711)
[ 4812.625649] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 4813.518932] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 06:12:53 (1786356773)
[ 4871.935083] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 06:13:51 (1786356831)
[ 4873.162507] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 4874.108635] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 06:13:54 (1786356834)
[ 4874.884091] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 4875.930368] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 06:13:55 (1786356835)
[ 4893.102035] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 06:14:13 (1786356853)
[ 4899.477628] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4922.335237] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 192.168.201.130@tcp) was lost; in progress operations using this service will fail
[ 4922.342687] LustreError: Skipped 2 previous similar messages
[ 4922.355280] Lustre: lustre-MDT0000-mdc-ffff8e2349bde800: Connection to lustre-MDT0000 (at 192.168.201.130@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4922.356093] Lustre: Evicted from MGS (at 192.168.201.130@tcp) after server handle changed from 0x3a95f99911a3b993 to 0x3a95f99911a3d3a9
[ 4922.362524] Lustre: Skipped 17 previous similar messages
[ 4922.368656] Lustre: Skipped 2 previous similar messages
[ 4927.241915] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4928.167675] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4934.188562] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 06:14:54 (1786356894)
[ 4935.810792] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 4936.800663] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 06:14:56 (1786356896)
[ 4947.956132] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 192.168.201.130@tcp) was lost; in progress operations using this service will fail
[ 4947.972272] Lustre: Evicted from MGS (at 192.168.201.130@tcp) after server handle changed from 0x3a95f99911a3d3a9 to 0x3a95f99911a3d6e3
[ 4951.534778] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 06:15:11 (1786356911)
[ 4985.620931] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 06:15:44 (1786356944)
[ 5055.271544] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5061.228301] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5148.132126] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 192.168.201.130@tcp) was lost; in progress operations using this service will fail
[ 5148.143893] LustreError: Skipped 1 previous similar message
[ 5148.171368] Lustre: Evicted from MGS (at 192.168.201.130@tcp) after server handle changed from 0x3a95f99911a3da6a to 0x3a95f99911a3e8b7
[ 5148.181399] Lustre: Skipped 1 previous similar message
[ 5158.146490] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5159.514894] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5196.257093] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5197.436123] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5316.293337] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 06:21:15 (1786357275)
[ 5326.929466] LustreError: lustre-OST0000-osc-ffff8e2349bde800: operation ost_setattr to node 192.168.201.130@tcp failed: rc = -19
[ 5326.942770] LustreError: Skipped 51 previous similar messages
[ 5373.728636] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5377.746403] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5604.774751] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 06:26:04 (1786357564)
[ 5725.168641] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 192.168.201.130@tcp) was lost; in progress operations using this service will fail
[ 5725.179474] LustreError: Skipped 1 previous similar message
[ 5725.181429] Lustre: lustre-MDT0000-mdc-ffff8e2349bde800: Connection to lustre-MDT0000 (at 192.168.201.130@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5725.188595] Lustre: Evicted from MGS (at 192.168.201.130@tcp) after server handle changed from 0x3a95f99911a3ebe3 to 0x3a95f99911b38dd6
[ 5725.198810] Lustre: Skipped 6 previous similar messages
[ 5725.208799] Lustre: Skipped 1 previous similar message
[ 5725.228642] Lustre: MGC192.168.201.130@tcp: Connection restored to 192.168.201.130@tcp (at 192.168.201.130@tcp)
[ 5725.234312] Lustre: Skipped 12 previous similar messages
[ 5751.424791] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 06:28:31 (1786357711)
[ 5752.473498] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 5753.697155] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 06:28:33 (1786357713)
[ 5755.164922] LustreError: lustre-MDT0000-mdc-ffff8e2349bde800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5760.367377] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 06:28:40 (1786357720)
[ 5926.630872] Lustre: Evicted from lustre-OST0000_UUID (at 192.168.201.130@tcp) after server handle changed from 0x3a95f99911b39179 to 0x3a95f99911b39283
[ 5926.634989] LustreError: lustre-OST0000-osc-ffff8e2349bde800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5927.610943] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 06:31:27 (1786357887)
[ 5950.943124] Lustre: 2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786357891/real 1786357891]  req@ffff8e2350789880 x1873125965653632/t0(0) o4->lustre-OST0000-osc-ffff8e2349bde800@192.168.201.130@tcp:6/4 lens 4584/448 e 0 to 1 dl 1786357911 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 5950.954621] Lustre: 2358:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 5965.260036] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 06:32:05 (1786357925)
[ 5966.139341] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 5967.267696] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 06:32:07 (1786357927)
[ 5968.458701] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 5969.667361] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 06:32:09 (1786357929)
[ 5998.302946] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 06:32:38 (1786357958)
[ 6032.867201] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 192.168.201.130@tcp) was lost; in progress operations using this service will fail
[ 6038.956720] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 06:33:18 (1786357998)
[ 6039.697241] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 6040.631916] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 06:33:20 (1786358000)
[ 6041.517707] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 6042.694459] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 06:33:22 (1786358002)
[ 6050.020052] Lustre: Unmounted lustre-client
[ 6050.832819] Lustre: Mounted lustre-client - version 2.17.56_50_g4393166
[ 6052.302825] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6071.264581] Lustre: 2359:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786358016/real 1786358016]  req@ffff8e2377905500 x1873125965838336/t0(0) o400->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 224/224 e 0 to 1 dl 1786358032 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6074.031293] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2