[ 1653.884507] Lustre: *** cfs_fail_loc=513, val=601***
[ 1654.008864] LustreError: 6496:0:(service.c:2341:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1876834179698432
[ 1654.751801] Lustre: *** cfs_fail_loc=513, val=601***
[ 1654.753163] Lustre: Skipped 28 previous similar messages
[ 1655.775468] Lustre: *** cfs_fail_loc=513, val=601***
[ 1655.776855] Lustre: Skipped 4 previous similar messages
[ 1659.024884] Lustre: *** cfs_fail_loc=513, val=601***
[ 1659.027450] Lustre: Skipped 5 previous similar messages
[ 1664.144700] Lustre: *** cfs_fail_loc=513, val=601***
[ 1664.146593] Lustre: Skipped 22 previous similar messages
[ 1668.575238] Lustre: 43632:0:(service.c:1612:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@ffff8fe9ef37fb80 x1876834173749632/t0(0) o4->419c915b-7832-4215-a53a-5e9f02d6a2fe@192.168.203.38@tcp:319/0 lens 488/448 e 1 to 0 dl 1789890144 ref 2 fl Interpret:/600/0 rc 0/0 job:'dd.60000' uid:60000 gid:60000 projid:0
[ 1669.599153] Lustre: 21184:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789890124/real 1789890124]  req@ffff8fe9f8130a80 x1876834179698432/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1789890140 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'ll_ost_io00_004.0' uid:0 gid:0 projid:4294967295
[ 1669.609026] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1669.615840] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0000_UUID (at 0@lo) reconnecting
[ 1669.618720] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 1669.640074] LustreError: 6497:0:(service.c:2341:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1876834179707520
[ 1672.159450] Lustre: *** cfs_fail_loc=513, val=601***
[ 1672.161625] Lustre: Skipped 45 previous similar messages
[ 1685.983441] Lustre: 3627:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789890140/real 1789890140]  req@ffff8fe8c586e300 x1876834179707520/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1789890156 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'ll_ost_io00_001.0' uid:0 gid:0 projid:4294967295
[ 1685.990839] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1685.996223] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0000_UUID (at 0@lo) reconnecting
[ 1685.998888] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 1689.040491] Lustre: *** cfs_fail_loc=513, val=601***
[ 1689.042935] Lustre: Skipped 80 previous similar messages
[ 1689.049814] LustreError: 16538:0:(service.c:2341:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1876834179717248
[ 1704.415138] Lustre: 21184:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789890159/real 1789890159]  req@ffff8fe8ce99e300 x1876834179717248/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1789890175 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'ll_ost_io00_004.0' uid:0 gid:0 projid:4294967295
[ 1704.426103] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1704.433695] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0000_UUID (at 0@lo) reconnecting
[ 1704.437213] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 1718.996291] Lustre: DEBUG MARKER: == sanity-quota test 7a: Quota reintegration (global index) ========================================================== 03:43:09 (1789890189)
[ 1728.554635] Lustre: Failing over lustre-OST0000
[ 1728.607285] Lustre: server umount lustre-OST0000 complete
[ 1730.017672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1730.021136] Lustre: Skipped 1 previous similar message
[ 1733.573360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 1733.658392] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 1733.668599] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect
[ 1734.995327] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect
[ 1735.014518] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted.
[ 1735.014592] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo)
[ 1735.459615] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1737.777147] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 1739.336361] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 1740.958288] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 1742.553976] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 1744.197781] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 1745.889196] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 1760.278731] Lustre: Failing over lustre-OST0000
[ 1760.309953] Lustre: server umount lustre-OST0000 complete
[ 1760.735921] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107
[ 1760.735959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1760.738114] LustreError: Skipped 1 previous similar message
[ 1760.743120] LustreError: 8405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1760.749670] LustreError: 8405:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages
[ 1761.431841] LustreError: 8406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1763.103806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 1763.208579] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 1763.217027] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect
[ 1764.385162] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect
[ 1764.696802] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo)
[ 1764.696906] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted.
[ 1764.699114] Lustre: Skipped 1 previous similar message
[ 1765.127439] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1767.776439] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 1769.457183] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 1771.116387] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 1772.832361] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 1774.556609] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 1776.252134] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 1792.706511] Lustre: DEBUG MARKER: == sanity-quota test 7b: Quota reintegration (slave index) ========================================================== 03:44:22 (1789890262)
[ 1804.761367] Lustre: *** cfs_fail_loc=a02, val=0***
[ 1807.104360] LustreError: 3628:0:(qsd_handler.c:318:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0000 qtype:usr lqe: ffff8fe9e07301c8 id:60000 enforced:1 granted: 1024 pending:0 waiting:0 req:1 usage: 2048 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 1807.114072] Lustre: Failing over lustre-OST0000
[ 1807.153264] Lustre: server umount lustre-OST0000 complete
[ 1807.510506] LustreError: 43916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1809.377585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1809.381591] Lustre: Skipped 2 previous similar messages
[ 1809.885866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 1809.990668] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 1809.997973] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect
[ 1811.360957] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect
[ 1811.515514] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo)
[ 1811.515677] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted.
[ 1811.517613] Lustre: Skipped 1 previous similar message
[ 1811.849840] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1814.344443] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 1816.029482] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 1817.683450] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 1819.391505] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 1821.097565] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 1822.823120] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 1839.807096] Lustre: DEBUG MARKER: == sanity-quota test 7c: Quota reintegration (restart mds during reintegration) ========================================================== 03:45:09 (1789890309)
[ 1850.447755] LustreError: 103277:0:(qsd_reint.c:488:qsd_reint_main()) cfs_fail_timeout id a03 sleeping for 10000ms
[ 1850.452637] LustreError: 103277:0:(qsd_reint.c:488:qsd_reint_main()) Skipped 5 previous similar messages
[ 1851.060502] Lustre: Failing over lustre-MDT0000
[ 1851.598365] Lustre: server umount lustre-MDT0000 complete
[ 1852.743075] LustreError: 103280:0:(qsd_reint.c:488:qsd_reint_main()) cfs_fail_timeout interrupted
[ 1852.745506] LustreError: lustre-MDT0000-lwp-OST0001: operation dt_index_read to node 0@lo failed: rc = -107
[ 1852.745988] LustreError: 103280:0:(qsd_reint.c:488:qsd_reint_main()) Skipped 5 previous similar messages
[ 1852.746234] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1852.746674] LustreError: 11215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1852.746687] LustreError: 11215:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages
[ 1852.750286] LustreError: Skipped 4 previous similar messages
[ 1854.840764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1854.883088] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1854.943595] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 1854.983875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1855.014381] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 1856.238900] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1858.566418] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 1858.705163] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 1860.066823] LustreError: 3624:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-lwp-OST0000: namespace resource [0x200000006:0x2020000:0x0].0x0 (ffff8fe9c39e5800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 1860.071308] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 1860.073160] Lustre: Skipped 1 previous similar message
[ 1860.090364] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted.
[ 1860.096132] Lustre: 103282:0:(qsd_reint.c:249:qsd_reint_index()) lustre-OST0001: index version for fid [0x200000005:0x100a:0x0] is 0, but index isn't empty (1)
[ 1860.112171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:149 to 0x280000401:193)
[ 1860.112190] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:115 to 0x2c0000401:161)
[ 1973.640461] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 1975.358237] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 1977.018116] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 1978.721552] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 1980.429520] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 1993.054757] Lustre: DEBUG MARKER: == sanity-quota test 7d: Quota reintegration (Transfer index in multiple bulks) ========================================================== 03:47:43 (1789890463)
[ 2003.005943] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 2004.674454] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 2022.908660] Lustre: DEBUG MARKER: == sanity-quota test 7e: Quota reintegration (inode limits) ========================================================== 03:48:12 (1789890492)
[ 2033.064304] Lustre: Failing over lustre-MDT0001
[ 2033.159283] Lustre: server umount lustre-MDT0001 complete
[ 2034.146070] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2034.146760] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107
[ 2034.147230] LustreError: 6493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2034.147237] LustreError: 6493:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages
[ 2034.151915] Lustre: Skipped 4 previous similar messages
[ 2038.571039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2038.740845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 2038.776667] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect
[ 2039.891749] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 2040.120688] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2042.728384] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 2043.875897] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 2043.879236] Lustre: Skipped 3 previous similar messages
[ 2043.886450] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 2043.906972] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33)
[ 2044.460353] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475
[ 2046.176717] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 2047.821371] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475
[ 2049.467142] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 2050.990388] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475
[ 2071.287651] Lustre: Failing over lustre-MDT0001
[ 2071.447897] Lustre: server umount lustre-MDT0001 complete
[ 2074.291465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2074.470688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 2074.491324] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect
[ 2075.800777] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 2075.862122] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2078.383757] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 2079.722700] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 2079.736975] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:65)
[ 2079.961832] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475
[ 2081.413656] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 2082.873129] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475
[ 2084.344658] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 2085.902656] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475
[ 2108.148524] Lustre: DEBUG MARKER: == sanity-quota test 7f: Quota reintegration automatically ========================================================== 03:49:38 (1789890578)
[ 2112.919661] Lustre: *** cfs_fail_loc=a11, val=0***
[ 2112.921248] Lustre: Skipped 3 previous similar messages
[ 2118.252648] Lustre: *** cfs_fail_loc=a11, val=0***
[ 2145.737620] Lustre: DEBUG MARKER: == sanity-quota test 8: Run dbench with quota enabled ==== 03:50:15 (1789890615)
[ 2212.927064] Lustre: 124089:0:(qsd_reint.c:249:qsd_reint_index()) lustre-MDT0001: index version for fid [0x200000005:0x1008:0x0] is 0, but index isn't empty (1)
[ 2309.882766] Lustre: DEBUG MARKER: == sanity-quota test 9: Block limit larger than 4GB (b10707) ========================================================== 03:52:59 (1789890779)
[ 2310.403483] Lustre: DEBUG MARKER: OST0_SIZE: 3601232 required: 4900000
[ 2312.397260] Lustre: DEBUG MARKER: == sanity-quota test 10: Test quota for root user ======== 03:53:02 (1789890782)
[ 2338.270893] Lustre: DEBUG MARKER: == sanity-quota test 11: Chown/chgrp ignores quota ======= 03:53:27 (1789890807)
[ 2363.389470] Lustre: DEBUG MARKER: == sanity-quota test 12a: Block quota rebalancing ======== 03:53:53 (1789890833)
[ 2432.445962] Lustre: DEBUG MARKER: == sanity-quota test 12b: Inode quota rebalancing ======== 03:55:01 (1789890901)
[ 2539.752796] Lustre: DEBUG MARKER: == sanity-quota test 13: Cancel per-ID lock in the LRU list ========================================================== 03:56:49 (1789891009)
[ 2580.027160] Lustre: DEBUG MARKER: == sanity-quota test 14: check panic in qmt_site_recalc_cb ========================================================== 03:57:28 (1789891048)
[ 2600.105414] Lustre: Failing over lustre-OST0000
[ 2600.232745] Lustre: server umount lustre-OST0000 complete
[ 2601.954257] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107
[ 2601.962248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2601.966627] LustreError: 124076:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2601.973160] Lustre: Skipped 4 previous similar messages
[ 2601.991528] LustreError: 124076:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages
[ 2611.237562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2611.635045] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 2611.667794] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect
[ 2612.789630] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect
[ 2613.657080] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted.
[ 2613.657123] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo)
[ 2613.684256] Lustre: Skipped 5 previous similar messages
[ 2617.048872] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2646.433647] Lustre: DEBUG MARKER: == sanity-quota test 15: Set over 4T block quota ========= 03:58:35 (1789891115)
[ 2662.370522] Lustre: DEBUG MARKER: == sanity-quota test 16a: lfs quota should skip the inactive MDT/OST ========================================================== 03:58:52 (1789891132)
[ 2668.880900] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 2669.512277] Lustre: DEBUG MARKER: sanity-quota test_16a: @@@@@@ FAIL: failed to get quota after deactivate OSC
[ 2671.972886] Lustre: lustre-OST0000: Client 419c915b-7832-4215-a53a-5e9f02d6a2fe (at 192.168.203.38@tcp) reconnecting
[ 2684.844278] Lustre: DEBUG MARKER: == sanity-quota test 16b: lfs quota should skip the nonexistent MDT/OST ========================================================== 03:59:14 (1789891154)
[ 2685.357715] Lustre: DEBUG MARKER: SKIP: sanity-quota test_16b needs >= 3 MDTs
[ 2685.937193] Lustre: DEBUG MARKER: == sanity-quota test 17: DQACQ return recoverable error == 03:59:15 (1789891155)
[ 2689.285466] Lustre: DEBUG MARKER: sanity-quota test_17: @@@@@@ FAIL: Used space(10240) for user quota_usr isn't 0.