[ 5035.191944] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5040.787243] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5048.765964] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5050.256489] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5052.183489] Lustre: Failing over lustre-MDT0000
[ 5052.386756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5052.395730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5052.411888] Lustre: Skipped 4 previous similar messages
[ 5052.430462] Lustre: Skipped 3 previous similar messages
[ 5052.646347] Lustre: server umount lustre-MDT0000 complete
[ 5056.703882] LustreError: 6517:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5056.729641] LustreError: 6517:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages
[ 5074.405378] Lustre: 3646:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789503103/real 1789503103]  req@ffff8ab248fcce00 x1876424821038336/t0(0) o400->MGC192.168.203.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789503119 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5074.467038] Lustre: 3646:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5074.485698] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5076.538675] LDISKFS-fs (dm-0): 7 truncates cleaned up
[ 5076.542393] LDISKFS-fs (dm-0): recovery complete
[ 5076.551381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5084.668952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up)
[ 5084.988593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5085.060148] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5086.094674] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 5090.281826] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5090.296178] Lustre: Skipped 1 previous similar message
[ 5090.353231] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 5090.406227] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:726 to 0x280000401:769)
[ 5090.406982] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:724 to 0x2c0000401:769)
[ 5090.482267] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5100.326040] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5102.022546] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5107.021436] Lustre: DEBUG MARKER: (dd_pid=130478, time=0, timeout=600)
[ 5133.491591] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5139.384528] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5146.903647] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5148.725418] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5150.656056] Lustre: Failing over lustre-MDT0000
[ 5151.711914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 5151.717997] LustreError: 116479:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5151.722179] LustreError: Skipped 2 previous similar messages
[ 5151.753684] LustreError: 116479:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 36 previous similar messages
[ 5152.931360] Lustre: server umount lustre-MDT0000 complete
[ 5168.095391] Lustre: 3644:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789503197/real 1789503197]  req@ffff8ab376902d80 x1876424821099520/t0(0) o400->MGC192.168.203.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789503213 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5168.139514] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5174.634063] LDISKFS-fs (dm-0): 3 truncates cleaned up
[ 5174.636206] LDISKFS-fs (dm-0): recovery complete
[ 5174.646702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5178.853392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5178.930085] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5179.552915] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 5183.977078] LustreError: 3643:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-lwp-MDT0001: namespace resource [0x200000006:0x20000:0x0].0x0 (ffff8ab343e60b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5183.993872] LustreError: 3643:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 5 previous similar messages
[ 5184.036755] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted.
[ 5184.098584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801)
[ 5184.099828] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:724 to 0x2c0000401:801)
[ 5184.722482] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5196.578316] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5198.503660] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5204.418935] Lustre: DEBUG MARKER: (dd_pid=132867, time=0, timeout=600)
[ 5249.935046] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 16:14:52 (1789503292)
[ 5298.996439] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 16:15:43 (1789503343)
[ 5326.126870] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 5340.188368] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 5342.330537] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 5344.609543] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 5347.341363] Lustre: DEBUG MARKER: Set quota for 1 times
[ 5351.263205] Lustre: DEBUG MARKER: Set quota for 2 times
[ 5354.982951] Lustre: DEBUG MARKER: Set quota for 3 times
[ 5358.761840] Lustre: DEBUG MARKER: Set quota for 4 times
[ 5362.289618] Lustre: DEBUG MARKER: Set quota for 5 times
[ 5366.067300] Lustre: DEBUG MARKER: Set quota for 6 times
[ 5369.784794] Lustre: DEBUG MARKER: Set quota for 7 times
[ 5373.996140] Lustre: DEBUG MARKER: Set quota for 8 times
[ 5399.372382] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 16:17:23 (1789503443)
[ 5419.488070] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5419.488215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 5419.488894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5419.499806] Lustre: Skipped 6 previous similar messages
[ 5421.356933] Lustre: server umount lustre-MDT0000 complete
[ 5424.609274] LustreError: 6517:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5424.627189] LustreError: 6517:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 34 previous similar messages
[ 5425.491020] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789503471 with bad export cookie 9745662576463091417
[ 5425.491815] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5425.497454] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 5425.930982] Lustre: server umount lustre-MDT0001 complete
[ 5440.668993] Lustre: server umount lustre-OST0000 complete
[ 5445.614576] Lustre: 3644:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789503475/real 1789503475]  req@ffff8ab37ae1c700 x1876424821292544/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789503491 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5446.223449] Lustre: server umount lustre-OST0001 complete
[ 5462.307769] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local
[ 5472.998355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5473.541865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5477.843766] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5486.428584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5486.822600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 5490.763668] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5493.394803] Lustre: 158516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5499.775982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5500.242926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5506.335991] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5514.904273] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5515.211441] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5518.252847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:804 to 0x280000401:833)
[ 5518.255271] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:97)
[ 5518.263742] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:803 to 0x2c0000401:833)
[ 5521.851760] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5529.457214] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5532.735304] Lustre: 160393:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5553.632905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 5553.663163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5553.669365] Lustre: Skipped 3 previous similar messages
[ 5556.523553] Lustre: server umount lustre-MDT0000 complete
[ 5560.593445] LustreError: 157343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789503606 with bad export cookie 9745662576463102043
[ 5560.608213] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5560.623626] LustreError: 157343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages
[ 5561.339756] Lustre: server umount lustre-MDT0001 complete
[ 5575.750421] Lustre: server umount lustre-OST0000 complete
[ 5589.799916] Lustre: server umount lustre-OST0001 complete
[ 5608.068817] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local
[ 5619.825566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5620.494859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5626.231989] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5634.077880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5638.549971] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5641.233615] Lustre: 164184:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5647.777447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5654.304487] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5662.145969] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5662.451847] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5662.458079] Lustre: Skipped 2 previous similar messages
[ 5665.523363] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:129)
[ 5665.530860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:804 to 0x280000401:865)
[ 5665.625088] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:803 to 0x2c0000401:865)
[ 5669.359780] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5678.632441] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5683.021067] Lustre: 166056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5694.010679] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 16:22:18 (1789503738)
[ 5695.622258] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 6144
[ 5697.158863] Lustre: DEBUG MARKER: run for 4MB test file
[ 5707.845864] Lustre: DEBUG MARKER: User quota (limit: 4 MB)
[ 5713.295650] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 5715.022087] Lustre: DEBUG MARKER: Write half of file
[ 5717.353804] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5719.074805] Lustre: DEBUG MARKER: Step1: done
[ 5720.737742] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 5722.624818] Lustre: DEBUG MARKER: Step2: done
[ 5746.835847] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 61440
[ 5748.626397] Lustre: DEBUG MARKER: run for 40MB test file
[ 5760.829796] Lustre: DEBUG MARKER: User quota (limit: 40 MB)
[ 5767.773872] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 5769.345359] Lustre: DEBUG MARKER: Write half of file
[ 5772.713181] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5775.994246] Lustre: DEBUG MARKER: Step1: done
[ 5777.718972] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 5779.557497] Lustre: DEBUG MARKER: Step2: done
[ 5829.896784] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 16:24:33 (1789503873)
[ 5872.459066] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 16:25:16 (1789503916)
[ 5910.796449] Lustre: DEBUG MARKER: Write...
[ 5912.921430] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5964.011346] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 16:26:48 (1789504008)
[ 5971.027767] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 16:26:55 (1789504015)
[ 5980.773776] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 16:27:04 (1789504024)
[ 5988.955101] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 16:27:13 (1789504033)
[ 5996.734548] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 16:27:20 (1789504040)
[ 6049.681796] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6172.881391] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 6313.455209] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 16:32:37 (1789504357)
[ 6359.131721] Lustre: DEBUG MARKER: Restart...
[ 6366.688801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 6366.698309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6366.708741] Lustre: Skipped 6 previous similar messages
[ 6366.717049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6366.727452] Lustre: Skipped 4 previous similar messages
[ 6369.703854] Lustre: server umount lustre-MDT0000 complete
[ 6372.328623] LustreError: 163044:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6372.347528] LustreError: 163044:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 16 previous similar messages
[ 6373.417717] LustreError: 163023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789504419 with bad export cookie 9745662576463104346
[ 6373.423865] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6373.426441] LustreError: 163023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 6373.835726] Lustre: server umount lustre-MDT0001 complete
[ 6387.804854] Lustre: server umount lustre-OST0000 complete
[ 6401.804647] Lustre: server umount lustre-OST0001 complete
[ 6416.238289] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local
[ 6426.318101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6426.668427] LustreError: 189155:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6426.764236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6431.807302] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6440.090694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6440.720039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 6445.706217] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6448.730442] Lustre: 190300:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6455.556747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6462.159488] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6470.931603] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6471.210556] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6471.218559] Lustre: Skipped 1 previous similar message
[ 6472.243422] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:873 to 0x2c0000401:897)
[ 6472.245078] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:161)
[ 6472.248065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:877 to 0x280000401:897)
[ 6477.637430] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6485.176204] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6488.964318] Lustre: 192172:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6551.068569] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 16:36:35 (1789504595)
[ 6600.287741] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 16:37:24 (1789504644)
[ 8123.949525] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 17:02:48 (1789506168)
[ 8134.130080] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8134.136191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 8134.136481] Lustre: Skipped 5 previous similar messages
[ 8134.157817] Lustre: Skipped 7 previous similar messages
[ 8138.972710] Lustre: server umount lustre-MDT0000 complete
[ 8139.237110] LustreError: 189150:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8139.267693] LustreError: 189150:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages
[ 8142.846067] LustreError: 189136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789506188 with bad export cookie 9745662576463113453
[ 8142.862041] LustreError: 189136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 8142.864035] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8143.284942] Lustre: server umount lustre-MDT0001 complete
[ 8157.276495] Lustre: server umount lustre-OST0000 complete
[ 8160.229013] Lustre: 3644:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789506190/real 1789506190]  req@ffff8ab248fc3100 x1876424825170944/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789506206 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 8161.365351] Lustre: server umount lustre-OST0001 complete
[ 8176.745569] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local
[ 8187.083555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8187.515187] LustreError: 199693:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8187.711475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8193.334369] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8202.018058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8202.700150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 8208.747655] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8212.372144] Lustre: 200838:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 8220.095467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 8220.434749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 8227.097840] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8227.626494] LustreError: 201191:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8227.661225] LustreError: 201191:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages
[ 8233.980217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5899 to 0x280000401:5921)
[ 8235.624214] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 8241.139648] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5899 to 0x2c0000401:5921)
[ 8241.158813] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:193)
[ 8243.728916] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8251.549319] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8255.508674] Lustre: 202711:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 8283.633252] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 17:05:27 (1789506327)
[ 8313.995837] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 17:05:58 (1789506358)
[ 8342.910754] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 17:06:27 (1789506387)
[ 8375.004315] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 17:06:59 (1789506419)
[ 8410.245479] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 17:07:34 (1789506454)
[ 8484.411853] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 17:08:48 (1789506528)
[ 8517.741378] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 17:09:22 (1789506562)
[ 8575.292703] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 17:10:19 (1789506619)
[ 8583.275028] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8583.284952] Lustre: Skipped 1 previous similar message
[ 8585.359663] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8585.361535] Lustre: Skipped 53 previous similar messages
[ 8589.395137] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8589.410723] Lustre: Skipped 133 previous similar messages
[ 8597.412547] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8597.416680] Lustre: Skipped 309 previous similar messages
[ 8613.436164] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8613.444719] Lustre: Skipped 803 previous similar messages
[ 8645.436300] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8645.442058] Lustre: Skipped 1499 previous similar messages
[ 8709.488762] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8709.500792] Lustre: Skipped 2999 previous similar messages
[ 9036.138275] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9036.151436] Lustre: Skipped 2197 previous similar messages
[ 9036.166739] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9114.080664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 9114.100053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9114.122302] Lustre: Skipped 2 previous similar messages
[ 9114.128816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9117.246310] Lustre: server umount lustre-MDT0000 complete
[ 9119.715241] LustreError: 199693:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9119.732225] LustreError: 199693:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages
[ 9121.227618] LustreError: 208283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789507167 with bad export cookie 9745662576464890235
[ 9121.232315] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9121.248255] LustreError: 208283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 9124.831851] LustreError: 202874:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9124.832993] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9124.834187] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 9124.834194] Lustre: Skipped 3 previous similar messages
[ 9124.850481] LustreError: 202874:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message
[ 9124.874173] Lustre: Skipped 4 previous similar messages
[ 9126.885085] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 9126.890136] Lustre: Skipped 2 previous similar messages
[ 9127.679162] Lustre: server umount lustre-MDT0001 complete
[ 9133.066362] Lustre: server umount lustre-OST0000 complete
[ 9137.626948] Lustre: server umount lustre-OST0001 complete
[ 9144.755297] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid
[ 9152.594737] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local
[ 9212.311634] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local
[ 9222.377477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9222.660190] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 9222.695501] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 9222.778100] Lustre: lustre-MDT0000: new disk, initializing
[ 9222.826520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9222.835181] Lustre: Skipped 1 previous similar message
[ 9222.845500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 9226.825617] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9237.283537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9237.438404] Lustre: 219510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17
[ 9237.473980] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
[ 9237.479107] Lustre: Skipped 1 previous similar message
[ 9237.594870] Lustre: lustre-MDT0001: new disk, initializing
[ 9237.698528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 9237.749626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt
[ 9237.765370] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt]
[ 9242.500183] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9247.752256] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 9254.378196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 9254.584867] Lustre: lustre-OST0000: new disk, initializing
[ 9254.588881] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 9254.594889] Lustre: 221142:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0]
[ 9254.659861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9256.055551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost
[ 9256.060254] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost]
[ 9256.120690] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401
[ 9261.833892] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9273.428956] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 9273.646189] Lustre: lustre-OST0001: new disk, initializing
[ 9273.650593] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 9273.656342] Lustre: 222014:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0]
[ 9273.728992] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9275.622793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost
[ 9275.637515] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost]
[ 9275.699718] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401
[ 9281.306897] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9290.774608] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9294.392466] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 9320.988456] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 17:22:45 (1789507365)
[ 9327.913930] Lustre: DEBUG MARKER: == sanity-quota test 49c: lfs quota long options don't consume an extra argument ========================================================== 17:22:52 (1789507372)
[ 9335.133571] Lustre: DEBUG MARKER: == sanity-quota test 49d: lfs quota -d and --delimiter both work ========================================================== 17:22:59 (1789507379)
[ 9341.686798] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 17:23:06 (1789507386)
[ 9376.523467] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 17:23:40 (1789507420)
[ 9423.122142] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 17:24:27 (1789507467)
[ 9445.793197] Lustre: DEBUG MARKER: rename directory return 255
[ 9486.713970] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 17:25:30 (1789507530)
[ 9507.546730] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 17:25:51 (1789507551)
[ 9540.700461] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 17:26:24 (1789507584)
[ 9666.586594] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 17:28:30 (1789507710)
[ 9694.317904] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 17:28:58 (1789507738)
[ 9728.631597] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 17:29:32 (1789507772)
[ 9869.290382] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 17:31:53 (1789507913)
[ 9874.913342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9874.917507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9874.937539] Lustre: Skipped 1 previous similar message
[ 9880.045240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9880.049602] Lustre: Skipped 6 previous similar messages
[ 9880.431057] Lustre: server umount lustre-MDT0000 complete
[ 9884.509569] LustreError: 219503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789507930 with bad export cookie 9745662576465262404
[ 9884.519181] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9884.530546] LustreError: 219503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 9885.161882] LustreError: 219516:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9885.163633] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9885.191666] LustreError: 219516:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages
[ 9885.207178] Lustre: Skipped 2 previous similar messages
[ 9885.339165] Lustre: server umount lustre-MDT0001 complete
[ 9899.356104] Lustre: server umount lustre-OST0000 complete
[ 9903.396929] Lustre: server umount lustre-OST0001 complete
[ 9924.516915] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local
[ 9935.928440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9936.467726] LustreError: 239623:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9936.630181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9936.639925] LustreError: lustre-MDT0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 9941.912523] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9941.991222] LustreError: 239624:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9947.116429] LustreError: 239623:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9950.324735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9950.630406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 9950.638788] LustreError: lustre-MDT0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 9954.623260] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9957.607591] Lustre: 240769:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9964.147801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 9964.450530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9964.460304] LustreError: lustre-OST0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 9970.762415] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9975.729220] LustreError: 241713:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9979.462944] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 9979.666210] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9979.683916] LustreError: lustre-OST0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 9982.790217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:65)
[ 9982.790217] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:20 to 0x2c0000401:65)
[ 9986.399458] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9993.121428] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9996.347221] Lustre: 242643:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[10003.872876] LustreError: 239620:0:(osd_handler.c:3424:osd_quota_transfer()) lustre-MDT0000: quota transfer failed. Is project enforcement enabled on the ldiskfs filesystem? rc = -95
[10008.545988] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10008.547019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10008.556312] Lustre: Skipped 3 previous similar messages
[10008.577663] Lustre: Skipped 3 previous similar messages
[10013.664053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10013.679571] Lustre: Skipped 3 previous similar messages
[10014.709969] Lustre: server umount lustre-MDT0000 complete
[10017.826634] LustreError: 239603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789508063 with bad export cookie 9745662576465311250
[10017.828837] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10017.862991] LustreError: 239603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[10018.190561] Lustre: server umount lustre-MDT0001 complete
[10028.000348] LustreError: 3645:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff8ab244098000 x1876424832295936/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[10028.048336] LustreError: 3644:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0000 qtype:usr lqe: ffff8ab249221380 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 1516 qunit:0 qtune:0 edquot:0 default:no revoke:0
[10028.091662] LustreError: 3644:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 1 previous similar message
[10032.017222] Lustre: server umount lustre-OST0000 complete
[10034.271539] Lustre: 3644:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789508064/real 1789508064]  req@ffff8ab2532fbb80 x1876424832295040/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789508080 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[10034.294569] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10035.681279] Lustre: server umount lustre-OST0001 complete
[10056.191812] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local
[10067.167362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[10067.582829] LustreError: 245032:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10067.657683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10071.993765] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10079.709049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[10084.769168] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10087.759387] Lustre: 246179:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[10096.116349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[10103.746628] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10104.547945] LustreError: 246535:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10104.576300] LustreError: 246535:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages
[10112.793851] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[10112.993593] Lustre: lustre-OST0001: Not available for connect from 0@lo (not set up)
[10113.010865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97)
[10113.120390] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10113.128539] Lustre: Skipped 2 previous similar messages
[10118.148028] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:20 to 0x2c0000401:97)
[10119.613732] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10126.599990] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10129.912302] Lustre: 248057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[10152.815900] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 17:36:37 (1789508197)
[10194.983711] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[10196.550069] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 17:37:20 (1789508240)
[10215.626266] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[10217.150793] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 17:37:41 (1789508261)
[10255.719349] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[10257.212138] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 17:38:21 (1789508301)
[10286.340280] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 17:38:50 (1789508330)
[10297.171334] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[10303.505194] Lustre: 254662:0:(qsd_reint.c:249:qsd_reint_index()) lustre-OST0000: index version for fid [0x200000005:0x1007:0x0] is 0, but index isn't empty (1)
[10308.157844] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[10313.768790] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[10317.298740] Lustre: DEBUG MARKER: Write...
[10336.967643] LustreError: 256008:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[10342.933352] Lustre: DEBUG MARKER: Write...
[10354.556053] Lustre: DEBUG MARKER: Write...
[10425.289543] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 17:41:09 (1789508469)
[10449.174443] LustreError: 259973:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[10491.433461] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 17:42:15 (1789508535)
[10518.667921] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[10520.480199] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[10606.598530] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 17:44:10 (1789508650)
[10648.337763] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 17:44:52 (1789508692)
[10682.017215] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 17:45:26 (1789508726)
[10693.180896] Lustre: DEBUG MARKER: User quota (block hardlimit:100 MB)
[10714.127149] Lustre: DEBUG MARKER: Write...
[10715.910879] Lustre: DEBUG MARKER: Write out of block quota ...
[10798.461850] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 17:47:22 (1789508842)
[10809.397588] Lustre: DEBUG MARKER: User quota (block hardlimit:1000 MB)
[10887.261763] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 17:48:51 (1789508931)
[10897.805575] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[10918.166752] Lustre: DEBUG MARKER: Write...
[10919.936674] Lustre: DEBUG MARKER: Write out of block quota ...
[10920.213725] LustreError: 254668:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:10240 soft:0 granted:10240 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[10973.667591] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 17:50:17 (1789509017)
[11000.711948] Lustre: DEBUG MARKER: set to use default quota
[11002.592164] Lustre: DEBUG MARKER: set default quota
[11004.357579] Lustre: DEBUG MARKER: get default quota
[11009.859304] Lustre: DEBUG MARKER: Test not out of quota
[11012.865990] Lustre: DEBUG MARKER: Test out of quota
[11023.070797] Lustre: DEBUG MARKER: Increase default quota
[11044.809230] Lustre: DEBUG MARKER: Set quota to override default quota
[11044.854669] LustreError: 254159:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45056 time:1790113890 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11054.996949] Lustre: DEBUG MARKER: Set to use default quota again
[11074.350311] Lustre: DEBUG MARKER: Cleanup
[11131.997418] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 17:52:56 (1789509176)
[11152.808743] Lustre: DEBUG MARKER: set default quota for qpool1
[11154.964234] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[11194.354078] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 17:53:58 (1789509238)
[11265.354328] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 17:55:09 (1789509309)
[11326.030684] Lustre: DEBUG MARKER: Write...
[11328.719116] Lustre: DEBUG MARKER: Write out of block quota ...
[11424.732636] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 17:57:49 (1789509469)
[11460.244802] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 17:58:24 (1789509504)
[11466.454213] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 17:58:31 (1789509511)
[11500.203919] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 17:59:04 (1789509544)
[11541.181762] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 17:59:45 (1789509585)
[11559.939261] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 18:00:04 (1789509604)
[11583.543418] Lustre: *** cfs_fail_loc=a06, val=0***
[11583.724278] LustreError: 245046:0:(qmt_lock.c:479:qmt_lvbo_update()) $$$ failed to release quota space on glimpse 0!=2048 : rc = -11
                 qmt:lustre-QMT0000 pool:dt-0x0 id:60000 enforced:1 hard:102400 soft:0 granted:24576 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no
[11585.518063] Lustre: DEBUG MARKER: Files for user (quota_usr), count=5:
[11586.959365] Lustre: DEBUG MARKER: sanity-quota test_80: @@@@@@ FAIL: write failed