[ 4724.519751] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 4732.149883] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 4740.864798] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4742.693501] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 4744.819995] Lustre: Failing over lustre-MDT0000
[ 4745.239995] Lustre: server umount lustre-MDT0000 complete
[ 4747.889475] LustreError: 6530:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4748.257486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 4748.263114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4748.277164] Lustre: Skipped 5 previous similar messages
[ 4765.151909] Lustre: 3649:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788817801/real 1788817801]  req@ffff8fc748c9d880 x1875706586180352/t0(0) o400->MGC192.168.201.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788817817 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4765.179888] Lustre: 3649:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 4765.190912] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4768.480996] LDISKFS-fs (dm-0): 7 truncates cleaned up
[ 4768.484287] LDISKFS-fs (dm-0): recovery complete
[ 4768.492241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4775.607808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4775.643423] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4776.544074] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 4779.318434] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4781.034652] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 4781.048400] Lustre: Skipped 5 previous similar messages
[ 4781.101298] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted.
[ 4781.148860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:847 to 0x280000401:865)
[ 4781.149295] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:812 to 0x2c0000401:833)
[ 4788.132446] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4789.712609] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4795.763664] Lustre: DEBUG MARKER: (dd_pid=131674, time=0, timeout=600)
[ 4826.021778] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 4834.844269] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 4846.116647] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4847.919338] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 4850.421083] Lustre: Failing over lustre-MDT0000
[ 4850.830095] Lustre: server umount lustre-MDT0000 complete
[ 4852.709487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 4852.715019] LustreError: 6530:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4852.733628] LustreError: 6530:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 37 previous similar messages
[ 4869.087365] Lustre: 3651:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788817905/real 1788817905]  req@ffff8fc74e630700 x1875706586246144/t0(0) o400->MGC192.168.201.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788817921 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4869.125973] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4877.510183] LDISKFS-fs (dm-0): 3 truncates cleaned up
[ 4877.518324] LDISKFS-fs (dm-0): recovery complete
[ 4877.527847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4878.623119] LustreError: 149822:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0)
[ 4878.631278] LustreError: 149822:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff8fc749738380 x1875706586251904/t0(0) o250->MGC192.168.201.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1788817931 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4878.646174] LustreError: 149822:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out.
[ 4879.337323] LustreError: 3648:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff8fc749738380 x1875706586255616/t0(0) o250->MGC192.168.201.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4879.620759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4879.657616] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4879.943272] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 4884.693504] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4884.989896] LustreError: 3648:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-lwp-MDT0001: namespace resource [0x200000006:0x10000:0x0].0x0 (ffff8fc85ff9ad00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4885.016635] LustreError: 3648:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 5 previous similar messages
[ 4885.086115] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted.
[ 4885.135857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:867 to 0x280000401:897)
[ 4885.135915] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:812 to 0x2c0000401:865)
[ 4897.937176] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4900.547105] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4907.772942] Lustre: DEBUG MARKER: (dd_pid=134113, time=0, timeout=600)
[ 4947.507113] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 17:53:19 (1788817999)
[ 4996.011839] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 17:54:06 (1788818046)
[ 5022.805112] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 5034.507779] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 5036.395592] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 5038.337958] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 5040.339665] Lustre: DEBUG MARKER: Set quota for 1 times
[ 5043.666667] Lustre: DEBUG MARKER: Set quota for 2 times
[ 5047.147098] Lustre: DEBUG MARKER: Set quota for 3 times
[ 5050.283682] Lustre: DEBUG MARKER: Set quota for 4 times
[ 5053.568956] Lustre: DEBUG MARKER: Set quota for 5 times
[ 5056.499854] Lustre: DEBUG MARKER: Set quota for 6 times
[ 5059.744456] Lustre: DEBUG MARKER: Set quota for 7 times
[ 5063.341207] Lustre: DEBUG MARKER: Set quota for 8 times
[ 5066.533333] Lustre: DEBUG MARKER: Set quota for 9 times
[ 5099.797734] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 17:55:50 (1788818150)
[ 5118.437049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5118.443745] Lustre: Skipped 3 previous similar messages
[ 5122.547926] Lustre: server umount lustre-MDT0000 complete
[ 5123.552732] LustreError: 117709:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5123.626779] LustreError: 117709:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 33 previous similar messages
[ 5126.717267] LustreError: 6513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788818179 with bad export cookie 8354984234699059115
[ 5126.727300] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5126.741430] LustreError: 6513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 5127.107831] Lustre: server umount lustre-MDT0001 complete
[ 5141.543750] Lustre: server umount lustre-OST0000 complete
[ 5144.870919] Lustre: 3651:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788818181/real 1788818181]  req@ffff8fc8815eb100 x1875706586446976/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788818197 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5146.312986] Lustre: server umount lustre-OST0001 complete
[ 5164.044339] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local
[ 5176.782150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5177.363354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5182.684438] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5192.581840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5193.309435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 5198.923585] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5202.338561] Lustre: 159755:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5210.735720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5211.192070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5218.451525] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5222.467141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929)
[ 5228.246612] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5228.592472] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5233.644154] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:868 to 0x2c0000401:897)
[ 5233.716810] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:97)
[ 5235.292936] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5244.046468] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5248.213812] Lustre: 161634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5269.475833] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5269.479755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5269.516459] Lustre: Skipped 13 previous similar messages
[ 5274.596609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5274.610931] Lustre: Skipped 6 previous similar messages
[ 5274.756084] Lustre: server umount lustre-MDT0000 complete
[ 5278.876399] LustreError: 160134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788818331 with bad export cookie 8354984234699071302
[ 5278.887056] LustreError: 160134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 5278.897821] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5279.273137] Lustre: server umount lustre-MDT0001 complete
[ 5293.871802] Lustre: server umount lustre-OST0000 complete
[ 5296.096102] Lustre: 3651:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788818332/real 1788818332]  req@ffff8fc87f51a300 x1875706586530304/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788818348 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5298.325739] Lustre: server umount lustre-OST0001 complete
[ 5322.175341] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local
[ 5335.829391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5336.488030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5341.540869] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5350.410715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5357.226397] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5360.475844] Lustre: 165373:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5367.601861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5375.653233] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5381.350619] LustreError: 165728:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5381.363601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:961)
[ 5381.371511] LustreError: 165728:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages
[ 5383.438410] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5383.728876] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5383.735730] Lustre: Skipped 2 previous similar messages
[ 5388.791518] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:868 to 0x2c0000401:929)
[ 5388.803895] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:129)
[ 5391.256949] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5399.029508] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5402.652409] Lustre: 167245:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5416.368901] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 18:01:07 (1788818467)
[ 5418.707429] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 6144
[ 5420.651899] Lustre: DEBUG MARKER: run for 4MB test file
[ 5438.493780] Lustre: DEBUG MARKER: User quota (limit: 4 MB)
[ 5446.783397] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 5449.883149] Lustre: DEBUG MARKER: Write half of file
[ 5452.745657] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5454.632902] Lustre: DEBUG MARKER: Step1: done
[ 5456.515458] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 5458.355553] Lustre: DEBUG MARKER: Step2: done
[ 5487.234577] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 61440
[ 5489.824899] Lustre: DEBUG MARKER: run for 40MB test file
[ 5504.031397] Lustre: DEBUG MARKER: User quota (limit: 40 MB)
[ 5511.352826] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 5513.448990] Lustre: DEBUG MARKER: Write half of file
[ 5516.732813] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5519.871047] Lustre: DEBUG MARKER: Step1: done
[ 5521.843142] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 5523.216113] Lustre: DEBUG MARKER: Step2: done
[ 5581.680495] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 18:03:53 (1788818633)
[ 5630.132915] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 18:04:41 (1788818681)
[ 5668.665202] Lustre: DEBUG MARKER: Write...
[ 5671.489356] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5723.082742] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 18:06:14 (1788818774)
[ 5731.246345] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 18:06:22 (1788818782)
[ 5741.799589] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 18:06:33 (1788818793)
[ 5752.131890] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 18:06:43 (1788818803)
[ 5760.281773] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 18:06:52 (1788818812)
[ 5812.499886] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 5945.216449] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 6077.953497] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 18:12:09 (1788819129)
[ 6127.490173] Lustre: DEBUG MARKER: Restart...
[ 6134.241682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 6134.244530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6134.247817] Lustre: Skipped 3 previous similar messages
[ 6134.253458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6136.306980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6136.321348] Lustre: Skipped 1 previous similar message
[ 6137.945113] Lustre: server umount lustre-MDT0000 complete
[ 6141.411395] LustreError: 164233:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6141.431602] LustreError: 164233:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages
[ 6141.628468] LustreError: 176948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788819194 with bad export cookie 8354984234699073605
[ 6141.630374] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6141.631883] LustreError: 176948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 6141.852303] Lustre: server umount lustre-MDT0001 complete
[ 6155.873695] Lustre: server umount lustre-OST0000 complete
[ 6170.117391] Lustre: server umount lustre-OST0001 complete
[ 6190.221377] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local
[ 6202.140461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6202.572701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6208.744507] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6218.829019] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6219.172598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 6223.402034] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6226.607396] Lustre: 191421:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6233.744226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6242.303611] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6247.543510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:972 to 0x280000401:993)
[ 6253.646677] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6253.926627] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6253.930259] Lustre: Skipped 1 previous similar message
[ 6259.189251] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:938 to 0x2c0000401:961)
[ 6259.334135] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:161)
[ 6264.061589] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6274.103430] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6278.990753] Lustre: 193294:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6352.555140] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 18:16:43 (1788819403)
[ 6430.326692] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 18:18:01 (1788819481)
[ 8214.685627] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 18:47:46 (1788821266)
[ 8224.195868] Lustre: server umount lustre-MDT0000 complete
[ 8225.259228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8225.261152] LustreError: 191012:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8225.291487] Lustre: Skipped 6 previous similar messages
[ 8225.319944] LustreError: 191012:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages
[ 8227.974275] LustreError: 190261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788821280 with bad export cookie 8354984234699082712
[ 8227.984611] LustreError: 190261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 8227.984783] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8228.209210] Lustre: server umount lustre-MDT0001 complete
[ 8242.223074] Lustre: server umount lustre-OST0000 complete
[ 8256.670849] Lustre: server umount lustre-OST0001 complete
[ 8274.651846] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local
[ 8286.895398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8287.511598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8291.805988] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8299.946511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8300.303663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 8304.147187] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8307.945761] Lustre: 201939:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 8315.361657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 8315.762917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 8324.970771] LustreError: 202294:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8325.005023] LustreError: 202294:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages
[ 8325.104583] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8332.287750] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5995 to 0x280000401:6017)
[ 8336.348976] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 8336.622365] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 8342.004143] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:193)
[ 8342.007204] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5963 to 0x2c0000401:5985)
[ 8344.701795] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8355.149692] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8359.130663] Lustre: 203814:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 8394.229699] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 18:50:45 (1788821445)
[ 8430.860745] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 18:51:22 (1788821482)
[ 8461.020756] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 18:51:52 (1788821512)
[ 8503.742525] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 18:52:33 (1788821553)
[ 8553.160283] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 18:53:24 (1788821604)
[ 8641.138712] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 18:54:52 (1788821692)
[ 8677.916608] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 18:55:28 (1788821728)
[ 8753.627678] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 18:56:44 (1788821804)
[ 8760.591756] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8760.595356] Lustre: Skipped 2 previous similar messages
[ 8762.591158] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8762.593155] Lustre: Skipped 85 previous similar messages
[ 8766.600113] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8766.602082] Lustre: Skipped 161 previous similar messages
[ 8774.619644] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8774.622259] Lustre: Skipped 393 previous similar messages
[ 8790.662732] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8790.665286] Lustre: Skipped 761 previous similar messages
[ 8822.663325] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8822.670837] Lustre: Skipped 1491 previous similar messages
[ 8886.675842] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8886.681568] Lustre: Skipped 2489 previous similar messages
[ 9319.394838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 9319.409813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9319.417876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9319.422125] Lustre: Skipped 1 previous similar message
[ 9325.028382] LustreError: 201860:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9325.055068] LustreError: 201860:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages
[ 9326.222346] Lustre: server umount lustre-MDT0000 complete
[ 9330.815109] LustreError: 200778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788822383 with bad export cookie 8354984234700862126
[ 9330.820714] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9330.829879] LustreError: 200778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 9335.267866] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9335.277204] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 9335.294560] Lustre: Skipped 4 previous similar messages
[ 9335.301294] Lustre: Skipped 3 previous similar messages
[ 9337.178185] Lustre: server umount lustre-MDT0001 complete
[ 9342.016210] Lustre: server umount lustre-OST0000 complete
[ 9347.875987] Lustre: server umount lustre-OST0001 complete
[ 9356.490739] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid
[ 9366.900158] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local
[ 9418.064634] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local
[ 9429.919460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9430.103575] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 9430.119238] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 9430.204601] Lustre: lustre-MDT0000: new disk, initializing
[ 9430.267295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9430.285899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 9435.305815] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9446.396216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9446.529810] Lustre: 220680:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17
[ 9446.565673] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
[ 9446.569705] Lustre: Skipped 1 previous similar message
[ 9446.642283] Lustre: lustre-MDT0001: new disk, initializing
[ 9446.706823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 9446.724754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt
[ 9446.737742] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt]
[ 9451.241106] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9456.019258] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 9463.089895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 9463.411847] Lustre: lustre-OST0000: new disk, initializing
[ 9463.414667] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 9463.417772] Lustre: 222313:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0]
[ 9463.556394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9465.442436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost
[ 9465.452281] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost]
[ 9465.540817] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401
[ 9470.855992] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9484.457518] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 9484.540093] Lustre: lustre-OST0001: new disk, initializing
[ 9484.542951] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 9484.551369] Lustre: 223184:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0]
[ 9484.611437] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9486.074210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost
[ 9486.090511] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost]
[ 9486.208744] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401
[ 9491.743948] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9501.893770] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9506.120446] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 9534.571704] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 19:09:46 (1788822586)
[ 9540.844491] Lustre: DEBUG MARKER: == sanity-quota test 49c: lfs quota long options don't consume an extra argument ========================================================== 19:09:52 (1788822592)
[ 9548.018297] Lustre: DEBUG MARKER: == sanity-quota test 49d: lfs quota -d and --delimiter both work ========================================================== 19:09:59 (1788822599)
[ 9554.949651] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 19:10:06 (1788822606)
[ 9590.786396] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 19:10:42 (1788822642)
[ 9659.615482] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 19:11:50 (1788822710)
[ 9684.078844] Lustre: DEBUG MARKER: rename directory return 255
[ 9725.317964] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 19:12:56 (1788822776)
[ 9751.069576] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 19:13:23 (1788822803)
[ 9782.073734] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 19:13:53 (1788822833)
[ 9896.134271] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 19:15:48 (1788822948)
[ 9926.535987] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 19:16:17 (1788822977)
[ 9966.050568] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 19:16:57 (1788823017)
[10252.632731] Lustre: DEBUG MARKER: sanity-quota test_58: @@@@@@ FAIL: wait_delete_completed failed