[ 4542.112770] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 4545.721763] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 4553.625318] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4555.287465] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 4557.281866] Lustre: Failing over lustre-MDT0000
[ 4557.562156] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.30@tcp (stopping)
[ 4557.570853] LustreError: 137525:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup
[ 4557.580922] LustreError: 137525:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages
[ 4557.886464] Lustre: server umount lustre-MDT0000 complete
[ 4558.307324] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 4558.315728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4558.331398] LustreError: 14444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4558.347057] LustreError: 14444:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages
[ 4576.735612] Lustre: 6804:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319819/real 1767319819]  req@ffffa0740adf7100 x1853164499288832/t0(0) o400->MGC192.168.202.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767319835 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4576.757072] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4584.418989] LDISKFS-fs (dm-0): 7 truncates cleaned up
[ 4584.420960] LDISKFS-fs (dm-0): recovery complete
[ 4584.428620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4585.956453] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe8e8e5e42550b95d
[ 4585.978810] Lustre: MGC192.168.202.130@tcp: Connection restored to 0@lo (at 0@lo)
[ 4585.986356] Lustre: Skipped 1 previous similar message
[ 4586.193894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4586.233029] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4586.268132] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 4590.430456] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4591.671427] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted.
[ 4591.710054] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:765 to 0x280000401:801)
[ 4591.711093] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:698 to 0x2c0000401:737)
[ 4597.059587] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4598.581280] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4603.338893] Lustre: DEBUG MARKER: (dd_pid=111389, time=0, timeout=600)
[ 4627.632364] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 4632.964495] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 4640.707337] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4642.321269] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 4644.308730] Lustre: Failing over lustre-MDT0000
[ 4644.476322] LustreError: 140717:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup
[ 4644.484787] LustreError: 140717:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 4644.586725] Lustre: server umount lustre-MDT0000 complete
[ 4647.927507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 4664.293546] Lustre: 6804:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319907/real 1767319907]  req@ffffa0740e1e3b80 x1853164499348096/t0(0) o400->MGC192.168.202.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767319923 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4664.323483] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4672.146250] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 4672.150370] LDISKFS-fs (dm-0): recovery complete
[ 4672.162666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4674.336116] LustreError: 141625:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0)
[ 4674.351044] LustreError: 141625:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffffa07541c18a80 x1853164499354112/t0(0) o250->MGC192.168.202.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1767319933 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4674.377391] LustreError: 141625:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out.
[ 4674.860963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4674.896038] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4674.934049] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 4679.449773] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4680.172889] LustreError: 6802:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-lwp-MDT0001: namespace resource [0x200000006:0x20000:0x0].0x0 (ffffa0750e3a7000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4680.192172] LustreError: 6802:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 5 previous similar messages
[ 4680.249384] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted.
[ 4680.278055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:803 to 0x280000401:833)
[ 4680.285288] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:698 to 0x2c0000401:769)
[ 4686.485363] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4687.716513] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4693.254296] Lustre: DEBUG MARKER: (dd_pid=113680, time=0, timeout=600)
[ 4725.243992] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 21:13:04 (1767319984)
[ 4766.235136] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 21:13:45 (1767320025)
[ 4784.029523] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 4794.995979] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 4796.444607] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 4797.793738] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 4799.388778] Lustre: DEBUG MARKER: Set quota for 1 times
[ 4802.338601] Lustre: DEBUG MARKER: Set quota for 2 times
[ 4805.240471] Lustre: DEBUG MARKER: Set quota for 3 times
[ 4808.389534] Lustre: DEBUG MARKER: Set quota for 4 times
[ 4811.493448] Lustre: DEBUG MARKER: Set quota for 5 times
[ 4814.345834] Lustre: DEBUG MARKER: Set quota for 6 times
[ 4817.571380] Lustre: DEBUG MARKER: Set quota for 7 times
[ 4820.510679] Lustre: DEBUG MARKER: Set quota for 8 times
[ 4823.666641] Lustre: DEBUG MARKER: Set quota for 9 times
[ 4826.364942] Lustre: DEBUG MARKER: Set quota for 10 times
[ 4829.092220] Lustre: DEBUG MARKER: Set quota for 11 times
[ 4860.056103] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 21:15:18 (1767320118)
[ 4874.719650] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 4874.721193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4874.726835] Lustre: Skipped 2 previous similar messages
[ 4879.358499] LustreError: 148156:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup
[ 4879.375135] LustreError: 148156:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 4879.431779] Lustre: server umount lustre-MDT0000 complete
[ 4883.245294] LustreError: 14425:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767320142 with bad export cookie 16782916779437507577
[ 4883.269675] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4883.660560] Lustre: server umount lustre-MDT0001 complete
[ 4899.777154] LustreError: 148704:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup
[ 4899.781550] LustreError: 148704:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages
[ 4899.898417] Lustre: server umount lustre-OST0000 complete
[ 4901.343798] Lustre: 6806:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767320144/real 1767320144]  req@ffffa0740adf4380 x1853164499526656/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767320160 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4903.119206] Lustre: server umount lustre-OST0001 complete
[ 4920.034090] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local
[ 4932.999164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4933.440262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4936.737546] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4942.099724] Lustre: 151264:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 4949.484657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 4949.733082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 4953.405188] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4958.459943] Lustre: 152356:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 4967.035751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 4967.266035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 4972.335350] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4974.446374] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97)
[ 4979.573559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:836 to 0x280000401:865)
[ 4983.101595] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 4983.450624] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 4985.189775] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801)
[ 4990.322735] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4996.728769] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5000.004662] Lustre: 154594:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5024.740226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5024.743922] Lustre: Skipped 1 previous similar message
[ 5027.972775] LustreError: 155445:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 5027.979673] LustreError: 155445:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages
[ 5028.063117] Lustre: server umount lustre-MDT0000 complete
[ 5031.683834] LustreError: 154596:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767320290 with bad export cookie 16782916779437515592
[ 5031.688377] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5031.695609] LustreError: 154596:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages
[ 5031.901908] Lustre: server umount lustre-MDT0001 complete
[ 5047.523961] Lustre: server umount lustre-OST0000 complete
[ 5060.814237] Lustre: server umount lustre-OST0001 complete
[ 5078.933403] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local
[ 5092.270890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5092.644545] LustreError: 157996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5092.662400] LustreError: 157996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 82 previous similar messages
[ 5092.759930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5096.364132] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5100.932877] Lustre: 158549:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5109.481282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 5113.914987] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5118.567294] Lustre: 159635:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5129.719993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 5130.332304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5130.338819] Lustre: Skipped 1 previous similar message
[ 5134.446330] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:129)
[ 5135.991832] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5137.527544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:836 to 0x280000401:897)
[ 5148.182988] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 5149.580802] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833)
[ 5154.344824] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5161.468771] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5164.888708] Lustre: 161867:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5178.705634] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 21:20:37 (1767320437)
[ 5179.900703] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 6144
[ 5181.011456] Lustre: DEBUG MARKER: run for 4MB test file
[ 5192.025811] Lustre: DEBUG MARKER: User quota (limit: 4 MB)
[ 5196.220646] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 5197.719242] Lustre: DEBUG MARKER: Write half of file
[ 5199.256332] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5200.730384] Lustre: DEBUG MARKER: Step1: done
[ 5202.125543] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 5203.619272] Lustre: DEBUG MARKER: Step2: done
[ 5225.701038] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 61440
[ 5227.361827] Lustre: DEBUG MARKER: run for 40MB test file
[ 5237.373635] Lustre: DEBUG MARKER: User quota (limit: 40 MB)
[ 5241.175465] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 5242.526617] Lustre: DEBUG MARKER: Write half of file
[ 5245.198965] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5247.829741] Lustre: DEBUG MARKER: Step1: done
[ 5249.023516] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 5250.418546] Lustre: DEBUG MARKER: Step2: done
[ 5294.995493] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 21:22:33 (1767320553)
[ 5334.422709] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 21:23:13 (1767320593)
[ 5368.129433] Lustre: DEBUG MARKER: Write...
[ 5369.934694] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5410.815398] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 21:24:29 (1767320669)
[ 5416.627677] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 21:24:35 (1767320675)
[ 5425.569572] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 21:24:44 (1767320684)
[ 5432.868949] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 21:24:51 (1767320691)
[ 5440.297698] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 21:24:58 (1767320698)
[ 5488.664241] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 5579.930881] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 5686.180489] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 21:29:04 (1767320944)
[ 5722.457482] Lustre: DEBUG MARKER: Restart...
[ 5728.226981] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5728.227455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5728.239092] Lustre: Skipped 18 previous similar messages
[ 5728.247709] Lustre: Skipped 3 previous similar messages
[ 5731.934962] LustreError: 180756:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 5731.940570] LustreError: 180756:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages
[ 5732.014696] Lustre: server umount lustre-MDT0000 complete
[ 5733.352428] LustreError: 157995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5733.384275] LustreError: 157995:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages
[ 5735.415152] LustreError: 157977:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767320994 with bad export cookie 16782916779437517916
[ 5735.425640] LustreError: 157977:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 5735.427819] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5735.706756] Lustre: server umount lustre-MDT0001 complete
[ 5751.879155] LustreError: 181303:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup
[ 5751.889461] LustreError: 181303:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages
[ 5751.926061] Lustre: server umount lustre-OST0000 complete
[ 5754.655150] Lustre: 6806:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767320997/real 1767320997]  req@ffffa0740f7a8700 x1853164500028416/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767321013 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5755.298101] Lustre: server umount lustre-OST0001 complete
[ 5774.791373] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local
[ 5792.297779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5792.760746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5792.763228] Lustre: Skipped 1 previous similar message
[ 5797.209853] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5802.366689] Lustre: 183859:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5812.377858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 5816.852640] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5822.268892] Lustre: 184946:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5831.629414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 5837.177329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:909 to 0x280000401:929)
[ 5837.189646] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:161)
[ 5838.124387] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5850.384574] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 5852.692902] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:841 to 0x2c0000401:865)
[ 5856.679358] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5864.280702] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5867.367163] Lustre: 187183:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5918.635934] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 21:32:57 (1767321177)
[ 5961.940269] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 21:33:40 (1767321220)
[ 7307.190671] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 21:56:06 (1767322566)
[ 7315.424903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7315.428029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7315.438655] Lustre: Skipped 5 previous similar messages
[ 7315.447066] Lustre: Skipped 2 previous similar messages
[ 7316.340928] LustreError: 192037:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 7316.343024] LustreError: 192037:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages
[ 7316.553274] Lustre: server umount lustre-MDT0000 complete
[ 7319.345799] LustreError: 184419:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767322578 with bad export cookie 16782916779437526897
[ 7319.346646] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7319.353854] LustreError: 184419:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 7319.447912] LustreError: 192238:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup
[ 7319.453690] LustreError: 192238:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 7319.613248] Lustre: server umount lustre-MDT0001 complete
[ 7335.391208] LustreError: 192583:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup
[ 7335.399058] LustreError: 192583:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 7335.564783] Lustre: server umount lustre-OST0000 complete
[ 7336.799125] Lustre: 6804:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767322579/real 1767322579]  req@ffffa07407c38380 x1853164503360384/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767322595 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7336.836735] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7338.690976] Lustre: server umount lustre-OST0001 complete
[ 7356.523088] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local
[ 7370.603277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7370.962129] LustreError: 194583:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7370.979801] LustreError: 194583:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages
[ 7371.068906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7371.076785] Lustre: Skipped 3 previous similar messages
[ 7374.514831] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7378.930445] Lustre: 195137:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7387.128516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 7387.505839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 7390.956916] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7395.455554] Lustre: 196223:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7402.642122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 7407.130498] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7412.084574] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:193)
[ 7415.396983] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 7418.155652] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5866 to 0x2c0000401:5889)
[ 7418.161643] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5932 to 0x280000401:5953)
[ 7419.918899] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7425.236813] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7427.674420] Lustre: 198458:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7451.598960] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 21:58:30 (1767322710)
[ 7476.331304] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 21:58:55 (1767322735)
[ 7495.560971] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 21:59:14 (1767322754)
[ 7515.695435] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 21:59:34 (1767322774)
[ 7535.739776] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 21:59:55 (1767322795)
[ 7597.165730] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 22:00:56 (1767322856)
[ 7622.562527] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 22:01:21 (1767322881)
[ 7676.580351] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 22:02:15 (1767322935)
[ 7681.569669] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7681.573693] Lustre: Skipped 2 previous similar messages
[ 7682.578350] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7682.584757] Lustre: Skipped 51 previous similar messages
[ 7684.624650] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7684.628955] Lustre: Skipped 105 previous similar messages
[ 7688.624210] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7688.625902] Lustre: Skipped 198 previous similar messages
[ 7696.643134] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7696.644955] Lustre: Skipped 410 previous similar messages
[ 7712.664568] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7712.671249] Lustre: Skipped 855 previous similar messages
[ 7744.679363] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7744.681467] Lustre: Skipped 1837 previous similar messages
[ 7808.709398] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7808.713500] Lustre: Skipped 4077 previous similar messages
[ 8078.815792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 8078.818888] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8078.821102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 8078.821109] Lustre: Skipped 1 previous similar message
[ 8078.852885] Lustre: Skipped 3 previous similar messages
[ 8082.484708] LustreError: 210766:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 8082.498384] LustreError: 210766:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages
[ 8082.654595] Lustre: server umount lustre-MDT0000 complete
[ 8083.939322] LustreError: 194582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8083.946894] LustreError: 194582:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages
[ 8089.249230] LustreError: 203948:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323348 with bad export cookie 16782916779439299759
[ 8089.257175] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8089.265983] LustreError: 203948:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 8094.176251] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8094.177544] LustreError: 194578:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8094.177872] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 8094.177877] Lustre: Skipped 3 previous similar messages
[ 8094.185517] Lustre: Skipped 1 previous similar message
[ 8094.199576] LustreError: 194578:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages
[ 8095.473473] LustreError: 211222:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup
[ 8095.479672] LustreError: 211222:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 8095.597495] Lustre: server umount lustre-MDT0001 complete
[ 8103.414621] LustreError: 211818:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup
[ 8103.422426] LustreError: 211818:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 8103.501072] Lustre: server umount lustre-OST0000 complete
[ 8109.399306] LustreError: 212268:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup
[ 8109.402621] LustreError: 212268:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages
[ 8109.499826] Lustre: server umount lustre-OST0001 complete
[ 8119.133326] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid
[ 8124.946353] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local
[ 8140.938056] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 8151.260562] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 8158.265438] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 8166.228035] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 8178.633881] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local
[ 8191.544766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 8191.591567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8191.779384] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 8191.810871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 8191.903898] Lustre: lustre-MDT0000: new disk, initializing
[ 8191.977862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8191.980627] Lustre: Skipped 2 previous similar messages
[ 8191.998974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 8194.981602] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8200.112680] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 8208.256102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 8208.319404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 8208.370401] Lustre: 217418:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17
[ 8208.398273] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
[ 8208.402615] Lustre: Skipped 1 previous similar message
[ 8208.478367] Lustre: lustre-MDT0001: new disk, initializing
[ 8208.539598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 8208.569287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt
[ 8208.579951] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt]
[ 8211.258722] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8216.624372] Lustre: 219336:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 8225.340659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 8225.387837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 8225.506383] Lustre: lustre-OST0000: new disk, initializing
[ 8225.509086] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 8225.550593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 8226.574515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost
[ 8226.587697] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost]
[ 8226.623285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401
[ 8229.137993] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8240.628770] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 8240.676461] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 8240.739486] Lustre: lustre-OST0001: new disk, initializing
[ 8240.742520] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 8242.312559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost
[ 8242.315837] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost]
[ 8242.385133] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401
[ 8245.048569] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8252.863090] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8255.701452] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 8275.887659] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 22:12:15 (1767323535)
[ 8279.853368] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 22:12:19 (1767323539)
[ 8306.365473] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 22:12:45 (1767323565)
[ 8341.988174] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 22:13:21 (1767323601)
[ 8354.904636] Lustre: DEBUG MARKER: rename directory return 255
[ 8381.867848] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 22:14:01 (1767323641)
[ 8393.488824] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 22:14:12 (1767323652)
[ 8413.688399] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 22:14:32 (1767323672)
[ 8458.439945] LustreError: 217427:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 8473.583696] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 22:15:32 (1767323732)
[ 8488.178948] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 22:15:47 (1767323747)
[ 8504.332321] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 22:16:03 (1767323763)
[ 8576.049380] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 22:17:15 (1767323835)
[ 8580.577843] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8580.586566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 8580.598132] Lustre: Skipped 4 previous similar messages
[ 8584.072767] LustreError: 235091:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup
[ 8584.080758] LustreError: 235091:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 8584.155834] Lustre: server umount lustre-MDT0000 complete
[ 8585.704098] LustreError: 218888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8585.718459] LustreError: 218888:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages
[ 8586.348670] LustreError: 217409:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323845 with bad export cookie 16782916779439671907
[ 8586.349509] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8586.354638] LustreError: 217409:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 8586.570625] Lustre: server umount lustre-MDT0001 complete
[ 8600.076127] LustreError: 235637:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup
[ 8600.080217] LustreError: 235637:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages
[ 8600.119295] Lustre: server umount lustre-OST0000 complete
[ 8612.568611] Lustre: server umount lustre-OST0001 complete
[ 8628.881538] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local
[ 8638.598453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8638.905345] LustreError: 238031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8638.959890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8638.964320] Lustre: Skipped 1 previous similar message
[ 8638.969315] LustreError: lustre-MDT0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 8641.664970] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8645.208488] Lustre: 238586:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 8651.776872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 8652.017529] LustreError: lustre-MDT0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 8654.723367] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8658.081743] Lustre: 239674:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 8664.479278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 8664.737603] LustreError: lustre-OST0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 8666.619904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:65)
[ 8668.445870] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8676.982455] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 8677.121484] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 8677.124907] Lustre: Skipped 2 previous similar messages
[ 8677.131602] LustreError: lustre-OST0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 8678.189956] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:20 to 0x2c0000401:65)
[ 8681.205111] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8685.989615] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8687.987515] Lustre: 241910:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 8696.769328] LustreError: 238027:0:(osd_handler.c:3458:osd_quota_transfer()) lustre-MDT0000: quota transfer failed. Is project enforcement enabled on the ldiskfs filesystem? rc = -95
[ 8698.853670] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8698.858348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 8698.867289] Lustre: Skipped 5 previous similar messages
[ 8698.870586] Lustre: Skipped 1 previous similar message
[ 8703.455738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 8703.459637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 8703.462498] Lustre: Skipped 1 previous similar message
[ 8704.795180] LustreError: 242303:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 8704.799766] LustreError: 242303:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 8704.845332] Lustre: server umount lustre-MDT0000 complete
[ 8707.056730] LustreError: 238012:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323966 with bad export cookie 16782916779439720634
[ 8707.061186] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8707.070601] LustreError: 238012:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 8707.238991] Lustre: server umount lustre-MDT0001 complete
[ 8721.299344] Lustre: server umount lustre-OST0000 complete
[ 8733.854126] Lustre: server umount lustre-OST0001 complete
[ 8750.183430] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local
[ 8759.043055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8759.266772] LustreError: 245243:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8759.279176] LustreError: 245243:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages
[ 8759.321951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8761.830504] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8765.102497] Lustre: 245798:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 8770.797810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 8772.851142] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8781.683161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 8783.909508] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97)
[ 8785.723567] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8793.221781] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 8794.819478] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:20 to 0x2c0000401:97)
[ 8796.777409] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8801.262122] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8803.057322] Lustre: 249123:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 8803.062969] Lustre: 249123:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message
[ 8826.311558] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 22:21:25 (1767324085)
[ 8857.515545] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[ 8858.630642] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 22:21:57 (1767324117)
[ 8870.389798] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[ 8871.272933] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 22:22:10 (1767324130)
[ 8891.027993] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[ 8891.918406] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 22:22:31 (1767324151)
[ 8908.690852] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 22:22:48 (1767324168)
[ 8914.332959] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[ 8916.620195] Lustre: 255240:0:(qsd_reint.c:241:qsd_reint_index()) lustre-OST0000: index version for fid [0x200000005:0x1007:0x0] is 0, but index isn't empty (1)
[ 8918.740418] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 8921.393204] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 8923.127883] Lustre: DEBUG MARKER: Write...
[ 8933.330677] LustreError: 256458:0:(mgs_handler.c:1067:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[ 8936.863491] Lustre: DEBUG MARKER: Write...
[ 8941.988898] Lustre: DEBUG MARKER: Write...
[ 8984.941788] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 22:24:04 (1767324244)
[ 9001.500743] LustreError: 260224:0:(mgs_handler.c:1067:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[ 9028.847733] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 22:24:48 (1767324288)
[ 9043.479426] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[ 9044.326488] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[ 9094.143227] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 22:25:53 (1767324353)
[ 9122.807418] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 22:26:22 (1767324382)
[ 9143.851466] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 22:26:43 (1767324403)
[ 9150.156108] Lustre: DEBUG MARKER: User quota (block hardlimit:100 MB)
[ 9166.381878] Lustre: DEBUG MARKER: Write...
[ 9167.640240] Lustre: DEBUG MARKER: Write out of block quota ...
[ 9227.218425] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 22:28:06 (1767324486)
[ 9233.123550] Lustre: DEBUG MARKER: User quota (block hardlimit:1000 MB)
[ 9251.016610] LustreError: 250306:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool2 id:60000 enforced:1 hard:10240 soft:0 granted:10240 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 9288.785272] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 22:29:08 (1767324548)
[ 9294.850936] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[ 9306.626699] Lustre: DEBUG MARKER: Write...
[ 9307.682877] Lustre: DEBUG MARKER: Write out of block quota ...
[ 9343.354276] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 22:30:02 (1767324602)
[ 9360.616319] Lustre: DEBUG MARKER: set to use default quota
[ 9361.357597] Lustre: DEBUG MARKER: set default quota
[ 9362.189237] Lustre: DEBUG MARKER: get default quota
[ 9364.189569] Lustre: DEBUG MARKER: Test not out of quota
[ 9365.578229] Lustre: DEBUG MARKER: Test out of quota
[ 9369.546765] Lustre: DEBUG MARKER: Increase default quota
[ 9380.106164] Lustre: DEBUG MARKER: Set quota to override default quota
[ 9380.135723] LustreError: 251420:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45056 time:1767929439 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 9384.799760] Lustre: DEBUG MARKER: Set to use default quota again
[ 9397.566706] Lustre: DEBUG MARKER: Cleanup
[ 9436.785972] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 22:31:36 (1767324696)
[ 9450.501934] Lustre: DEBUG MARKER: set default quota for qpool1
[ 9451.623811] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[ 9481.522330] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 22:32:20 (1767324740)
[ 9527.074438] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 22:33:06 (1767324786)
[ 9557.830877] Lustre: DEBUG MARKER: Write...
[ 9559.440902] Lustre: DEBUG MARKER: Write out of block quota ...
[ 9621.081564] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 22:34:40 (1767324880)
[ 9641.525152] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 22:35:00 (1767324900)
[ 9645.461520] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 22:35:04 (1767324904)
[ 9672.330545] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 22:35:31 (1767324931)
[ 9697.204806] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 22:35:56 (1767324956)
[ 9708.859681] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 22:36:08 (1767324968)
[ 9724.559546] Lustre: *** cfs_fail_loc=a06, val=0***
[ 9724.560789] Lustre: Skipped 459 previous similar messages
[ 9724.680153] LustreError: 245256:0:(qmt_lock.c:446:qmt_lvbo_update()) $$$ failed to release quota space on glimpse 0!=2048 : rc = -11
                 qmt:lustre-QMT0000 pool:dt-0x0 id:60000 enforced:1 hard:102400 soft:0 granted:23552 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no
[ 9728.890086] Lustre: Failing over lustre-OST0001
[ 9728.931294] LustreError: 294763:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup
[ 9728.933677] LustreError: 294763:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages
[ 9728.967294] Lustre: server umount lustre-OST0001 complete
[ 9732.064718] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 9732.067712] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9732.072150] Lustre: Skipped 1 previous similar message
[ 9732.074220] LustreError: 247435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9732.079805] LustreError: 247435:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages
[ 9733.094610] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107
[ 9735.503368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 9735.663973] Lustre: lustre-OST0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 9735.677028] Lustre: lustre-OST0001: in recovery but waiting for the first client to connect
[ 9737.143581] Lustre: lustre-OST0001: Will be in recovery for at least 1:00, or until 3 clients reconnect
[ 9737.279685] Lustre: lustre-OST0001: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted.
[ 9737.279838] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo)
[ 9737.289875] Lustre: Skipped 8 previous similar messages
[ 9738.369224] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9771.372665] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 22:37:10 (1767325030)
[ 9776.114440] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[ 9781.059955] LustreError: 297948:0:(qmt_pool.c:1295:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[ 9784.804659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 9784.807386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9784.813943] Lustre: Skipped 1 previous similar message
[ 9784.818928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9784.821722] Lustre: Skipped 3 previous similar messages
[ 9786.848959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9786.852724] Lustre: Skipped 4 previous similar messages
[ 9790.226547] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.30@tcp (stopping)
[ 9791.111159] LustreError: 297948:0:(qmt_pool.c:1295:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[ 9791.161078] LustreError: 298098:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 9791.163858] LustreError: 298098:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages
[ 9791.223252] Lustre: server umount lustre-MDT0000 complete
[ 9791.967841] LustreError: 246920:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9791.973605] LustreError: 246920:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages
[ 9797.079283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9797.168762] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9797.289582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9797.292696] Lustre: Skipped 3 previous similar messages
[ 9797.334569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:112 to 0x280000401:129)
[ 9797.334672] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:113 to 0x2c0000401:129)
[ 9799.443399] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9802.722695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo)
[ 9802.727066] Lustre: Skipped 1 previous similar message
[ 9815.205744] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 22:37:54 (1767325074)
[ 9824.974311] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 22:38:04 (1767325084)
[ 9834.294743] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 22:38:13 (1767325093)
[ 9857.638381] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9857.640259] Lustre: Skipped 8 previous similar messages
[ 9857.642835] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9899.181614] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 22:39:18 (1767325158)
[ 9916.017984] LustreError: 250306:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:3072 soft:0 granted:3072 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 9951.138246] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 22:40:10 (1767325210)
[ 9986.436282] LustreError: 251420:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:16384 time:1767930045 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[10064.077219] LustreError: 246920:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:16384 time:1767930123 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[10148.100203] LustreError: 258748:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:1000 enforced:1 hard:2048 soft:2048 granted:16385 time:1767930207 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[10208.500182] Lustre: DEBUG MARKER: == sanity-quota test 87: lfs quota -a should print default quota setting ========================================================== 22:44:27 (1767325467)
[10211.648097] Lustre: *** cfs_fail_loc=a09, val=0***
[10211.649418] Lustre: Skipped 1 previous similar message
[10237.924607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[10237.928993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10237.933705] Lustre: Skipped 3 previous similar messages
[10237.937368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10244.425769] LustreError: 308946:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[10244.429033] LustreError: 308946:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[10244.479093] Lustre: server umount lustre-MDT0000 complete
[10245.906634] LustreError: 251420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[10245.914050] LustreError: 251420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages
[10250.091115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[10250.204158] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10250.342133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10250.396234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:132 to 0x280000401:161)
[10250.401383] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:113 to 0x2c0000401:161)
[10252.527269] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10255.329257] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[10255.338251] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[10255.341791] Lustre: Skipped 3 previous similar messages
[10260.085457] Lustre: DEBUG MARKER: == sanity-quota test 88: Writing over quota should not hang ========================================================== 22:45:19 (1767325519)
[10260.883192] Lustre: DEBUG MARKER: SKIP: sanity-quota test_88 require client with >4k pages
[10261.763471] Lustre: DEBUG MARKER: == sanity-quota test 89: Show default quota with squash_uid ========================================================== 22:45:21 (1767325521)
[10271.864669] Lustre: DEBUG MARKER: == sanity-quota test 90a: lfs quota should work without mount point ========================================================== 22:45:31 (1767325531)
[10285.662342] Lustre: DEBUG MARKER: == sanity-quota test 90b: lfs quota should work with multiple mount points ========================================================== 22:45:45 (1767325545)
[10300.288393] Lustre: DEBUG MARKER: == sanity-quota test 91: new quota index files in quota_master ========================================================== 22:45:59 (1767325559)
[10316.768608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10316.769954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10316.770197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[10316.773772] Lustre: Skipped 5 previous similar messages
[10316.776775] Lustre: Skipped 8 previous similar messages
[10319.299233] LustreError: 313663:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[10319.303575] LustreError: 313663:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[10319.344786] Lustre: server umount lustre-MDT0000 complete
[10321.889703] LustreError: 245239:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10321.897887] LustreError: 245239:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages
[10323.186873] LustreError: 245224:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325582 with bad export cookie 16782916779440824835
[10323.190545] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10323.192474] LustreError: 245224:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages
[10329.409973] Lustre: server umount lustre-MDT0001 complete
[10340.485273] Lustre: server umount lustre-OST0000 complete
[10350.702646] Lustre: server umount lustre-OST0001 complete
[10357.692474] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid
[10361.534395] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local
[10371.983618] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[10378.094434] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[10382.909190] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[10388.216929] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[10393.441079] loop: module loaded
[10395.455364] Lustre: DEBUG MARKER: sanity-quota test_91: @@@@@@ FAIL: Cannot start mds1