[ 5863.025912] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5868.678300] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5871.486954] LustreError: 125064:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5872.458888] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5874.246427] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5876.684792] Lustre: Failing over lustre-MDT0000
[ 5876.714945] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5877.054131] Lustre: server umount lustre-MDT0000 complete
[ 5895.618422] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5895.726865] LustreError: 125717:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5896.207237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5896.283172] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5897.203020] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5897.307420] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5897.419965] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5897.468186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:590 to 0x240000400:609)
[ 5897.468644] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:586 to 0x280000400:609)
[ 5898.464168] Lustre: 3304:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789602114/real 1789602114]  req@ffffa067839bf800 x1876527753187072/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789602130 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5898.506548] Lustre: 3304:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5901.549141] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5902.304784] Lustre: 3306:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789602119/real 1789602119]  req@ffffa068c1534380 x1876527753187328/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789602135 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5907.424124] Lustre: 3303:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789602124/real 1789602124]  req@ffffa067839be300 x1876527753187584/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789602140 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5910.290924] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5911.540525] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5916.258637] Lustre: DEBUG MARKER: (dd_pid=117455, time=0, timeout=600)
[ 5959.575807] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5965.744527] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5969.121810] LustreError: 127973:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5970.161567] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5971.602366] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5973.525878] Lustre: Failing over lustre-MDT0000
[ 5973.997227] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5974.005237] Lustre: Skipped 1 previous similar message
[ 5974.014747] LustreError: 125717:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5974.088782] Lustre: server umount lustre-MDT0000 complete
[ 5990.945039] Lustre: 3304:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789602207/real 1789602207]  req@ffffa067c9e06680 x1876527753222784/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1789602223 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5991.631728] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5992.178389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5992.240471] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5993.007761] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5993.225883] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5993.304466] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:611 to 0x240000400:641)
[ 5993.307383] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:586 to 0x280000400:641)
[ 5994.212108] LustreError: 3302:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-lwp-OST0001: namespace resource [0x200000006:0x20000:0x0].0x0 (ffffa067c28eef00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5994.245423] LustreError: 3302:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 5 previous similar messages
[ 5994.254172] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5994.260630] Lustre: Skipped 1 previous similar message
[ 5997.285360] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6000.096458] Lustre: 3303:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789602217/real 1789602217]  req@ffffa067c9e07b80 x1876527753223424/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789602233 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6000.141938] Lustre: 3303:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 6006.742689] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6008.410856] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6018.748128] Lustre: DEBUG MARKER: (dd_pid=119889, time=4, timeout=600)
[ 6077.323657] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 19:45:07 (1789602307)
[ 6098.475966] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6100.273588] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6106.232995] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6108.103286] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6144.529172] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 19:46:14 (1789602374)
[ 6168.341345] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 6184.276690] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 6186.303180] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 6188.529703] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 6192.568385] Lustre: DEBUG MARKER: Set quota for 1 times
[ 6196.513427] Lustre: DEBUG MARKER: Set quota for 2 times
[ 6200.481656] Lustre: DEBUG MARKER: Set quota for 3 times
[ 6204.519531] Lustre: DEBUG MARKER: Set quota for 4 times
[ 6208.275260] Lustre: DEBUG MARKER: Set quota for 5 times
[ 6212.679867] Lustre: DEBUG MARKER: Set quota for 6 times
[ 6216.698667] Lustre: DEBUG MARKER: Set quota for 7 times
[ 6220.129805] Lustre: DEBUG MARKER: Set quota for 8 times
[ 6260.109340] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 19:48:10 (1789602490)
[ 6271.981063] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6271.982750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6272.000553] Lustre: Skipped 2 previous similar messages
[ 6276.656504] Lustre: server umount lustre-MDT0000 complete
[ 6281.106269] LustreError: 5824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789602514 with bad export cookie 9456992672550485469
[ 6281.106796] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6281.117941] LustreError: 5824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 6281.240442] Lustre: server umount lustre-OST0000 complete
[ 6285.447517] Lustre: server umount lustre-OST0001 complete
[ 6298.877936] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local
[ 6308.891504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6313.762769] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6316.575226] Lustre: 137700:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6322.728889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6328.963328] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6334.948847] LustreError: 138068:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6335.002063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:643 to 0x240000400:673)
[ 6338.884256] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6344.193990] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:644 to 0x280000400:673)
[ 6347.049341] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6356.345615] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6360.991872] Lustre: 139622:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6377.776868] Lustre: server umount lustre-MDT0000 complete
[ 6382.468264] LustreError: 137245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789602615 with bad export cookie 9456992672550494716
[ 6382.477889] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6392.878938] Lustre: server umount lustre-OST0000 complete
[ 6395.361833] Lustre: 3303:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789602612/real 1789602612]  req@ffffa06891989880 x1876527753382784/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789602628 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6395.398411] Lustre: 3303:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 6397.632946] Lustre: server umount lustre-OST0001 complete
[ 6415.140650] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local
[ 6426.597366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6432.054250] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6435.570725] Lustre: 142280:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6442.747960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6449.825845] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6451.914704] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:643 to 0x240000400:705)
[ 6464.006637] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:644 to 0x280000400:705)
[ 6466.045464] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6473.956819] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6477.754617] Lustre: 144187:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6488.115477] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 19:51:58 (1789602718)
[ 6489.437598] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6491.282753] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 19:52:01 (1789602721)
[ 6544.981645] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 19:52:55 (1789602775)
[ 6589.449476] Lustre: DEBUG MARKER: Write...
[ 6592.286314] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6644.550677] LustreError: 3304:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-OST0001: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 6647.737454] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 19:54:38 (1789602878)
[ 6654.224436] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 19:54:44 (1789602884)
[ 6664.212255] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 19:54:54 (1789602894)
[ 6672.759987] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 19:55:03 (1789602903)
[ 6681.725612] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 19:55:12 (1789602912)
[ 6745.901504] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6893.795854] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 7089.118552] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 20:01:59 (1789603319)
[ 7161.525512] Lustre: DEBUG MARKER: Restart...
[ 7168.093084] Lustre: server umount lustre-MDT0000 complete
[ 7172.628764] LustreError: 141834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789603405 with bad export cookie 9456992672550496256
[ 7172.648727] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7182.929271] Lustre: server umount lustre-OST0000 complete
[ 7186.484555] LustreError: 3305:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0001 qtype:usr lqe: ffffa0689fc98780 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2377 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 7196.940082] Lustre: server umount lustre-OST0001 complete
[ 7214.366576] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local
[ 7225.056467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7225.060118] Lustre: Skipped 1 previous similar message
[ 7230.486044] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7233.725470] Lustre: 162987:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7241.171168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7249.041154] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7249.314552] LustreError: 163358:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7249.333191] LustreError: 163358:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages
[ 7249.341246] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:714 to 0x240000400:737)
[ 7258.999682] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7264.273452] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:714 to 0x280000400:737)
[ 7266.215220] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7274.460504] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7278.666593] Lustre: 164915:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7343.556755] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 20:06:13 (1789603573)
[ 7422.928833] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 20:07:33 (1789603653)
[ 9037.724907] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 20:34:27 (1789605267)
[ 9056.237961] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9056.265516] Lustre: Skipped 1 previous similar message
[ 9056.276453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9056.288194] Lustre: Skipped 2 previous similar messages
[ 9059.628907] Lustre: server umount lustre-MDT0000 complete
[ 9064.281793] LustreError: 162544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789605297 with bad export cookie 9456992672550502745
[ 9064.289172] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9064.299687] LustreError: 162544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 9064.453857] Lustre: server umount lustre-OST0000 complete
[ 9068.825931] Lustre: server umount lustre-OST0001 complete
[ 9084.141403] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local
[ 9102.356117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9108.267835] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9111.519606] Lustre: 173211:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9119.558696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9126.636731] LustreError: 173581:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9126.654430] LustreError: 173581:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message
[ 9126.687324] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5739 to 0x240000400:5761)
[ 9127.031421] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9132.000715] LustreError: 173581:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9136.497439] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9141.781629] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5739 to 0x280000400:5761)
[ 9142.978964] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9151.740310] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9156.451077] Lustre: 175142:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9196.048699] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 20:37:06 (1789605426)
[ 9242.972295] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 20:37:53 (1789605473)
[ 9293.419258] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 20:38:43 (1789605523)
[ 9295.435718] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9297.460885] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 20:38:47 (1789605527)
[ 9299.318803] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9301.283820] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 20:38:51 (1789605531)
[ 9387.691513] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 20:40:18 (1789605618)
[ 9439.167455] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 20:41:09 (1789605669)
[ 9527.900609] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 20:42:38 (1789605758)
[ 9540.046754] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9540.053096] Lustre: Skipped 2 previous similar messages
[ 9541.109827] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9541.111559] Lustre: Skipped 31 previous similar messages
[ 9543.120440] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9543.123562] Lustre: Skipped 103 previous similar messages
[ 9547.121810] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9547.124608] Lustre: Skipped 207 previous similar messages
[ 9555.140283] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9555.143723] Lustre: Skipped 343 previous similar messages
[ 9571.167072] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9571.169531] Lustre: Skipped 811 previous similar messages
[ 9603.188705] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9603.190138] Lustre: Skipped 1493 previous similar messages
[ 9667.213367] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9667.217492] Lustre: Skipped 3295 previous similar messages
[ 9930.035970] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9930.041289] Lustre: Skipped 1711 previous similar messages
[ 9930.045739] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9930.061570] Lustre: Skipped 1 previous similar message
[10078.524462] Lustre: *** cfs_fail_loc=a08, val=0***
[10078.528103] Lustre: Skipped 13 previous similar messages
[10252.769714] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10252.774333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10252.780515] Lustre: Skipped 2 previous similar messages
[10252.800915] Lustre: Skipped 1 previous similar message
[10257.226154] Lustre: server umount lustre-MDT0000 complete
[10262.297972] LustreError: 172763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789606495 with bad export cookie 9456992672552278155
[10262.322146] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10262.876429] Lustre: server umount lustre-OST0000 complete
[10267.362961] Lustre: server umount lustre-OST0001 complete
[10275.020548] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid
[10283.176697] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local
[10293.517392]  vdc: vdc1 vdc9
[10293.541292]  vdc: vdc1 vdc9
[10293.587668]  vdc: vdc1 vdc9
[10306.037298]  vde: vde1 vde9
[10306.074053]  vde: vde1 vde9
[10316.953296]  vdf: vdf1 vdf9
[10334.899313] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local
[10344.644220] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[10344.928722] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[10344.993936] Lustre: lustre-MDT0000: new disk, initializing
[10345.481315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10345.592055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[10350.535537] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10355.914405] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[10362.623051] Lustre: lustre-OST0000: new disk, initializing
[10362.628757] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[10362.637371] Lustre: Skipped 1 previous similar message
[10362.818394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10364.213808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[10364.226926] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[10364.351243] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[10369.278462] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10380.268905] Lustre: lustre-OST0001: new disk, initializing
[10380.274096] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[10380.370488] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10382.233959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[10382.244183] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[10382.375077] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[10388.253333] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10397.640813] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10401.687870] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[10439.452409] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 20:57:49 (1789606669)
[10446.364523] Lustre: DEBUG MARKER: == sanity-quota test 49c: lfs quota long options don't consume an extra argument ========================================================== 20:57:57 (1789606677)
[10452.840788] Lustre: DEBUG MARKER: == sanity-quota test 49d: lfs quota -d and --delimiter both work ========================================================== 20:58:03 (1789606683)
[10459.647531] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 20:58:10 (1789606690)
[10508.418551] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 20:58:58 (1789606738)
[10572.700292] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 21:00:03 (1789606803)
[10604.655569] Lustre: DEBUG MARKER: rename directory return 255
[10653.094469] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 21:01:23 (1789606883)
[10684.761548] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 21:01:55 (1789606915)
[10726.405329] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 21:02:36 (1789606956)
[10881.270046] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 21:05:11 (1789607111)
[10920.085474] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 21:05:50 (1789607150)
[10970.700757] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 21:06:41 (1789607201)
[11115.296995] LustreError: 189982:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:1001 enforced:1 hard:204800 soft:0 granted:204800 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11156.649307] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 21:09:46 (1789607386)
[11158.752175] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[11160.808335] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 21:09:51 (1789607391)
[11224.409320] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[11226.504401] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 21:10:56 (1789607456)
[11258.023628] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[11259.954287] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 21:11:30 (1789607490)
[11313.085521] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[11314.747390] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 21:12:25 (1789607545)
[11371.447664] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 21:13:21 (1789607601)
[11373.300896] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[11375.658411] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 21:13:25 (1789607605)
[11406.471703] LustreError: 213475:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[11454.291993] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 21:14:44 (1789607684)
[11480.870900] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[11482.509594] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[11670.542575] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 21:18:20 (1789607900)
[11725.919604] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 21:19:16 (1789607956)
[11763.703272] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 21:19:53 (1789607993)
[11765.710640] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[11767.812863] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 21:19:58 (1789607998)
[11769.961743] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[11772.738843] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 21:20:02 (1789608002)
[11791.621573] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[11811.301217] Lustre: DEBUG MARKER: Write...
[11815.177868] Lustre: DEBUG MARKER: Write out of block quota ...
[11879.913383] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 21:21:50 (1789608110)
[11910.104276] Lustre: DEBUG MARKER: set to use default quota
[11912.182111] Lustre: DEBUG MARKER: set default quota
[11913.536090] Lustre: DEBUG MARKER: get default quota
[11919.926499] Lustre: DEBUG MARKER: Test not out of quota
[11926.629444] Lustre: DEBUG MARKER: Test out of quota
[11943.030884] Lustre: DEBUG MARKER: Increase default quota
[11981.235575] Lustre: DEBUG MARKER: Set quota to override default quota
[11981.280588] LustreError: 189978:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1790213014 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11993.458803] Lustre: DEBUG MARKER: Set to use default quota again
[12019.638992] Lustre: DEBUG MARKER: Cleanup
[12100.805715] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 21:25:31 (1789608331)
[12127.811531] Lustre: DEBUG MARKER: set default quota for qpool1
[12130.047789] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[12182.306508] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 21:26:52 (1789608412)
[12269.742813] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 21:28:19 (1789608499)
[12353.355452] Lustre: DEBUG MARKER: Write...
[12357.023708] Lustre: DEBUG MARKER: Write out of block quota ...
[12495.314592] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 21:32:05 (1789608725)
[12540.524326] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 21:32:50 (1789608770)
[12549.270830] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 21:32:59 (1789608779)
[12551.607382] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A fallocate not supported
[12553.539591] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 21:33:04 (1789608784)
[12555.847803] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a fallocate not supported
[12557.715798] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 21:33:08 (1789608788)
[12577.798685] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 21:33:28 (1789608808)
[12579.273824] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[12581.082400] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 21:33:31 (1789608811)
[12600.402795] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[12607.352105] LustreError: 238800:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[12612.577444] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[12612.601939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[12612.609362] Lustre: Skipped 1 previous similar message
[12613.142569] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.31@tcp (stopping)
[12617.376631] LustreError: 238800:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[12617.743777] Lustre: server umount lustre-MDT0000 complete
[12626.504572] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[12627.140624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[12627.336592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[12627.340615] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[12632.365434] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[12633.069602] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[12633.079953] Lustre: Skipped 1 previous similar message
[12669.410699] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 21:34:59 (1789608899)
[12704.825475] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 21:35:35 (1789608935)
[12738.748655] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 21:36:09 (1789608969)
[12795.114808] Lustre: *** cfs_fail_loc=a08, val=0***
[12795.131696] Lustre: Skipped 28 previous similar messages
[12795.151875] Lustre: *** cfs_fail_loc=a08, val=0***
[12795.157933] Lustre: Skipped 15 previous similar messages
[12910.200253] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 21:39:00 (1789609140)
[13003.637217] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 21:40:34 (1789609234)
[13080.638638] LustreError: 239379:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1790214113 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13241.490373] LustreError: 239379:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1790214274 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13250.778743] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed