[ 5151.025067] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5153.117482] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5154.286855] LustreError: 126830:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5154.671069] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5155.244235] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5155.955792] Lustre: Failing over lustre-MDT0000
[ 5156.362349] Lustre: server umount lustre-MDT0000 complete
[ 5169.188563] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5169.305991] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5169.317375] Lustre: Skipped 1 previous similar message
[ 5169.380694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5169.412844] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5169.556278] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5169.577125] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5169.598587] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1000 to 0x280000400:1025)
[ 5169.598866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1002 to 0x240000400:1025)
[ 5170.950739] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5173.408084] Lustre: 3317:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789307591/real 1789307591]  req@ffff91057a35df80 x1876219669676160/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789307607 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5173.418763] Lustre: 3317:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5174.753983] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5174.919793] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5175.464116] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5177.889157] Lustre: 3318:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789307596/real 1789307596]  req@ffff91057a35f480 x1876219669676672/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789307612 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5177.922265] Lustre: DEBUG MARKER: (dd_pid=118347, time=0, timeout=600)
[ 5183.008136] Lustre: 3316:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789307601/real 1789307601]  req@ffff910442985f80 x1876219669677056/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789307617 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5183.016088] Lustre: 3316:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5208.149553] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5210.023684] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5211.095119] LustreError: 129866:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5211.431128] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5212.038124] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5212.802467] Lustre: Failing over lustre-MDT0000
[ 5213.130353] Lustre: server umount lustre-MDT0000 complete
[ 5226.042809] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5226.198627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5226.223298] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5227.752232] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5229.216172] Lustre: 3316:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789307647/real 1789307647]  req@ffff91056e320e00 x1876219669704704/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1789307663 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5229.223556] Lustre: 3316:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5229.763563] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5229.799607] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5229.815367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1027 to 0x240000400:1057)
[ 5229.815382] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1000 to 0x280000400:1057)
[ 5231.613331] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5232.124474] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5237.728242] Lustre: 3318:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789307655/real 1789307655]  req@ffff91057ff31c00 x1876219669705472/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789307671 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5237.736794] Lustre: 3318:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5238.736156] Lustre: DEBUG MARKER: (dd_pid=120899, time=4, timeout=600)
[ 5281.597252] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 09:55:15 (1789307715)
[ 5293.180951] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5293.764421] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5295.720117] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5296.314616] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5318.072398] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 09:55:51 (1789307751)
[ 5319.161779] LustreError: 3315:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 5319.171334] LustreError: 3315:0:(qsd_reint.c:633:qqi_reint_delayed()) Skipped 1 previous similar message
[ 5328.036584] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 5336.907060] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 5337.406834] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 5337.923370] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 5338.585281] Lustre: DEBUG MARKER: Set quota for 1 times
[ 5340.506992] Lustre: DEBUG MARKER: Set quota for 2 times
[ 5342.342640] Lustre: DEBUG MARKER: Set quota for 3 times
[ 5344.137190] Lustre: DEBUG MARKER: Set quota for 4 times
[ 5345.988885] Lustre: DEBUG MARKER: Set quota for 5 times
[ 5347.858551] Lustre: DEBUG MARKER: Set quota for 6 times
[ 5349.617455] Lustre: DEBUG MARKER: Set quota for 7 times
[ 5351.369448] Lustre: DEBUG MARKER: Set quota for 8 times
[ 5353.099886] Lustre: DEBUG MARKER: Set quota for 9 times
[ 5354.926396] Lustre: DEBUG MARKER: Set quota for 10 times
[ 5356.735647] Lustre: DEBUG MARKER: Set quota for 11 times
[ 5358.425462] Lustre: DEBUG MARKER: Set quota for 12 times
[ 5360.208768] Lustre: DEBUG MARKER: Set quota for 13 times
[ 5361.974250] Lustre: DEBUG MARKER: Set quota for 14 times
[ 5363.795586] Lustre: DEBUG MARKER: Set quota for 15 times
[ 5365.527079] Lustre: DEBUG MARKER: Set quota for 16 times
[ 5367.299633] Lustre: DEBUG MARKER: Set quota for 17 times
[ 5395.156382] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 09:57:09 (1789307829)
[ 5400.544495] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5400.544730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5400.548661] Lustre: Skipped 3 previous similar messages
[ 5400.552137] Lustre: Skipped 1 previous similar message
[ 5402.289834] Lustre: server umount lustre-MDT0000 complete
[ 5403.886839] LustreError: 5843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307838 with bad export cookie 6881733252440681268
[ 5403.887938] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5403.892282] LustreError: 5843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 5403.931703] Lustre: server umount lustre-OST0000 complete
[ 5405.460949] Lustre: server umount lustre-OST0001 complete
[ 5410.642128] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local
[ 5414.412656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5416.061136] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5417.169904] Lustre: 140596:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5419.523490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5421.829983] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5424.615909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1060 to 0x240000400:1089)
[ 5424.663833] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5426.715552] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5429.742475] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1059 to 0x280000400:1089)
[ 5430.094823] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5431.468020] Lustre: 142513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5439.971295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5439.974835] Lustre: Skipped 1 previous similar message
[ 5440.384107] Lustre: server umount lustre-MDT0000 complete
[ 5442.022680] LustreError: 140147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307876 with bad export cookie 6881733252440704172
[ 5442.024093] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5442.026578] LustreError: 140147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 5442.055973] Lustre: server umount lustre-OST0000 complete
[ 5443.612472] Lustre: server umount lustre-OST0001 complete
[ 5448.890364] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local
[ 5452.627469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5454.093637] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5455.088750] Lustre: 145202:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5460.062964] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5462.753621] LustreError: 145577:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5462.767093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1060 to 0x240000400:1121)
[ 5465.351647] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5468.569675] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5468.651103] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1059 to 0x280000400:1121)
[ 5469.774074] Lustre: 147096:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5473.140346] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 09:58:26 (1789307906)
[ 5473.662001] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 5474.267552] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 09:58:28 (1789307908)
[ 5508.658089] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 09:59:02 (1789307942)
[ 5529.103255] Lustre: DEBUG MARKER: Write...
[ 5530.022367] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5565.027587] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 09:59:58 (1789307998)
[ 5566.979734] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 10:00:00 (1789308000)
[ 5571.261127] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 10:00:05 (1789308005)
[ 5574.712792] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 10:00:08 (1789308008)
[ 5577.987683] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 10:00:11 (1789308011)
[ 5621.562314] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 5705.102287] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 5832.034929] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 10:04:25 (1789308265)
[ 5868.609550] Lustre: DEBUG MARKER: Restart...
[ 5873.123060] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5873.130484] Lustre: Skipped 2 previous similar messages
[ 5873.134612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5873.138267] Lustre: Skipped 1 previous similar message
[ 5875.909758] Lustre: server umount lustre-MDT0000 complete
[ 5877.264933] LustreError: 144756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308311 with bad export cookie 6881733252440705705
[ 5877.266765] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5877.268845] LustreError: 144756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 5877.299807] Lustre: server umount lustre-OST0000 complete
[ 5878.684204] Lustre: server umount lustre-OST0001 complete
[ 5883.707859] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local
[ 5887.097937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5887.099720] Lustre: Skipped 2 previous similar messages
[ 5888.447272] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5889.387472] Lustre: 167213:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5893.457233] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5897.128615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1131 to 0x240000400:1153)
[ 5897.128834] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1129 to 0x280000400:1153)
[ 5898.100236] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5901.237931] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5902.397020] Lustre: 169112:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5932.877065] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 10:06:06 (1789308366)
[ 5983.154625] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 10:06:56 (1789308416)
[ 6428.597398] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 10:14:22 (1789308862)
[ 6438.722580] Lustre: server umount lustre-MDT0000 complete
[ 6440.109309] LustreError: 167600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308874 with bad export cookie 6881733252440712117
[ 6440.113286] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6450.694723] Lustre: server umount lustre-OST0000 complete
[ 6451.964656] LustreError: 3316:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0001 qtype:usr lqe: ffff91057fb0a780 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2384 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 6462.234717] Lustre: server umount lustre-OST0001 complete
[ 6467.196451] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local
[ 6470.659612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6470.663104] Lustre: Skipped 2 previous similar messages
[ 6472.039851] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6472.997350] Lustre: 177337:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6476.981143] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6480.745611] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6155 to 0x240000400:6177)
[ 6480.748289] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6155 to 0x280000400:6177)
[ 6481.624428] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6484.805171] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6510.736971] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 10:15:44 (1789308944)
[ 6540.957070] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 10:16:14 (1789308974)
[ 6572.798079] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 10:16:46 (1789309006)
[ 6573.286546] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 6573.855700] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 10:16:47 (1789309007)
[ 6574.398488] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 6574.924075] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 10:16:48 (1789309008)
[ 6629.217815] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 10:17:43 (1789309063)
[ 6656.659405] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 10:18:10 (1789309090)
[ 6705.936155] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 10:18:59 (1789309139)
[ 6712.985968] Lustre: *** cfs_fail_loc=a09, val=0***
[ 6712.987272] Lustre: Skipped 1 previous similar message
[ 6713.991469] Lustre: *** cfs_fail_loc=a09, val=0***
[ 6713.992531] Lustre: Skipped 185 previous similar messages
[ 6715.991829] Lustre: *** cfs_fail_loc=a09, val=0***
[ 6715.993861] Lustre: Skipped 250 previous similar messages
[ 6720.001279] Lustre: *** cfs_fail_loc=a09, val=0***
[ 6720.002941] Lustre: Skipped 396 previous similar messages
[ 6728.001381] Lustre: *** cfs_fail_loc=a09, val=0***
[ 6728.002892] Lustre: Skipped 817 previous similar messages
[ 6744.006063] Lustre: *** cfs_fail_loc=a09, val=0***
[ 6744.007650] Lustre: Skipped 793 previous similar messages
[ 6776.007054] Lustre: *** cfs_fail_loc=a09, val=0***
[ 6776.008779] Lustre: Skipped 2102 previous similar messages
[ 7193.511795] Lustre: *** cfs_fail_loc=a08, val=0***
[ 7193.523406] Lustre: *** cfs_fail_loc=a08, val=0***
[ 7193.526114] Lustre: Skipped 3455 previous similar messages
[ 7275.442255] Lustre: *** cfs_fail_loc=a08, val=0***
[ 7275.449782] Lustre: Skipped 15 previous similar messages
[ 7463.847926] Lustre: *** cfs_fail_loc=a08, val=0***
[ 7463.849897] Lustre: Skipped 26 previous similar messages
[ 7463.926902] Lustre: *** cfs_fail_loc=a08, val=0***
[ 7463.932232] Lustre: Skipped 15 previous similar messages
[ 7466.963810] Lustre: *** cfs_fail_loc=a08, val=0***
[ 7466.978142] Lustre: Skipped 88 previous similar messages
[ 7473.062792] Lustre: *** cfs_fail_loc=a08, val=0***
[ 7473.068214] Lustre: Skipped 76 previous similar messages
[ 7483.318956] Lustre: *** cfs_fail_loc=a08, val=0***
[ 7483.320848] Lustre: Skipped 317 previous similar messages
[ 7503.787841] Lustre: *** cfs_fail_loc=a08, val=0***
[ 7503.800044] Lustre: Skipped 872 previous similar messages
[ 7535.591490] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7535.607885] Lustre: Skipped 1 previous similar message
[ 7535.611772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7536.077098] Lustre: server umount lustre-MDT0000 complete
[ 7541.729457] LustreError: 191439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789309975 with bad export cookie 6881733252442451813
[ 7541.738695] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7541.957502] Lustre: server umount lustre-OST0000 complete
[ 7547.476338] Lustre: server umount lustre-OST0001 complete
[ 7557.360100] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid
[ 7569.873063] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local
[ 7583.026672]  vdc: vdc1 vdc9
[ 7594.858776]  vde: vde1 vde9
[ 7606.633234]  vdf: vdf1 vdf9
[ 7629.373527] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local
[ 7641.021940] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 7641.425163] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 7641.535513] Lustre: lustre-MDT0000: new disk, initializing
[ 7642.243498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7642.247347] Lustre: Skipped 2 previous similar messages
[ 7642.306080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 7647.583698] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7653.869270] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 7661.417296] Lustre: lustre-OST0000: new disk, initializing
[ 7661.423402] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 7661.432489] Lustre: Skipped 1 previous similar message
[ 7661.580238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7663.514249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[ 7663.518432] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[ 7663.596274] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[ 7670.159172] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7683.978230] Lustre: lustre-OST0001: new disk, initializing
[ 7683.980863] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 7684.098884] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7685.319265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[ 7685.325981] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[ 7685.541194] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[ 7693.453442] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7705.100381] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7709.636975] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 7750.938773] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 10:36:23 (1789310183)
[ 7759.243584] Lustre: DEBUG MARKER: == sanity-quota test 49c: lfs quota long options don't consume an extra argument ========================================================== 10:36:31 (1789310191)
[ 7767.421934] Lustre: DEBUG MARKER: == sanity-quota test 49d: lfs quota -d and --delimiter both work ========================================================== 10:36:40 (1789310200)
[ 7774.208210] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 10:36:47 (1789310207)
[ 7825.317218] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 10:37:37 (1789310257)
[ 7893.085328] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 10:38:45 (1789310325)
[ 7925.772116] Lustre: DEBUG MARKER: rename directory return 255
[ 7973.232649] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 10:40:05 (1789310405)
[ 8012.078805] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 10:40:44 (1789310444)
[ 8059.792433] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 10:41:32 (1789310492)
[ 8191.544803] LustreError: 199307:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 8228.688505] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 10:44:21 (1789310661)
[ 8265.063640] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 10:44:57 (1789310697)
[ 8311.463990] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 10:45:44 (1789310744)
[ 8508.328908] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 10:49:00 (1789310940)
[ 8510.075432] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[ 8512.036951] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 10:49:04 (1789310944)
[ 8576.055733] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[ 8577.856576] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 10:50:10 (1789311010)
[ 8612.909775] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[ 8616.030208] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 10:50:48 (1789311048)
[ 8675.721823] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[ 8677.648866] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 10:51:50 (1789311110)
[ 8732.404589] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 10:52:45 (1789311165)
[ 8734.111770] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[ 8736.107590] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 10:52:48 (1789311168)
[ 8766.005329] LustreError: 218523:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[ 8816.249541] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 10:54:08 (1789311248)
[ 8845.804751] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[ 8847.511838] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[ 9044.475273] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 10:57:57 (1789311477)
[ 9101.571694] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 10:58:53 (1789311533)
[ 9141.492538] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 10:59:34 (1789311574)
[ 9143.689060] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[ 9146.378239] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 10:59:38 (1789311578)
[ 9148.883864] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[ 9151.212581] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 10:59:43 (1789311583)
[ 9169.752328] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[ 9190.173391] Lustre: DEBUG MARKER: Write...
[ 9193.256064] Lustre: DEBUG MARKER: Write out of block quota ...
[ 9254.519886] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 11:01:27 (1789311687)
[ 9284.014019] Lustre: DEBUG MARKER: set to use default quota
[ 9285.607178] Lustre: DEBUG MARKER: set default quota
[ 9287.744289] Lustre: DEBUG MARKER: get default quota
[ 9295.420861] Lustre: DEBUG MARKER: Test not out of quota
[ 9300.149925] Lustre: DEBUG MARKER: Test out of quota
[ 9314.408619] Lustre: DEBUG MARKER: Increase default quota
[ 9352.163291] Lustre: DEBUG MARKER: Set quota to override default quota
[ 9352.246068] LustreError: 201318:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1789916586 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 9367.384607] Lustre: DEBUG MARKER: Set to use default quota again
[ 9391.247734] Lustre: DEBUG MARKER: Cleanup
[ 9469.505755] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 11:05:02 (1789311902)
[ 9497.699912] Lustre: DEBUG MARKER: set default quota for qpool1
[ 9500.445390] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[ 9560.333856] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 11:06:32 (1789311992)
[ 9652.687192] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 11:08:05 (1789312085)
[ 9734.764535] Lustre: DEBUG MARKER: Write...
[ 9738.508582] Lustre: DEBUG MARKER: Write out of block quota ...
[ 9865.689758] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 11:11:38 (1789312298)
[ 9913.573607] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 11:12:26 (1789312346)
[ 9920.684722] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 11:12:33 (1789312353)
[ 9923.127481] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A fallocate not supported
[ 9924.959857] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 11:12:37 (1789312357)
[ 9927.429542] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a fallocate not supported
[ 9929.058941] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 11:12:42 (1789312362)
[ 9946.474422] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 11:12:59 (1789312379)
[ 9947.786641] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[ 9949.188781] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 11:13:02 (1789312382)
[ 9964.397028] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[ 9971.317757] LustreError: 243545:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[ 9971.329202] LustreError: 243545:0:(qmt_pool.c:1407:qmt_pool_recalc()) Skipped 5 previous similar messages
[ 9975.714496] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.46@tcp (stopping)
[ 9975.728651] Lustre: Skipped 1 previous similar message
[ 9975.777563] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9975.784913] Lustre: Skipped 2 previous similar messages
[ 9980.813318] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.46@tcp (stopping)
[ 9980.819651] Lustre: Skipped 2 previous similar messages
[ 9981.416112] LustreError: 243545:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[ 9981.689442] Lustre: server umount lustre-MDT0000 complete
[ 9988.582742] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9988.994351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9989.181113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[ 9989.187993] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[ 9993.791376] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10006.308367] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[10006.314372] Lustre: Skipped 3 previous similar messages
[10034.682857] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 11:14:27 (1789312467)
[10065.735902] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 11:14:58 (1789312498)
[10095.732552] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 11:15:28 (1789312528)
[10142.261236] Lustre: *** cfs_fail_loc=a08, val=0***
[10142.268766] Lustre: Skipped 2058 previous similar messages
[10142.274675] Lustre: *** cfs_fail_loc=a08, val=0***
[10142.276844] Lustre: Skipped 701 previous similar messages
[10246.143253] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 11:17:59 (1789312679)
[10318.585835] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 11:19:11 (1789312751)
[10389.598743] LustreError: 244129:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1789917623 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[10523.139652] LustreError: 244128:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1789917757 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[10658.931732] LustreError: 244126:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:1000 enforced:1 hard:2048 soft:2048 granted:65536 time:1789917893 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[10666.536928] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed