[ 5133.392220] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5139.059855] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5141.704196] LustreError: 122324:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5142.543931] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5144.101603] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5146.328503] Lustre: Failing over lustre-MDT0000
[ 5146.891096] Lustre: server umount lustre-MDT0000 complete
[ 5162.976975] Lustre: 3285:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786724581/real 1786724581]  req@ffff9ba4d020ca80 x1873511176070656/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1786724597 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5163.034782] Lustre: 3285:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5163.050060] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5165.145601] Lustre: 3284:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786724583/real 1786724583]  req@ffff9ba4d020ea00 x1873511176070784/t0(0) o400->MGC192.168.206.110@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786724599 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5165.181643] Lustre: 3284:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5165.200040] LustreError: MGC192.168.206.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5170.335145] Lustre: 3284:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786724588/real 1786724588]  req@ffff9ba4d020e300 x1873511176071296/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786724604 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5175.263088] Lustre: 3285:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786724593/real 1786724593]  req@ffff9ba4d81ef480 x1873511176071680/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786724609 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5175.298957] Lustre: 3285:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5175.398264] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x58f78de25706bbc
[ 5175.412458] Lustre: MGC192.168.206.110@tcp: Connection restored to 0@lo (at 0@lo)
[ 5175.855851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5175.942156] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5177.484568] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5177.696410] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5177.750478] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:639 to 0x280000400:673)
[ 5177.755642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:643 to 0x240000400:673)
[ 5180.400743] Lustre: DEBUG MARKER: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5189.482757] Lustre: DEBUG MARKER: oleg610-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5190.900611] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5206.970242] Lustre: DEBUG MARKER: (dd_pid=115604, time=9, timeout=600)
[ 5254.730754] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5260.516337] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5263.238557] LustreError: 125257:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5264.159684] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5265.704965] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5267.554305] Lustre: Failing over lustre-MDT0000
[ 5267.997086] Lustre: server umount lustre-MDT0000 complete
[ 5284.449182] Lustre: 3283:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786724702/real 1786724702]  req@ffff9ba4fc3e9500 x1873511176113152/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1786724718 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5284.527409] Lustre: 3283:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5284.552287] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5284.582776] Lustre: Skipped 1 previous similar message
[ 5288.249043] LustreError: MGC192.168.206.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5288.677154] LustreError: 125904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5288.705310] LustreError: 125904:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages
[ 5289.423908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5289.553913] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5294.441848] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 5294.450838] Lustre: Skipped 2 previous similar messages
[ 5295.092791] Lustre: DEBUG MARKER: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5295.229565] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5295.356080] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5295.407784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:675 to 0x240000400:705)
[ 5295.417819] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:639 to 0x280000400:705)
[ 5300.643924] Lustre: 3282:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786724718/real 1786724718]  req@ffff9ba3c8915c00 x1873511176114560/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786724734 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5300.663989] Lustre: 3282:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 5305.074990] Lustre: DEBUG MARKER: oleg610-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5306.851297] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5313.784502] Lustre: DEBUG MARKER: (dd_pid=118065, time=2, timeout=600)
[ 5376.113651] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 12:26:48 (1786724808)
[ 5397.210316] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5398.681554] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5403.887428] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5405.431508] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5439.639533] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 12:27:52 (1786724872)
[ 5460.018427] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 5476.522879] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 5478.232892] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 5479.532847] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 5481.576970] Lustre: DEBUG MARKER: Set quota for 1 times
[ 5485.406819] Lustre: DEBUG MARKER: Set quota for 2 times
[ 5488.848709] Lustre: DEBUG MARKER: Set quota for 3 times
[ 5492.676832] Lustre: DEBUG MARKER: Set quota for 4 times
[ 5496.422723] Lustre: DEBUG MARKER: Set quota for 5 times
[ 5500.010836] Lustre: DEBUG MARKER: Set quota for 6 times
[ 5504.420892] Lustre: DEBUG MARKER: Set quota for 7 times
[ 5507.524371] Lustre: DEBUG MARKER: Set quota for 8 times
[ 5510.877757] Lustre: DEBUG MARKER: Set quota for 9 times
[ 5561.342449] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 12:29:53 (1786724993)
[ 5575.189480] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5575.208021] Lustre: Skipped 1 previous similar message
[ 5575.212761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5575.218299] Lustre: Skipped 1 previous similar message
[ 5575.574225] Lustre: server umount lustre-MDT0000 complete
[ 5580.916526] LustreError: 5796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786725014 with bad export cookie 400671787365463122
[ 5580.925506] LustreError: MGC192.168.206.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5580.932272] LustreError: 5796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 5581.102339] Lustre: server umount lustre-OST0000 complete
[ 5585.998317] Lustre: server umount lustre-OST0001 complete
[ 5599.009677] Lustre: DEBUG MARKER: oleg610-server.virtnet: executing load_modules_local
[ 5607.478411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5611.378966] Lustre: DEBUG MARKER: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5614.058952] Lustre: 135203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5619.948389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5626.067235] Lustre: DEBUG MARKER: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5632.169300] LustreError: 135579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5632.204893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:707 to 0x240000400:737)
[ 5633.761259] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5639.167573] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:708 to 0x280000400:737)
[ 5639.496901] Lustre: DEBUG MARKER: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5646.876692] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5650.173948] Lustre: 137134:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5673.294045] Lustre: server umount lustre-MDT0000 complete
[ 5677.839224] LustreError: 134750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786725111 with bad export cookie 400671787365470514
[ 5677.855520] LustreError: MGC192.168.206.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5687.360771] Lustre: server umount lustre-OST0000 complete
[ 5691.359331] Lustre: 3284:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786725109/real 1786725109]  req@ffff9ba4ddf37100 x1873511176276224/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786725125 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5691.396858] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5691.414766] Lustre: Skipped 1 previous similar message
[ 5691.841430] Lustre: server umount lustre-OST0001 complete
[ 5706.323996] Lustre: DEBUG MARKER: oleg610-server.virtnet: executing load_modules_local
[ 5714.359045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5718.509636] Lustre: DEBUG MARKER: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5721.269337] Lustre: 139845:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5733.155351] Lustre: DEBUG MARKER: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5739.491674] LustreError: 140214:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5739.512788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:707 to 0x240000400:769)
[ 5740.527669] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5740.531434] Lustre: Skipped 1 previous similar message
[ 5745.641680] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:708 to 0x280000400:769)
[ 5746.431838] Lustre: DEBUG MARKER: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5752.650808] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5755.643693] Lustre: 141763:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5763.505097] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 12:33:16 (1786725196)
[ 5765.392911] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 5767.190968] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 12:33:19 (1786725199)
[ 5821.054641] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 12:34:13 (1786725253)
[ 5866.007134] Lustre: DEBUG MARKER: Write...
[ 5868.782159] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5925.934442] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 12:35:58 (1786725358)
[ 5933.843379] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 12:36:06 (1786725366)
[ 5944.286442] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 12:36:17 (1786725377)
[ 5952.895735] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 12:36:25 (1786725385)
[ 5960.772615] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 12:36:33 (1786725393)
[ 6025.543888] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6156.500957] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 6329.381344] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 12:42:42 (1786725762)
[ 6384.042296] Lustre: DEBUG MARKER: Restart...
[ 6387.749981] Lustre: server umount lustre-MDT0000 complete
[ 6391.336375] LustreError: 145016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786725825 with bad export cookie 400671787365472047
[ 6391.342126] LustreError: 3283:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0000 qtype:usr lqe: ffff9ba4fc3ad080 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2378 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 6391.348743] LustreError: MGC192.168.206.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6401.637613] Lustre: server umount lustre-OST0000 complete
[ 6404.999850] LustreError: 3282:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0001 qtype:usr lqe: ffff9ba4e7df0a80 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2378 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 6415.399525] Lustre: server umount lustre-OST0001 complete
[ 6428.819231] Lustre: DEBUG MARKER: oleg610-server.virtnet: executing load_modules_local
[ 6438.992680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6442.739987] Lustre: DEBUG MARKER: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6444.970484] Lustre: 160711:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6450.073821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6455.923307] Lustre: DEBUG MARKER: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6463.331257] LustreError: 161075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6463.366922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:779 to 0x240000400:801)
[ 6463.950660] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6469.144758] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:777 to 0x280000400:801)
[ 6470.175499] Lustre: DEBUG MARKER: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6477.868374] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6481.219195] Lustre: 162617:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6546.201103] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 12:46:18 (1786725978)
[ 6617.043326] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 12:47:29 (1786726049)
[ 8069.475899] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 13:11:42 (1786727502)
[ 8085.232982] Lustre: server umount lustre-MDT0000 complete
[ 8088.601246] LustreError: 167396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786727522 with bad export cookie 400671787365478536
[ 8088.608419] LustreError: MGC192.168.206.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8098.925205] Lustre: server umount lustre-OST0000 complete
[ 8112.282885] Lustre: server umount lustre-OST0001 complete
[ 8123.602892] Lustre: DEBUG MARKER: oleg610-server.virtnet: executing load_modules_local
[ 8133.002684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8138.030697] Lustre: DEBUG MARKER: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8140.695975] Lustre: 171094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 8145.943844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 8151.407616] Lustre: DEBUG MARKER: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8157.153534] LustreError: 171459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8157.173561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5803 to 0x240000400:5825)
[ 8159.146246] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 8164.357257] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5803 to 0x280000400:5825)
[ 8165.105407] Lustre: DEBUG MARKER: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8172.413974] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8175.771737] Lustre: 173011:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 8213.698350] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 13:14:06 (1786727646)
[ 8257.993214] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 13:14:50 (1786727690)
[ 8299.159266] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 13:15:32 (1786727732)
[ 8300.356950] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 8301.770889] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 13:15:34 (1786727734)
[ 8302.934830] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 8304.574177] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 13:15:37 (1786727737)
[ 8384.981622] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 13:16:57 (1786727817)
[ 8430.534612] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 13:17:43 (1786727863)
[ 8515.694741] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 13:19:08 (1786727948)
[ 8525.489996] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8525.493504] Lustre: Skipped 1 previous similar message
[ 8526.521266] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8526.536042] Lustre: Skipped 67 previous similar messages
[ 8528.533939] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8528.542792] Lustre: Skipped 105 previous similar messages
[ 8532.561804] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8532.565111] Lustre: Skipped 257 previous similar messages
[ 8540.571050] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8540.575830] Lustre: Skipped 399 previous similar messages
[ 8556.601195] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8556.603603] Lustre: Skipped 855 previous similar messages
[ 8588.621277] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8588.625149] Lustre: Skipped 1741 previous similar messages
[ 8652.636814] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8652.639408] Lustre: Skipped 3549 previous similar messages
[ 8931.368916] Lustre: *** cfs_fail_loc=a08, val=0***
[ 8931.370548] Lustre: Skipped 1019 previous similar messages
[ 8931.377495] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9062.452853] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9062.456766] Lustre: Skipped 15 previous similar messages
[ 9219.040946] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9219.054996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9221.437049] Lustre: server umount lustre-MDT0000 complete
[ 9225.773300] LustreError: 170648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786728659 with bad export cookie 400671787367252126
[ 9225.787990] LustreError: MGC192.168.206.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9226.103000] Lustre: server umount lustre-OST0000 complete
[ 9230.801513] Lustre: server umount lustre-OST0001 complete
[ 9238.945301] Lustre: DEBUG MARKER: oleg610-server.virtnet: executing set_hostid
[ 9247.366705] Lustre: DEBUG MARKER: oleg610-server.virtnet: executing load_modules_local
[ 9260.482454]  vdc: vdc1 vdc9
[ 9271.998987]  vde: vde1 vde9
[ 9281.149033]  vdf: vdf1 vdf9
[ 9281.185542]  vdf: vdf1 vdf9
[ 9281.210865]  vdf: vdf1 vdf9
[ 9296.107251] Lustre: DEBUG MARKER: oleg610-server.virtnet: executing load_modules_local
[ 9304.507328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 9304.768337] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 9304.857983] Lustre: lustre-MDT0000: new disk, initializing
[ 9305.183915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9305.235830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 9309.413686] Lustre: DEBUG MARKER: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9313.360506] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 9318.921896] Lustre: lustre-OST0000: new disk, initializing
[ 9318.926506] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 9318.930311] Lustre: Skipped 1 previous similar message
[ 9319.051371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9320.942715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[ 9320.952525] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[ 9321.068669] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[ 9324.784631] Lustre: DEBUG MARKER: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9334.543737] Lustre: lustre-OST0001: new disk, initializing
[ 9334.548730] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 9334.632635] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9336.214100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[ 9336.220043] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[ 9336.317976] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[ 9340.443740] Lustre: DEBUG MARKER: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9349.548975] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9353.385809] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 9389.211287] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 13:33:41 (1786728821)
[ 9395.668972] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 13:33:48 (1786728828)
[ 9443.269498] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 13:34:36 (1786728876)
[ 9499.551076] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 13:35:32 (1786728932)
[ 9529.978377] Lustre: DEBUG MARKER: rename directory return 255
[ 9572.068550] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 13:36:44 (1786729004)
[ 9600.141462] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 13:37:13 (1786729033)
[ 9640.354417] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 13:37:53 (1786729073)
[ 9772.055356] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 13:40:04 (1786729204)
[ 9806.223120] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 13:40:38 (1786729238)
[ 9848.940938] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 13:41:21 (1786729281)
[10011.994588] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 13:44:04 (1786729444)
[10013.603623] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[10015.441585] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 13:44:08 (1786729448)
[10081.202294] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[10083.162543] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 13:45:15 (1786729515)
[10110.159274] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[10111.466627] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 13:45:44 (1786729544)
[10156.189513] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[10157.857870] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 13:46:30 (1786729590)
[10206.248227] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 13:47:19 (1786729639)
[10207.785815] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[10209.564893] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 13:47:22 (1786729642)
[10236.876350] LustreError: 211360:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[10281.190353] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 13:48:33 (1786729713)
[10307.818350] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[10309.418947] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[10482.754781] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 13:51:55 (1786729915)
[10533.277975] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 13:52:45 (1786729965)
[10566.182486] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 13:53:19 (1786729999)
[10567.382473] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[10568.959730] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 13:53:21 (1786730001)
[10570.092973] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[10571.569874] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 13:53:24 (1786730004)
[10586.910190] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[10601.377478] Lustre: DEBUG MARKER: Write...
[10603.702745] Lustre: DEBUG MARKER: Write out of block quota ...
[10657.582409] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 13:54:50 (1786730090)
[10683.653405] Lustre: DEBUG MARKER: set to use default quota
[10685.392629] Lustre: DEBUG MARKER: set default quota
[10687.165769] Lustre: DEBUG MARKER: get default quota
[10692.639580] Lustre: DEBUG MARKER: Test not out of quota
[10696.328525] Lustre: DEBUG MARKER: Test out of quota
[10706.495821] Lustre: DEBUG MARKER: Increase default quota
[10744.911645] Lustre: DEBUG MARKER: Set quota to override default quota
[10744.997519] LustreError: 190967:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1787334979 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[10757.149349] Lustre: DEBUG MARKER: Set to use default quota again
[10777.990745] Lustre: DEBUG MARKER: Cleanup
[10850.758310] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 13:58:03 (1786730283)
[10874.517770] Lustre: DEBUG MARKER: set default quota for qpool1
[10876.302458] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[10929.718963] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 13:59:22 (1786730362)
[11018.442679] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 14:00:50 (1786730450)
[11090.173379] Lustre: DEBUG MARKER: Write...
[11093.197179] Lustre: DEBUG MARKER: Write out of block quota ...
[11217.221898] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 14:04:09 (1786730649)
[11263.572886] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 14:04:55 (1786730695)
[11273.322455] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 14:05:05 (1786730705)
[11275.963931] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate
[11278.225776] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 14:05:10 (1786730710)
[11280.278372] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate
[11282.074513] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 14:05:14 (1786730714)
[11301.879688] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 14:05:34 (1786730734)
[11303.588941] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[11305.003838] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 14:05:37 (1786730737)
[11323.780592] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[11330.348188] LustreError: 236963:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[11333.092887] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[11333.103722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[11333.116154] Lustre: Skipped 2 previous similar messages
[11333.120801] Lustre: Skipped 1 previous similar message
[11335.819568] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.10@tcp (stopping)
[11338.209330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[11338.214053] Lustre: Skipped 1 previous similar message
[11340.351121] LustreError: 236963:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[11340.614396] Lustre: server umount lustre-MDT0000 complete
[11348.642631] LustreError: MGC192.168.206.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[11349.094385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[11349.248782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[11349.251464] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[11354.290294] Lustre: DEBUG MARKER: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11363.311137] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[11363.320685] Lustre: Skipped 1 previous similar message
[11386.986574] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 14:06:59 (1786730819)
[11420.187030] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 14:07:32 (1786730852)
[11435.885117] LustreError: 3283:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[11461.175786] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 14:08:13 (1786730893)
[11516.649038] Lustre: *** cfs_fail_loc=a08, val=0***
[11516.652620] Lustre: Skipped 29 previous similar messages
[11516.658712] Lustre: *** cfs_fail_loc=a08, val=0***
[11516.660174] Lustre: Skipped 12 previous similar messages
[11626.445146] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 14:10:58 (1786731058)
[11717.394756] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 14:12:30 (1786731150)
[11793.556358] LustreError: 245785:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1787336027 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11801.852670] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed