[ 5162.909489] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5168.370158] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5171.257460] LustreError: 122492:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5172.274205] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5174.261496] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5176.514619] Lustre: Failing over lustre-MDT0000
[ 5176.900453] Lustre: server umount lustre-MDT0000 complete
[ 5194.272937] Lustre: 3311:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787836044/real 1787836044]  req@ffff8d270df25a40 x1874676608571264/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1787836060 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5194.295450] Lustre: 3311:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5194.300196] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5195.356641] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5195.882531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5195.930411] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5196.351671] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5196.512992] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5196.580396] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:655 to 0x280000400:673)
[ 5196.587350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:659 to 0x240000400:705)
[ 5196.768650] Lustre: 3313:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787836046/real 1787836046]  req@ffff8d27c31903c0 x1874676608571648/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787836062 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5199.844042] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5200.333367] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5200.993794] Lustre: 3312:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787836051/real 1787836051]  req@ffff8d27c61e3480 x1874676608571904/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787836067 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5201.026089] Lustre: 3312:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5205.985527] Lustre: 3312:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787836056/real 1787836056]  req@ffff8d27091fd680 x1874676608572288/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787836072 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5206.025283] Lustre: 3312:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5210.074654] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5212.205260] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5223.139963] Lustre: DEBUG MARKER: (dd_pid=115367, time=4, timeout=600)
[ 5269.619138] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5276.021708] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5279.534155] LustreError: 125397:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5280.399748] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5281.908169] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5284.226576] Lustre: Failing over lustre-MDT0000
[ 5284.633922] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 5284.648822] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5284.666117] Lustre: Skipped 1 previous similar message
[ 5284.693936] LustreError: 123134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5284.730396] LustreError: 123134:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message
[ 5284.873331] Lustre: server umount lustre-MDT0000 complete
[ 5300.194673] Lustre: 125591:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787836150/real 1787836150]  req@ffff8d27c7a59a40 x1874676608610944/t0(0) o101->lustre-MDT0000-lwp-OST0001@0@lo:23/10 lens 456/496 e 0 to 1 dl 1787836166 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'qsd_reint_0.lus.0' uid:0 gid:0 projid:4294967295
[ 5300.230358] Lustre: 125591:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5302.177295] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5312.416567] LustreError: 3310:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff8d26c1c54b40 x1874676608614528/t0(0) o250->MGC192.168.203.110@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5313.171722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5313.249141] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5315.084735] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5315.339112] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5315.409540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:707 to 0x240000400:737)
[ 5315.413539] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:655 to 0x280000400:705)
[ 5319.834417] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5321.384710] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 5321.397589] Lustre: Skipped 1 previous similar message
[ 5325.537791] LustreError: 3310:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-lwp-OST0001: namespace resource [0x200000006:0x20000:0x0].0x0 (ffff8d2708ded000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5325.604571] LustreError: 3310:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 5 previous similar messages
[ 5332.403216] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5334.562487] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5340.863973] Lustre: DEBUG MARKER: (dd_pid=117809, time=1, timeout=600)
[ 5412.581712] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 09:11:16 (1787836276)
[ 5435.731567] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5437.217942] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5442.877876] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5444.314223] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5479.059385] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 09:12:23 (1787836343)
[ 5504.782667] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 5519.573452] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 5521.617969] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 5523.920274] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 5526.458278] Lustre: DEBUG MARKER: Set quota for 1 times
[ 5530.268566] Lustre: DEBUG MARKER: Set quota for 2 times
[ 5534.193325] Lustre: DEBUG MARKER: Set quota for 3 times
[ 5538.023346] Lustre: DEBUG MARKER: Set quota for 4 times
[ 5542.008619] Lustre: DEBUG MARKER: Set quota for 5 times
[ 5545.403366] Lustre: DEBUG MARKER: Set quota for 6 times
[ 5549.053549] Lustre: DEBUG MARKER: Set quota for 7 times
[ 5552.941932] Lustre: DEBUG MARKER: Set quota for 8 times
[ 5612.879521] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 09:14:37 (1787836477)
[ 5627.011582] Lustre: server umount lustre-MDT0000 complete
[ 5631.286451] LustreError: 5831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836497 with bad export cookie 16332865063291266477
[ 5631.299575] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5641.853957] Lustre: server umount lustre-OST0000 complete
[ 5643.745518] Lustre: 3313:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787836493/real 1787836493]  req@ffff8d27d085bc00 x1874676608738560/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787836509 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5643.781510] Lustre: 3313:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 5643.788374] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5643.797789] Lustre: Skipped 1 previous similar message
[ 5647.426615] Lustre: server umount lustre-OST0001 complete
[ 5666.002074] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local
[ 5678.131231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5684.085263] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5687.609488] Lustre: 135206:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5695.150817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5703.167962] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5703.307173] LustreError: 135576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5703.410180] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:739 to 0x240000400:769)
[ 5713.347810] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5718.532243] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:708 to 0x280000400:737)
[ 5719.462178] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5726.841980] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5730.538993] Lustre: 137127:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5749.219560] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5749.220926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5749.234526] Lustre: Skipped 1 previous similar message
[ 5749.244638] Lustre: Skipped 1 previous similar message
[ 5749.501578] Lustre: server umount lustre-MDT0000 complete
[ 5753.781903] LustreError: 134754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836619 with bad export cookie 16332865063291273659
[ 5753.792521] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5753.845737] Lustre: server umount lustre-OST0000 complete
[ 5758.681786] Lustre: server umount lustre-OST0001 complete
[ 5776.505815] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local
[ 5789.082312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5794.250819] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5798.627311] Lustre: 139824:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5806.148275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5813.281665] LustreError: 140193:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5813.314886] LustreError: 140193:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message
[ 5813.319052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:739 to 0x240000400:801)
[ 5813.973464] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5829.661536] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:708 to 0x280000400:769)
[ 5832.251361] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5840.805178] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5844.893414] Lustre: 141740:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5856.512794] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 09:18:40 (1787836720)
[ 5858.190898] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 5860.227455] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 09:18:44 (1787836724)
[ 5918.586527] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 09:19:43 (1787836783)
[ 5970.873581] Lustre: DEBUG MARKER: Write...
[ 5973.375826] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6029.637350] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 09:21:34 (1787836894)
[ 6036.211429] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 09:21:40 (1787836900)
[ 6046.420330] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 09:21:50 (1787836910)
[ 6055.133714] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 09:21:59 (1787836919)
[ 6063.059305] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 09:22:07 (1787836927)
[ 6125.952892] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6281.125701] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 6469.521972] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 09:28:54 (1787837334)
[ 6535.998559] Lustre: DEBUG MARKER: Restart...
[ 6544.353971] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6544.371594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6544.384125] Lustre: Skipped 1 previous similar message
[ 6546.953084] Lustre: server umount lustre-MDT0000 complete
[ 6550.846556] LustreError: 139381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837416 with bad export cookie 16332865063291275192
[ 6550.854229] LustreError: 139381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 6550.855897] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6550.938357] Lustre: server umount lustre-OST0000 complete
[ 6555.071879] Lustre: server umount lustre-OST0001 complete
[ 6569.686065] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local
[ 6580.697671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6580.703594] Lustre: Skipped 1 previous similar message
[ 6585.360775] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6588.276188] Lustre: 160466:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6595.061398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6601.256070] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6605.286671] LustreError: 161383:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6605.313264] LustreError: 161383:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages
[ 6605.319657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:810 to 0x240000400:833)
[ 6609.650382] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6615.032912] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:778 to 0x280000400:801)
[ 6615.672318] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6622.849713] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6626.554658] Lustre: 162383:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6693.077211] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 09:32:37 (1787837557)
[ 6770.686591] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 09:33:55 (1787837635)
[ 8381.655871] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 10:00:46 (1787839246)
[ 8396.046915] Lustre: server umount lustre-MDT0000 complete
[ 8400.184586] LustreError: 160019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839266 with bad export cookie 16332865063291281688
[ 8400.200189] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8410.742349] Lustre: server umount lustre-OST0000 complete
[ 8412.641408] Lustre: 3311:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787839262/real 1787839262]  req@ffff8d2705c79680 x1874676611947136/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787839278 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 8412.688894] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8412.697853] Lustre: Skipped 1 previous similar message
[ 8415.271918] Lustre: server umount lustre-OST0001 complete
[ 8430.592208] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local
[ 8439.513096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8443.788560] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8446.442084] Lustre: 170925:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 8452.691425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 8458.465351] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8464.881741] LustreError: 171286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8464.908068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5836 to 0x240000400:5857)
[ 8466.388696] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 8471.555570] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5802 to 0x280000400:5825)
[ 8472.352327] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8479.115297] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8482.462971] Lustre: 172834:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 8513.322448] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 10:02:58 (1787839378)
[ 8558.232514] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 10:03:43 (1787839423)
[ 8607.332614] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 10:04:32 (1787839472)
[ 8609.101539] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 8611.464454] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 10:04:35 (1787839475)
[ 8613.672387] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 8615.307431] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 10:04:40 (1787839480)
[ 8696.855830] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 10:06:01 (1787839561)
[ 8746.254772] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 10:06:51 (1787839611)
[ 8827.741592] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 10:08:12 (1787839692)
[ 8837.859515] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8837.863512] Lustre: Skipped 1 previous similar message
[ 8839.873487] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8839.877362] Lustre: Skipped 97 previous similar messages
[ 8843.891177] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8843.893201] Lustre: Skipped 193 previous similar messages
[ 8851.917110] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8851.921627] Lustre: Skipped 387 previous similar messages
[ 8867.935536] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8867.940602] Lustre: Skipped 777 previous similar messages
[ 8899.961170] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8899.963012] Lustre: Skipped 1549 previous similar messages
[ 8963.963890] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8963.965691] Lustre: Skipped 3569 previous similar messages
[ 9242.874619] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9242.891415] Lustre: Skipped 1424 previous similar messages
[ 9242.903759] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9242.916136] Lustre: Skipped 4 previous similar messages
[ 9375.970104] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9375.974020] Lustre: Skipped 12 previous similar messages
[ 9531.365438] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9531.369824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9531.385256] Lustre: Skipped 1 previous similar message
[ 9531.401173] Lustre: Skipped 1 previous similar message
[ 9536.482358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9536.494428] Lustre: Skipped 1 previous similar message
[ 9536.603220] Lustre: server umount lustre-MDT0000 complete
[ 9540.606918] LustreError: 170480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840406 with bad export cookie 16332865063293057987
[ 9540.625694] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9541.060833] Lustre: server umount lustre-OST0000 complete
[ 9545.291335] Lustre: server umount lustre-OST0001 complete
[ 9553.052273] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_hostid
[ 9561.976404] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local
[ 9571.398676]  vdc: vdc1 vdc9
[ 9581.390161]  vde: vde1 vde9
[ 9581.475878]  vde: vde1 vde9
[ 9593.053350]  vdf: vdf1 vdf9
[ 9614.884715] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local
[ 9623.785816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 9624.125411] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 9624.184339] Lustre: lustre-MDT0000: new disk, initializing
[ 9624.712453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9624.831036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 9629.308333] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9634.177267] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 9640.374360] Lustre: lustre-OST0000: new disk, initializing
[ 9640.378439] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 9640.384401] Lustre: Skipped 1 previous similar message
[ 9640.503055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9642.178222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[ 9642.185534] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[ 9642.315762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[ 9646.118392] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9655.163484] Lustre: lustre-OST0001: new disk, initializing
[ 9655.167461] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 9655.245616] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9656.493829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[ 9656.509231] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[ 9656.590188] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[ 9660.640022] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9669.577647] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9672.764247] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 9706.367834] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 10:22:51 (1787840571)
[ 9711.863267] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 10:22:56 (1787840576)
[ 9760.161572] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 10:23:45 (1787840625)
[ 9822.394931] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 10:24:47 (1787840687)
[ 9848.108434] Lustre: DEBUG MARKER: rename directory return 255
[ 9892.130303] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 10:25:57 (1787840757)
[ 9921.688855] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 10:26:26 (1787840786)
[ 9966.570206] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 10:27:11 (1787840831)
[10062.849257] LustreError: 187726:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[10102.190900] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 10:29:26 (1787840966)
[10138.589236] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 10:30:03 (1787841003)
[10184.282796] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 10:30:48 (1787841048)
[10360.994785] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 10:33:45 (1787841225)
[10362.782325] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[10365.195190] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 10:33:49 (1787841229)
[10423.773832] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[10426.008540] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 10:34:50 (1787841290)
[10460.510485] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[10462.407669] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 10:35:26 (1787841326)
[10519.747474] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[10522.023315] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 10:36:26 (1787841386)
[10574.483589] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 10:37:19 (1787841439)
[10575.857879] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[10577.335802] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 10:37:22 (1787841442)
[10603.942920] LustreError: 210978:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[10647.550251] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 10:38:32 (1787841512)
[10674.188483] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[10675.613430] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[10839.858200] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 10:41:44 (1787841704)
[10894.551336] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 10:42:39 (1787841759)
[10928.148900] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 10:43:12 (1787841792)
[10929.916967] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[10931.654202] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 10:43:16 (1787841796)
[10933.326597] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[10935.325900] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 10:43:19 (1787841799)
[10952.350711] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[10969.011645] Lustre: DEBUG MARKER: Write...
[10972.006172] Lustre: DEBUG MARKER: Write out of block quota ...
[11035.304724] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 10:44:59 (1787841899)
[11066.979664] Lustre: DEBUG MARKER: set to use default quota
[11068.937125] Lustre: DEBUG MARKER: set default quota
[11071.152152] Lustre: DEBUG MARKER: get default quota
[11079.772872] Lustre: DEBUG MARKER: Test not out of quota
[11083.922438] Lustre: DEBUG MARKER: Test out of quota
[11095.457605] Lustre: DEBUG MARKER: Increase default quota
[11126.049717] Lustre: DEBUG MARKER: Set quota to override default quota
[11126.106740] LustreError: 189282:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1788446792 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11138.045839] Lustre: DEBUG MARKER: Set to use default quota again
[11160.943439] Lustre: DEBUG MARKER: Cleanup
[11237.554028] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 10:48:22 (1787842102)
[11262.906352] Lustre: DEBUG MARKER: set default quota for qpool1
[11264.552184] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[11315.914570] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 10:49:40 (1787842180)
[11396.583435] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 10:51:01 (1787842261)
[11472.731320] Lustre: DEBUG MARKER: Write...
[11475.890798] Lustre: DEBUG MARKER: Write out of block quota ...
[11592.139944] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 10:54:16 (1787842456)
[11640.770837] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 10:55:05 (1787842505)
[11648.544870] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 10:55:13 (1787842513)
[11650.328378] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate
[11652.237574] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 10:55:16 (1787842516)
[11653.987805] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate
[11655.887069] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 10:55:20 (1787842520)
[11674.279194] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 10:55:39 (1787842539)
[11676.000741] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[11678.022990] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 10:55:42 (1787842542)
[11695.874791] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[11701.146982] LustreError: 236366:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[11707.925161] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.10@tcp (stopping)
[11709.409224] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[11709.411430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[11709.420946] Lustre: Skipped 1 previous similar message
[11709.443940] Lustre: Skipped 1 previous similar message
[11711.192118] LustreError: 236366:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[11711.570271] Lustre: server umount lustre-MDT0000 complete
[11719.712578] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[11720.259201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[11720.438889] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[11720.440103] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[11724.586361] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11729.383647] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[11729.393875] Lustre: Skipped 1 previous similar message
[11756.404819] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 10:57:01 (1787842621)
[11790.190321] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 10:57:34 (1787842654)
[11823.706463] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 10:58:08 (1787842688)
[11872.975783] Lustre: *** cfs_fail_loc=a08, val=0***
[11872.981893] Lustre: Skipped 26 previous similar messages
[11872.993606] Lustre: *** cfs_fail_loc=a08, val=0***
[11872.995736] Lustre: Skipped 13 previous similar messages
[11984.743996] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 11:00:48 (1787842848)
[12068.101939] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 11:02:12 (1787842932)
[12140.806248] LustreError: 237825:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1788447806 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12148.604676] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed