[ 5789.725942] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5797.480555] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5802.625878] LustreError: 125180:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5803.642578] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5805.719953] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5808.137109] Lustre: Failing over lustre-MDT0000
[ 5808.401304] Lustre: server umount lustre-MDT0000 complete
[ 5826.016365] Lustre: 3307:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789606856/real 1789606856]  req@ffff90485c38c380 x1876532814683008/t0(0) o400->MGC192.168.203.141@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789606872 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5826.016764] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5826.026763] Lustre: 3307:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5826.026822] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5831.138565] Lustre: 3308:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789606861/real 1789606861]  req@ffff90484dde6a00 x1876532814683520/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789606877 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5831.180359] Lustre: 3308:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5835.941390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5835.986291] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5836.834852] Lustre: 3307:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789606866/real 1789606866]  req@ffff9047520b9f80 x1876532814684032/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789606882 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5836.870129] Lustre: 3307:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5837.006220] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5837.113863] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5837.194317] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:640 to 0x280000400:673)
[ 5837.202471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:643 to 0x240000400:673)
[ 5840.129038] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5848.908598] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5850.598911] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5850.885324] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5856.578950] Lustre: DEBUG MARKER: (dd_pid=117609, time=0, timeout=600)
[ 5904.817918] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5912.740816] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5916.366894] LustreError: 128106:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5917.108343] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5918.664373] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5920.463858] Lustre: Failing over lustre-MDT0000
[ 5920.641130] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 5920.648975] LustreError: Skipped 1 previous similar message
[ 5920.652391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5920.866921] Lustre: server umount lustre-MDT0000 complete
[ 5938.337505] Lustre: 3307:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789606969/real 1789606969]  req@ffff904742d77480 x1876532814721152/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789606985 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5938.363568] Lustre: 3307:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 5938.640780] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5948.962073] LustreError: 128764:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5949.336968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5949.400030] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5951.083148] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5951.272186] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5951.299579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:675 to 0x240000400:705)
[ 5951.303197] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:640 to 0x280000400:705)
[ 5954.195204] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5964.569573] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5966.239731] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5986.140817] Lustre: DEBUG MARKER: (dd_pid=120061, time=13, timeout=600)
[ 6058.618519] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 21:05:02 (1789607102)
[ 6083.315512] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6085.406438] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6092.000273] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6093.713632] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6129.252964] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 21:06:13 (1789607173)
[ 6155.760419] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 6174.430950] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 6176.488971] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 6178.160597] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 6180.419764] Lustre: DEBUG MARKER: Set quota for 1 times
[ 6184.021471] Lustre: DEBUG MARKER: Set quota for 2 times
[ 6188.010552] Lustre: DEBUG MARKER: Set quota for 3 times
[ 6191.903639] Lustre: DEBUG MARKER: Set quota for 4 times
[ 6195.490455] Lustre: DEBUG MARKER: Set quota for 5 times
[ 6199.170994] Lustre: DEBUG MARKER: Set quota for 6 times
[ 6202.932586] Lustre: DEBUG MARKER: Set quota for 7 times
[ 6206.999592] Lustre: DEBUG MARKER: Set quota for 8 times
[ 6255.044767] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 21:08:19 (1789607299)
[ 6270.220707] Lustre: server umount lustre-MDT0000 complete
[ 6275.239918] LustreError: 14296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789607322 with bad export cookie 2889418993523324858
[ 6275.258730] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6286.001705] Lustre: server umount lustre-OST0000 complete
[ 6286.315934] Lustre: 3306:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789607317/real 1789607317]  req@ffff904789b6dc00 x1876532814849024/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789607333 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6286.385141] Lustre: 3306:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 6286.403234] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6286.421748] Lustre: Skipped 3 previous similar messages
[ 6292.300915] Lustre: server umount lustre-OST0001 complete
[ 6310.085312] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local
[ 6320.784766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6327.105397] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6330.398955] Lustre: 138040:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6339.837631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6345.521401] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6345.963092] LustreError: 138423:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6346.005451] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:708 to 0x240000400:737)
[ 6353.466912] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6358.533656] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:707 to 0x280000400:737)
[ 6359.698181] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6368.315414] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6372.036345] Lustre: 139976:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6388.287563] Lustre: server umount lustre-MDT0000 complete
[ 6392.843903] LustreError: 137587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789607439 with bad export cookie 2889418993523334910
[ 6392.854645] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6403.114735] Lustre: server umount lustre-OST0000 complete
[ 6406.112107] Lustre: 3305:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789607436/real 1789607436]  req@ffff904851d24a80 x1876532814890752/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789607452 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6406.149776] Lustre: 3305:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 6406.663408] Lustre: server umount lustre-OST0001 complete
[ 6422.893967] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local
[ 6435.521107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6440.297687] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6444.136920] Lustre: 142619:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6451.778617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6458.613568] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6460.968074] LustreError: 142985:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6460.998537] LustreError: 142985:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message
[ 6461.034689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:708 to 0x240000400:769)
[ 6473.220598] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:707 to 0x280000400:769)
[ 6474.861332] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6482.928983] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6486.813694] Lustre: 144519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6497.633761] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 21:12:22 (1789607542)
[ 6499.550745] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6501.605949] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 21:12:25 (1789607545)
[ 6555.863733] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 21:13:20 (1789607600)
[ 6604.514267] Lustre: DEBUG MARKER: Write...
[ 6607.256742] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6661.929712] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 21:15:06 (1789607706)
[ 6667.893593] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 21:15:12 (1789607712)
[ 6678.254427] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 21:15:22 (1789607722)
[ 6688.236351] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 21:15:32 (1789607732)
[ 6697.203799] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 21:15:41 (1789607741)
[ 6762.511761] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6922.765949] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 7109.670710] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 21:22:34 (1789608154)
[ 7183.879416] Lustre: DEBUG MARKER: Restart...
[ 7189.472222] LustreError: 3308:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff904788747800 x1876532815115776/t0(0) o601->lustre-MDT0000-lwp-MDT0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 7189.472646] LustreError: lustre-MDT0000-lwp-OST0001: operation quota_acquire to node 0@lo failed: rc = -107
[ 7189.473311] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7189.473316] Lustre: Skipped 1 previous similar message
[ 7189.473762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7189.503303] LustreError: 3308:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-MDT0000 qtype:usr lqe: ffff90484fbca780 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2660 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 7189.509029] LustreError: Skipped 1 previous similar message
[ 7191.294901] Lustre: server umount lustre-MDT0000 complete
[ 7195.825525] LustreError: 146592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789608242 with bad export cookie 2889418993523336436
[ 7195.827213] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7195.844591] LustreError: 146592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 7195.951112] Lustre: server umount lustre-OST0000 complete
[ 7201.089394] Lustre: server umount lustre-OST0001 complete
[ 7218.522771] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local
[ 7230.377350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7230.382107] Lustre: Skipped 1 previous similar message
[ 7235.400966] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7238.728609] Lustre: 162975:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7246.016252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7253.690590] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7255.203516] LustreError: 163337:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7255.227567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:778 to 0x240000400:801)
[ 7255.232068] LustreError: 163337:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message
[ 7260.647543] LustreError: 163337:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7263.371220] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7268.370735] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:778 to 0x280000400:801)
[ 7270.248712] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7278.433782] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7281.869646] Lustre: 164888:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7359.634767] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 21:26:43 (1789608403)
[ 7433.525888] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 21:27:58 (1789608478)
[ 9324.123676] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 21:59:28 (1789610368)
[ 9342.946316] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9342.950143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9342.962145] Lustre: Skipped 2 previous similar messages
[ 9342.975114] Lustre: Skipped 2 previous similar messages
[ 9346.351941] Lustre: server umount lustre-MDT0000 complete
[ 9350.897140] LustreError: 162528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789610397 with bad export cookie 2889418993523342939
[ 9350.904470] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9350.913687] LustreError: 162528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 9351.022247] Lustre: server umount lustre-OST0000 complete
[ 9355.802229] Lustre: server umount lustre-OST0001 complete
[ 9372.792639] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local
[ 9383.768500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9388.122200] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9391.231699] Lustre: 173482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9397.873461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9404.515327] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9409.057680] LustreError: 173849:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9409.076295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5803 to 0x240000400:5825)
[ 9412.597436] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9417.742709] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5803 to 0x280000400:5825)
[ 9419.930615] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9429.025514] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9432.722031] Lustre: 175403:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9469.380540] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 22:01:54 (1789610514)
[ 9515.782145] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 22:02:40 (1789610560)
[ 9563.236349] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 22:03:27 (1789610607)
[ 9564.987476] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9566.799281] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 22:03:31 (1789610611)
[ 9568.458684] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9570.842866] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 22:03:35 (1789610615)
[ 9660.164645] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 22:05:04 (1789610704)
[ 9709.995311] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 22:05:54 (1789610754)
[ 9796.519624] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 22:07:20 (1789610840)
[ 9808.796601] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9808.804776] Lustre: Skipped 3 previous similar messages
[ 9810.796692] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9810.802730] Lustre: Skipped 59 previous similar messages
[ 9814.873188] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9814.880891] Lustre: Skipped 143 previous similar messages
[ 9822.878610] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9822.883283] Lustre: Skipped 337 previous similar messages
[ 9838.899831] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9838.903510] Lustre: Skipped 667 previous similar messages
[ 9870.931126] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9870.936801] Lustre: Skipped 1281 previous similar messages
[ 9934.954704] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9934.962677] Lustre: Skipped 2639 previous similar messages
[10351.010292] Lustre: *** cfs_fail_loc=a08, val=0***
[10351.012633] Lustre: Skipped 2868 previous similar messages
[10351.024902] Lustre: *** cfs_fail_loc=a08, val=0***
[10636.261817] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10636.290439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10636.303284] Lustre: Skipped 1 previous similar message
[10640.039679] Lustre: server umount lustre-MDT0000 complete
[10645.619590] LustreError: 173035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789611692 with bad export cookie 2889418993525124096
[10645.632982] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10646.257776] Lustre: server umount lustre-OST0000 complete
[10651.287840] Lustre: server umount lustre-OST0001 complete
[10659.779836] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid
[10668.858964] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local
[10679.575772]  vdc: vdc1 vdc9
[10691.014818]  vde: vde1 vde9
[10691.041687]  vde: vde1 vde9
[10691.083654]  vde: vde1 vde9
[10702.747116]  vdf: vdf1 vdf9
[10721.658591] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local
[10732.350618] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[10732.563249] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[10732.634771] Lustre: lustre-MDT0000: new disk, initializing
[10732.977263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10733.022494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[10738.871533] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10744.139344] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[10751.786189] Lustre: lustre-OST0000: new disk, initializing
[10751.793087] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[10751.800642] Lustre: Skipped 1 previous similar message
[10751.876315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10753.599805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[10753.609686] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[10753.763882] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[10758.984709] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10770.086128] Lustre: lustre-OST0001: new disk, initializing
[10770.096854] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[10770.223744] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10771.926767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[10771.938645] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[10772.153065] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[10778.261707] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10788.760663] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10792.775394] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[10831.017317] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 22:24:35 (1789611875)
[10837.735735] Lustre: DEBUG MARKER: == sanity-quota test 49c: lfs quota long options don't consume an extra argument ========================================================== 22:24:42 (1789611882)
[10845.168815] Lustre: DEBUG MARKER: == sanity-quota test 49d: lfs quota -d and --delimiter both work ========================================================== 22:24:49 (1789611889)
[10851.359074] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 22:24:55 (1789611895)
[10907.000486] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 22:25:50 (1789611950)
[10975.236165] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 22:26:59 (1789612019)
[11005.431995] Lustre: DEBUG MARKER: rename directory return 255
[11048.601899] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 22:28:13 (1789612093)
[11079.979346] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 22:28:44 (1789612124)
[11124.919527] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 22:29:29 (1789612169)
[11231.103144] LustreError: 190324:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11268.360694] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 22:31:53 (1789612313)
[11309.230808] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 22:32:33 (1789612353)
[11357.743419] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 22:33:22 (1789612402)
[11546.326499] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 22:36:30 (1789612590)
[11548.173672] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[11549.789110] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 22:36:34 (1789612594)
[11614.417527] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[11616.185475] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 22:37:40 (1789612660)
[11648.835484] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[11650.784183] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 22:38:15 (1789612695)
[11705.648557] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[11707.314341] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 22:39:11 (1789612751)
[11756.587946] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 22:40:00 (1789612800)
[11758.539317] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[11760.725577] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 22:40:04 (1789612804)
[11790.353787] LustreError: 213849:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[11839.379814] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 22:41:23 (1789612883)
[11868.934427] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[11871.153884] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[12084.673278] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 22:45:28 (1789613128)
[12137.012583] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 22:46:20 (1789613180)
[12171.459170] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 22:46:55 (1789613215)
[12173.963247] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[12176.331960] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 22:47:00 (1789613220)
[12178.263952] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[12179.962075] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 22:47:04 (1789613224)
[12197.327158] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[12215.056792] Lustre: DEBUG MARKER: Write...
[12217.803128] Lustre: DEBUG MARKER: Write out of block quota ...
[12279.496503] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 22:48:43 (1789613323)
[12312.069288] Lustre: DEBUG MARKER: set to use default quota
[12313.863443] Lustre: DEBUG MARKER: set default quota
[12315.768590] Lustre: DEBUG MARKER: get default quota
[12324.003561] Lustre: DEBUG MARKER: Test not out of quota
[12328.576375] Lustre: DEBUG MARKER: Test out of quota
[12346.386491] Lustre: DEBUG MARKER: Increase default quota
[12381.796623] Lustre: DEBUG MARKER: Set quota to override default quota
[12381.869699] LustreError: 192588:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1790218228 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12395.729447] Lustre: DEBUG MARKER: Set to use default quota again
[12417.958952] Lustre: DEBUG MARKER: Cleanup
[12495.692842] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 22:52:19 (1789613539)
[12524.933144] Lustre: DEBUG MARKER: set default quota for qpool1
[12527.229799] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[12577.780432] LustreError: 3308:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[12581.287715] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 22:53:45 (1789613625)
[12669.041425] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 22:55:13 (1789613713)
[12747.640546] Lustre: DEBUG MARKER: Write...
[12751.021595] Lustre: DEBUG MARKER: Write out of block quota ...
[12870.267424] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 22:58:34 (1789613914)
[12919.877909] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 22:59:24 (1789613964)
[12927.242647] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 22:59:31 (1789613971)
[12929.863831] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A fallocate not supported
[12932.165391] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 22:59:36 (1789613976)
[12935.532216] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a fallocate not supported
[12937.545607] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 22:59:41 (1789613981)
[12961.667812] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 23:00:05 (1789614005)
[12963.958983] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[12965.745207] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 23:00:10 (1789614010)
[12982.530421] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[12990.185408] LustreError: 239058:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[12994.017050] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[12994.041899] Lustre: Skipped 1 previous similar message
[12994.049153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[12994.055698] Lustre: Skipped 1 previous similar message
[12997.192920] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.41@tcp (stopping)
[12997.601301] LustreError: 3307:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff9048771f5c00 x1876532824049024/t0(0) o601->lustre-MDT0000-lwp-MDT0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[12997.623513] LustreError: 3307:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-MDT0000 qtype:prj lqe: ffff904750eca900 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 374 qunit:0 qtune:0 edquot:0 default:no revoke:0
[12997.638995] LustreError: 3307:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 2 previous similar messages
[12999.138753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[12999.148566] Lustre: Skipped 1 previous similar message
[13000.200409] LustreError: 239058:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[13000.571266] Lustre: server umount lustre-MDT0000 complete
[13009.686504] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[13010.386988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[13010.538261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[13010.560212] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[13015.818912] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[13024.741280] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[13024.755418] Lustre: Skipped 3 previous similar messages
[13052.848235] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 23:01:37 (1789614097)
[13090.291741] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 23:02:14 (1789614134)
[13100.970752] LustreError: 3308:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[13123.616189] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 23:02:48 (1789614168)
[13174.688973] Lustre: *** cfs_fail_loc=a08, val=0***
[13174.697048] Lustre: Skipped 13 previous similar messages
[13174.710879] Lustre: *** cfs_fail_loc=a08, val=0***
[13174.717725] Lustre: Skipped 15 previous similar messages
[13299.900290] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 23:05:44 (1789614344)
[13386.117720] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 23:07:10 (1789614430)
[13474.613487] LustreError: 241773:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1790219321 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13483.120402] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed