[ 5480.863603] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5487.776212] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5491.024805] LustreError: 122134:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5492.138715] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5494.135557] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5496.631338] Lustre: Failing over lustre-MDT0000
[ 5497.168288] Lustre: server umount lustre-MDT0000 complete
[ 5514.209720] Lustre: 3277:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787147761/real 1787147761]  req@ffff8d9004a73100 x1873954545571200/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787147777 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5514.220480] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5514.247286] Lustre: 3277:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 5514.247354] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5515.296100] Lustre: 3277:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787147763/real 1787147763]  req@ffff8d9004a70380 x1873954545571328/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1787147779 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5515.328428] Lustre: 3277:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5518.368153] Lustre: 3280:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787147766/real 1787147766]  req@ffff8d9045538000 x1873954545571584/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787147782 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5523.489105] Lustre: 3279:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787147771/real 1787147771]  req@ffff8d9105b47800 x1873954545572096/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787147787 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5523.541476] Lustre: 3279:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5524.449443] LustreError: 3276:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff8d91306fd180 x1873954545574016/t0(0) o250->MGC192.168.203.102@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5524.958742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5525.041835] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5530.025865] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5539.302975] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5539.488805] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5539.653295] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5539.724754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:643 to 0x240000400:673)
[ 5539.728677] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:639 to 0x280000400:673)
[ 5544.378698] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5545.682257] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5557.291850] Lustre: DEBUG MARKER: (dd_pid=115228, time=6, timeout=600)
[ 5600.028328] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5606.413794] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5609.521288] LustreError: 125043:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5610.322024] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5611.885327] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5613.867069] Lustre: Failing over lustre-MDT0000
[ 5614.260276] Lustre: server umount lustre-MDT0000 complete
[ 5630.433302] Lustre: 3277:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787147878/real 1787147878]  req@ffff8d904b50f100 x1873954545611008/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1787147894 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5630.464688] Lustre: 3277:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5630.475022] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5630.494737] Lustre: Skipped 1 previous similar message
[ 5631.267789] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5641.441847] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x43a41ffcc39b40a9
[ 5641.455121] Lustre: MGC192.168.203.102@tcp: Connection restored to 0@lo (at 0@lo)
[ 5641.467408] Lustre: Skipped 1 previous similar message
[ 5642.069684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5642.129958] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5642.919267] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5643.098518] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5643.132195] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:639 to 0x280000400:705)
[ 5643.138189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:675 to 0x240000400:705)
[ 5646.319370] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5647.088735] Lustre: 3278:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787147894/real 1787147894]  req@ffff8d9042a53480 x1873954545612416/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787147910 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5647.120595] Lustre: 3278:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 5655.889506] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5657.714268] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5673.930024] Lustre: DEBUG MARKER: (dd_pid=117674, time=10, timeout=600)
[ 5737.474182] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 09:59:58 (1787147998)
[ 5758.064883] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5759.719250] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5764.464102] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5765.954965] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5800.037615] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 10:01:01 (1787148061)
[ 5823.420745] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 5839.837886] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 5841.947843] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 5843.978826] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 5846.218470] Lustre: DEBUG MARKER: Set quota for 1 times
[ 5849.758647] Lustre: DEBUG MARKER: Set quota for 2 times
[ 5854.035297] Lustre: DEBUG MARKER: Set quota for 3 times
[ 5858.000930] Lustre: DEBUG MARKER: Set quota for 4 times
[ 5861.641426] Lustre: DEBUG MARKER: Set quota for 5 times
[ 5865.341858] Lustre: DEBUG MARKER: Set quota for 6 times
[ 5868.533638] Lustre: DEBUG MARKER: Set quota for 7 times
[ 5871.962832] Lustre: DEBUG MARKER: Set quota for 8 times
[ 5875.305473] Lustre: DEBUG MARKER: Set quota for 9 times
[ 5926.860351] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 10:03:08 (1787148188)
[ 5938.145735] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5938.146427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5938.165187] Lustre: Skipped 2 previous similar messages
[ 5938.188696] Lustre: Skipped 1 previous similar message
[ 5943.268131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5943.272509] Lustre: Skipped 1 previous similar message
[ 5943.808698] Lustre: server umount lustre-MDT0000 complete
[ 5947.599054] LustreError: 5798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787148211 with bad export cookie 4874055867195670697
[ 5947.599259] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5947.609499] LustreError: 5798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 5947.718059] Lustre: server umount lustre-OST0000 complete
[ 5951.717549] Lustre: server umount lustre-OST0001 complete
[ 5966.983407] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local
[ 5977.319552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5982.103851] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5985.915893] Lustre: 134846:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5992.674458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5999.402665] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6001.830917] LustreError: 135218:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6001.857901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:707 to 0x240000400:737)
[ 6001.865884] LustreError: 135218:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message
[ 6007.266094] LustreError: 135218:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6007.801127] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6012.944572] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:708 to 0x280000400:737)
[ 6014.254400] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6021.612767] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6025.187443] Lustre: 136763:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6043.618045] Lustre: server umount lustre-MDT0000 complete
[ 6047.919400] LustreError: 134392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787148311 with bad export cookie 4874055867195677977
[ 6047.934591] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6057.542074] Lustre: server umount lustre-OST0000 complete
[ 6059.814795] Lustre: 3280:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787148307/real 1787148307]  req@ffff8d9004a72d80 x1873954545777920/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787148323 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6061.753518] Lustre: server umount lustre-OST0001 complete
[ 6075.651664] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local
[ 6084.631502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6089.092878] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6092.148735] Lustre: 139460:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6105.709820] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6109.986075] LustreError: 139831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6110.002350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:707 to 0x240000400:769)
[ 6113.785145] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6113.790160] Lustre: Skipped 1 previous similar message
[ 6118.944802] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:708 to 0x280000400:769)
[ 6119.642469] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6128.109782] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6131.704827] Lustre: 141373:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6142.520556] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 10:06:44 (1787148404)
[ 6144.012472] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6145.622728] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 10:06:47 (1787148407)
[ 6204.795806] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 10:07:46 (1787148466)
[ 6251.835620] Lustre: DEBUG MARKER: Write...
[ 6254.649929] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6319.786748] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 10:09:41 (1787148581)
[ 6327.343275] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 10:09:48 (1787148588)
[ 6338.842341] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 10:10:00 (1787148600)
[ 6348.006980] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 10:10:09 (1787148609)
[ 6355.914204] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 10:10:17 (1787148617)
[ 6420.182082] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6581.443415] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 6774.430342] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 10:17:16 (1787149036)
[ 6837.578561] Lustre: DEBUG MARKER: Restart...
[ 6841.879312] Lustre: server umount lustre-MDT0000 complete
[ 6845.807052] LustreError: 3280:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0000 qtype:usr lqe: ffff8d900365e300 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2378 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 6845.808963] LustreError: 143595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787149109 with bad export cookie 4874055867195679510
[ 6845.846907] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6856.315676] Lustre: server umount lustre-OST0000 complete
[ 6860.066509] LustreError: 3278:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0001 qtype:usr lqe: ffff8d9141a99080 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2377 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 6869.498937] Lustre: server umount lustre-OST0001 complete
[ 6884.892534] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local
[ 6895.572575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6901.105368] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6904.289468] Lustre: 160238:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6912.378294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6919.191821] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6920.549523] LustreError: 160602:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6920.590382] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:779 to 0x240000400:801)
[ 6925.806026] LustreError: 160602:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6927.696727] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6932.996909] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:777 to 0x280000400:801)
[ 6934.375076] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6942.476451] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6945.383524] Lustre: 162148:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7009.680972] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 10:21:11 (1787149271)
[ 7079.610375] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 10:22:20 (1787149340)
[ 8685.351513] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 10:49:06 (1787150946)
[ 8704.210726] Lustre: server umount lustre-MDT0000 complete
[ 8708.379643] LustreError: 166848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787150972 with bad export cookie 4874055867195685999
[ 8708.387788] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8718.942653] Lustre: server umount lustre-OST0000 complete
[ 8721.761396] Lustre: 3278:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787150969/real 1787150969]  req@ffff8d904998bb80 x1873954548972160/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787150985 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 8721.786244] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8721.795035] Lustre: Skipped 1 previous similar message
[ 8722.519637] LustreError: 3280:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0001 qtype:usr lqe: ffff8d9141a98d80 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2383 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 8722.690789] Lustre: server umount lustre-OST0001 complete
[ 8736.974975] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local
[ 8746.535497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8750.492165] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8753.050863] Lustre: 170621:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 8758.947378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 8764.992623] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8772.213449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5803 to 0x240000400:5825)
[ 8772.426601] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 8777.740789] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5803 to 0x280000400:5825)
[ 8778.149770] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8785.982720] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8789.761338] Lustre: 172543:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 8824.394673] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 10:51:26 (1787151086)
[ 8873.121066] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 10:52:14 (1787151134)
[ 8916.232629] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 10:52:57 (1787151177)
[ 8918.103228] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 8919.769813] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 10:53:01 (1787151181)
[ 8921.570812] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 8923.576654] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 10:53:04 (1787151184)
[ 9010.098223] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 10:54:31 (1787151271)
[ 9062.961779] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 10:55:24 (1787151324)
[ 9152.891869] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 10:56:54 (1787151414)
[ 9163.839468] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9163.842858] Lustre: Skipped 3 previous similar messages
[ 9165.847726] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9165.852038] Lustre: Skipped 122 previous similar messages
[ 9169.889964] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9169.898518] Lustre: Skipped 216 previous similar messages
[ 9177.902942] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9177.906747] Lustre: Skipped 435 previous similar messages
[ 9193.927656] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9193.929758] Lustre: Skipped 695 previous similar messages
[ 9225.943184] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9225.948327] Lustre: Skipped 1639 previous similar messages
[ 9289.968794] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9289.977234] Lustre: Skipped 3235 previous similar messages
[ 9699.560167] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9699.575282] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9699.575794] Lustre: Skipped 1657 previous similar messages
[ 9699.598192] Lustre: Skipped 8 previous similar messages
[ 9898.979686] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9898.981688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9899.005735] Lustre: Skipped 1 previous similar message
[ 9899.022769] Lustre: Skipped 1 previous similar message
[ 9904.089649] Lustre: server umount lustre-MDT0000 complete
[ 9908.712426] LustreError: 170176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787152172 with bad export cookie 4874055867197461206
[ 9908.725807] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9909.166498] Lustre: server umount lustre-OST0000 complete
[ 9913.262932] Lustre: server umount lustre-OST0001 complete
[ 9920.880809] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid
[ 9928.087992] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local
[ 9937.439385]  vdc: vdc1 vdc9
[ 9947.293714]  vde: vde1 vde9
[ 9957.207634]  vdf: vdf1 vdf9
[ 9957.232135]  vdf: vdf1 vdf9
[ 9973.519445] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local
[ 9983.450063] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 9983.774724] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 9983.859375] Lustre: lustre-MDT0000: new disk, initializing
[ 9984.392520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9984.483568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 9989.117071] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9994.320433] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[10003.481267] Lustre: lustre-OST0000: new disk, initializing
[10003.486398] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[10003.492890] Lustre: Skipped 1 previous similar message
[10003.667413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10005.454142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[10005.461475] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[10005.564465] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[10012.418698] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10027.851990] Lustre: lustre-OST0001: new disk, initializing
[10027.867934] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[10028.063942] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10029.447840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[10029.462397] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[10029.661263] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[10034.151106] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10042.933859] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10046.312852] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[10081.174619] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 11:12:23 (1787152343)
[10087.497918] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 11:12:29 (1787152349)
[10138.580970] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 11:13:20 (1787152400)
[10204.432866] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 11:14:26 (1787152466)
[10234.623790] Lustre: DEBUG MARKER: rename directory return 255
[10276.451648] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 11:15:38 (1787152538)
[10307.287524] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 11:16:08 (1787152568)
[10350.005426] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 11:16:51 (1787152611)
[10455.242707] LustreError: 197537:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[10494.056837] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 11:19:15 (1787152755)
[10534.182610] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 11:19:55 (1787152795)
[10579.481607] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 11:20:41 (1787152841)
[10746.985819] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 11:23:28 (1787153008)
[10748.468342] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[10749.770614] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 11:23:31 (1787153011)
[10804.340205] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[10806.061342] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 11:24:27 (1787153067)
[10837.583489] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[10839.335626] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 11:25:00 (1787153100)
[10896.336458] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[10898.055378] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 11:25:59 (1787153159)
[10948.942956] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 11:26:50 (1787153210)
[10950.736485] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[10952.476217] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 11:26:54 (1787153214)
[10980.969192] LustreError: 210168:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[11025.056268] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 11:28:06 (1787153286)
[11054.088632] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[11055.847925] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[11238.679782] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 11:31:40 (1787153500)
[11289.017435] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 11:32:30 (1787153550)
[11321.889339] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 11:33:03 (1787153583)
[11323.438107] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[11325.486560] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 11:33:06 (1787153586)
[11327.360429] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[11329.476382] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 11:33:10 (1787153590)
[11347.870588] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[11364.679597] Lustre: DEBUG MARKER: Write...
[11367.286922] Lustre: DEBUG MARKER: Write out of block quota ...
[11422.076524] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 11:34:43 (1787153683)
[11448.653443] Lustre: DEBUG MARKER: set to use default quota
[11450.845630] Lustre: DEBUG MARKER: set default quota
[11452.697854] Lustre: DEBUG MARKER: get default quota
[11458.910451] Lustre: DEBUG MARKER: Test not out of quota
[11463.510322] Lustre: DEBUG MARKER: Test out of quota
[11475.205472] Lustre: DEBUG MARKER: Increase default quota
[11504.831225] Lustre: DEBUG MARKER: Set quota to override default quota
[11504.889651] LustreError: 187380:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1787758568 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11515.741491] Lustre: DEBUG MARKER: Set to use default quota again
[11537.529309] Lustre: DEBUG MARKER: Cleanup
[11606.737455] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 11:37:48 (1787153868)
[11628.655784] Lustre: DEBUG MARKER: set default quota for qpool1
[11630.295834] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[11679.837161] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 11:39:01 (1787153941)
[11761.113907] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 11:40:22 (1787154022)
[11836.045386] Lustre: DEBUG MARKER: Write...
[11839.627694] Lustre: DEBUG MARKER: Write out of block quota ...
[11956.915321] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 11:43:38 (1787154218)
[12002.284332] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 11:44:23 (1787154263)
[12010.719788] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 11:44:32 (1787154272)
[12012.256840] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate
[12014.890559] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 11:44:35 (1787154275)
[12016.950972] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate
[12019.371185] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 11:44:40 (1787154280)
[12040.009360] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 11:45:01 (1787154301)
[12041.332114] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[12042.862858] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 11:45:04 (1787154304)
[12058.797635] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[12065.481414] LustreError: 235392:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[12065.486355] LustreError: 235392:0:(qmt_pool.c:1407:qmt_pool_recalc()) Skipped 1 previous similar message
[12069.556875] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.2@tcp (stopping)
[12072.417235] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[12072.427527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[12074.659861] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.2@tcp (stopping)
[12074.669187] Lustre: Skipped 1 previous similar message
[12075.528968] LustreError: 235392:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[12075.876520] Lustre: server umount lustre-MDT0000 complete
[12084.026368] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[12084.563254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[12084.699027] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[12084.699543] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[12089.142331] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[12092.903156] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[12092.908195] Lustre: Skipped 2 previous similar messages
[12129.750257] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 11:46:31 (1787154391)
[12158.835703] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 11:47:00 (1787154420)
[12196.476168] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 11:47:37 (1787154457)
[12251.232994] Lustre: *** cfs_fail_loc=a08, val=0***
[12251.235792] Lustre: Skipped 9 previous similar messages
[12251.253552] Lustre: *** cfs_fail_loc=a08, val=0***
[12251.259723] Lustre: Skipped 7 previous similar messages
[12356.962338] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 11:50:18 (1787154618)
[12441.497975] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 11:51:42 (1787154702)
[12516.602236] LustreError: 235974:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1787759580 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12516.676086] LustreError: 236009:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[12525.155868] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed