[ 4764.799223] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 4769.947963] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 4772.586470] LustreError: 126328:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 4773.316984] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4774.777533] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 4776.909348] Lustre: Failing over lustre-MDT0000
[ 4777.453792] Lustre: server umount lustre-MDT0000 complete
[ 4793.823154] Lustre: 3322:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782485211/real 1782485211]  req@ffff92a1c7812d80 x1869066362104704/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1782485227 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 4793.823414] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4793.842219] Lustre: 3322:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4793.859820] LustreError: MGC192.168.201.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4797.983269] Lustre: 3320:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782485215/real 1782485215]  req@ffff92a1c7811500 x1869066362104960/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782485231 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4798.006586] Lustre: 3320:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4803.168448] Lustre: 3320:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782485220/real 1782485220]  req@ffff92a0c8177b80 x1869066362105472/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782485236 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4803.191647] Lustre: 3320:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 4804.564431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4804.659685] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4807.678926] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 4807.943132] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 4807.988693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:676 to 0x240000400:705)
[ 4807.994255] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:672 to 0x280000400:705)
[ 4808.524226] Lustre: DEBUG MARKER: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4809.509752] Lustre: 3322:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782485226/real 1782485226]  req@ffff92a0c8174000 x1869066362105728/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782485242 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4809.530588] Lustre: 3322:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 4817.521653] Lustre: DEBUG MARKER: oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4818.919117] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 4818.950380] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4837.871059] Lustre: DEBUG MARKER: (dd_pid=117469, time=12, timeout=600)
[ 4879.715670] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 4885.246595] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 4887.929930] LustreError: 129163:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 4888.631120] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4889.915037] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 4891.426630] Lustre: Failing over lustre-MDT0000
[ 4891.572507] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 4891.576332] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4891.581530] Lustre: Skipped 1 previous similar message
[ 4891.583801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4891.586418] Lustre: Skipped 1 previous similar message
[ 4891.820091] Lustre: server umount lustre-MDT0000 complete
[ 4908.277417] LustreError: MGC192.168.201.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4908.499448] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4908.656703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4908.832353] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4910.099531] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 4910.269536] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 4910.303170] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:672 to 0x280000400:737)
[ 4910.305230] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:707 to 0x240000400:737)
[ 4910.694640] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 4910.698925] Lustre: Skipped 1 previous similar message
[ 4911.073385] Lustre: 3321:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782485328/real 1782485328]  req@ffff92a1fc540a80 x1869066362145408/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782485344 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4911.088299] Lustre: 3321:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 4912.204632] Lustre: DEBUG MARKER: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4919.123830] Lustre: DEBUG MARKER: oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4920.387189] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4921.743160] Lustre: 3319:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782485339/real 1782485339]  req@ffff92a10c3ea680 x1869066362146560/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782485355 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4921.774811] Lustre: 3319:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 4926.964810] Lustre: DEBUG MARKER: (dd_pid=119885, time=2, timeout=600)
[ 4977.448258] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 10:50:09 (1782485409)
[ 4995.774979] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 4996.993964] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5001.215333] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5002.238367] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5032.774694] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 10:51:05 (1782485465)
[ 5053.404456] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 5066.895896] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 5068.072369] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 5069.257294] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 5070.867189] Lustre: DEBUG MARKER: Set quota for 1 times
[ 5073.671287] Lustre: DEBUG MARKER: Set quota for 2 times
[ 5076.440621] Lustre: DEBUG MARKER: Set quota for 3 times
[ 5079.330190] Lustre: DEBUG MARKER: Set quota for 4 times
[ 5082.146592] Lustre: DEBUG MARKER: Set quota for 5 times
[ 5085.382688] Lustre: DEBUG MARKER: Set quota for 6 times
[ 5088.218905] Lustre: DEBUG MARKER: Set quota for 7 times
[ 5090.880845] Lustre: DEBUG MARKER: Set quota for 8 times
[ 5093.978185] Lustre: DEBUG MARKER: Set quota for 9 times
[ 5096.834140] Lustre: DEBUG MARKER: Set quota for 10 times
[ 5099.677718] Lustre: DEBUG MARKER: Set quota for 11 times
[ 5133.198766] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 10:52:45 (1782485565)
[ 5141.473107] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5141.479209] LustreError: 130755:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5141.480993] Lustre: Skipped 1 previous similar message
[ 5141.499419] LustreError: 130755:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message
[ 5141.534782] Lustre: server umount lustre-MDT0000 complete
[ 5144.334685] LustreError: 5793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485577 with bad export cookie 5810416026589945427
[ 5144.337414] LustreError: MGC192.168.201.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5144.339729] LustreError: 5793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 5146.494722] Lustre: server umount lustre-OST0000 complete
[ 5149.538592] Lustre: server umount lustre-OST0001 complete
[ 5159.882789] Lustre: DEBUG MARKER: oleg140-server.virtnet: executing load_modules_local
[ 5164.879126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5167.342630] Lustre: DEBUG MARKER: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5169.128918] Lustre: 138967:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5172.678657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5175.712842] LustreError: 139297:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5175.728889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:740 to 0x240000400:769)
[ 5176.294753] Lustre: DEBUG MARKER: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5180.899571] LustreError: 139297:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5181.236818] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5184.851183] Lustre: DEBUG MARKER: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5186.546866] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:739 to 0x280000400:769)
[ 5189.410525] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5191.401262] Lustre: 140792:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5207.011736] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5207.012767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5207.020950] Lustre: Skipped 1 previous similar message
[ 5207.029025] Lustre: Skipped 1 previous similar message
[ 5212.127589] LustreError: 139603:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5212.135705] LustreError: 139603:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message
[ 5212.764810] Lustre: server umount lustre-MDT0000 complete
[ 5215.109071] LustreError: 138516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485648 with bad export cookie 5810416026589954261
[ 5215.111427] LustreError: MGC192.168.201.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5215.119251] LustreError: 138516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 5215.208243] Lustre: server umount lustre-OST0000 complete
[ 5217.862853] Lustre: server umount lustre-OST0001 complete
[ 5226.404776] Lustre: DEBUG MARKER: oleg140-server.virtnet: executing load_modules_local
[ 5232.060287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5234.501731] Lustre: DEBUG MARKER: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5236.181847] Lustre: 143492:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5242.342374] LustreError: 143810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5242.364659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:740 to 0x240000400:801)
[ 5243.093969] Lustre: DEBUG MARKER: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5248.715178] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5248.718484] Lustre: Skipped 1 previous similar message
[ 5254.184542] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:739 to 0x280000400:801)
[ 5254.470932] Lustre: DEBUG MARKER: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5262.481539] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5267.590347] Lustre: 145312:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5281.956341] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 10:55:12 (1782485712)
[ 5284.361703] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 5287.983142] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 10:55:18 (1782485718)
[ 5350.901514] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 10:56:22 (1782485782)
[ 5396.258895] Lustre: DEBUG MARKER: Write...
[ 5399.060784] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5455.927332] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 10:58:07 (1782485887)
[ 5462.040484] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 10:58:14 (1782485894)
[ 5471.640847] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 10:58:23 (1782485903)
[ 5480.759748] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 10:58:32 (1782485912)
[ 5488.299402] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 10:58:40 (1782485920)
[ 5555.116693] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 5713.649752] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 5865.140739] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 11:04:57 (1782486297)
[ 5910.428295] Lustre: DEBUG MARKER: Restart...
[ 5914.592661] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5914.593855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5914.601905] Lustre: Skipped 1 previous similar message
[ 5914.610505] Lustre: Skipped 1 previous similar message
[ 5917.979429] Lustre: server umount lustre-MDT0000 complete
[ 5920.334674] LustreError: 143049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486353 with bad export cookie 5810416026589955794
[ 5920.335135] LustreError: MGC192.168.201.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5920.342803] LustreError: 143049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 5920.388762] Lustre: server umount lustre-OST0000 complete
[ 5922.510854] Lustre: server umount lustre-OST0001 complete
[ 5929.449756] Lustre: DEBUG MARKER: oleg140-server.virtnet: executing load_modules_local
[ 5933.822917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5935.919818] Lustre: DEBUG MARKER: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5937.459745] Lustre: 164068:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5940.174929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5943.680899] Lustre: DEBUG MARKER: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5944.228119] LustreError: 164390:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5944.242688] LustreError: 164390:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message
[ 5944.243733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:811 to 0x240000400:833)
[ 5950.758454] Lustre: DEBUG MARKER: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5952.999891] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:809 to 0x280000400:833)
[ 5955.436783] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5957.532782] Lustre: 165877:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6000.939945] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 11:07:13 (1782486433)
[ 6058.401798] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 11:08:10 (1782486490)
[ 6987.240685] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 11:23:40 (1782487420)
[ 6997.310276] Lustre: server umount lustre-MDT0000 complete
[ 6998.710387] LustreError: 163625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487431 with bad export cookie 5810416026589962283
[ 6998.715757] LustreError: MGC192.168.201.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7007.945281] Lustre: server umount lustre-OST0000 complete
[ 7019.041440] Lustre: server umount lustre-OST0001 complete
[ 7024.543736] Lustre: DEBUG MARKER: oleg140-server.virtnet: executing load_modules_local
[ 7027.856426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7027.860082] Lustre: Skipped 1 previous similar message
[ 7029.308754] Lustre: DEBUG MARKER: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7030.377931] Lustre: 174279:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7032.190160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7034.219415] Lustre: DEBUG MARKER: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7036.777310] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7037.801961] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5836 to 0x240000400:5857)
[ 7037.861431] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5834 to 0x280000400:5857)
[ 7039.235638] Lustre: DEBUG MARKER: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7042.627449] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7044.000197] Lustre: 176080:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7068.804216] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 11:25:01 (1782487501)
[ 7099.290774] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 11:25:32 (1782487532)
[ 7130.021168] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 11:26:02 (1782487562)
[ 7130.513388] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 7131.094641] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 11:26:03 (1782487563)
[ 7131.575335] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 7132.103064] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 11:26:04 (1782487564)
[ 7185.821561] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 11:26:58 (1782487618)
[ 7212.753515] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 11:27:25 (1782487645)
[ 7257.863372] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 11:28:10 (1782487690)
[ 7265.023762] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7265.025239] Lustre: Skipped 1 previous similar message
[ 7267.025470] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7267.026956] Lustre: Skipped 343 previous similar messages
[ 7271.026552] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7271.028614] Lustre: Skipped 699 previous similar messages
[ 7279.035215] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7279.036592] Lustre: Skipped 1387 previous similar messages
[ 7295.036154] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7295.037482] Lustre: Skipped 2843 previous similar messages
[ 7747.761899] Lustre: *** cfs_fail_loc=a08, val=0***
[ 7747.767925] Lustre: *** cfs_fail_loc=a08, val=0***
[ 7747.776747] Lustre: Skipped 2724 previous similar messages
[ 7747.799607] Lustre: Skipped 4 previous similar messages
[ 7827.625993] Lustre: *** cfs_fail_loc=a08, val=0***
[ 7827.627681] Lustre: Skipped 14 previous similar messages
[ 7827.632728] Lustre: *** cfs_fail_loc=a08, val=0***
[ 7827.648643] Lustre: Skipped 15 previous similar messages
[ 7921.633117] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7921.637396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7921.656791] Lustre: Skipped 1 previous similar message
[ 7921.690301] Lustre: Skipped 1 previous similar message
[ 7922.423328] Lustre: server umount lustre-MDT0000 complete
[ 7927.543669] LustreError: 179841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488360 with bad export cookie 5810416026591724869
[ 7927.569270] LustreError: MGC192.168.201.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7927.865374] Lustre: server umount lustre-OST0000 complete
[ 7933.158991] Lustre: server umount lustre-OST0001 complete
[ 7941.143463] Lustre: DEBUG MARKER: oleg140-server.virtnet: executing set_hostid
[ 7953.164165] Lustre: DEBUG MARKER: oleg140-server.virtnet: executing load_modules_local
[ 7965.974686]  vdc: vdc1 vdc9
[ 7978.005508]  vde: vde1 vde9
[ 7989.876957]  vdf: vdf1 vdf9
[ 7989.948459]  vdf: vdf1 vdf9
[ 7990.004390]  vdf: vdf1 vdf9
[ 8008.403703] Lustre: DEBUG MARKER: oleg140-server.virtnet: executing load_modules_local
[ 8021.964162] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 8022.182422] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 8022.256665] Lustre: lustre-MDT0000: new disk, initializing
[ 8022.813942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8022.860073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 8026.527752] Lustre: DEBUG MARKER: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8030.881711] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 8035.383682] Lustre: lustre-OST0000: new disk, initializing
[ 8035.387304] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 8035.389713] Lustre: Skipped 1 previous similar message
[ 8035.447807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 8036.918272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[ 8036.926397] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[ 8037.004233] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[ 8041.167368] Lustre: DEBUG MARKER: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8048.427879] Lustre: lustre-OST0001: new disk, initializing
[ 8048.430954] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 8048.514339] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 8049.973077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[ 8049.988408] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[ 8050.104977] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[ 8053.833512] Lustre: DEBUG MARKER: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8062.557761] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8066.206663] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 8094.879487] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 11:42:07 (1782488527)
[ 8097.622835] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 11:42:10 (1782488530)
[ 8127.589398] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 11:42:40 (1782488560)
[ 8162.158746] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 11:43:15 (1782488595)
[ 8174.688680] Lustre: DEBUG MARKER: rename directory return 255
[ 8203.153536] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 11:43:55 (1782488635)
[ 8220.849151] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 11:44:13 (1782488653)
[ 8239.797208] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 11:44:32 (1782488672)
[ 8275.295133] LustreError: 201746:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 8297.873543] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 11:45:30 (1782488730)
[ 8321.504829] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 11:45:53 (1782488753)
[ 8354.593623] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 11:46:26 (1782488786)
[ 8481.930606] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 11:48:34 (1782488914)
[ 8482.721240] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[ 8483.651534] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 11:48:36 (1782488916)
[ 8530.045675] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[ 8531.393409] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 11:49:23 (1782488963)
[ 8554.989579] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[ 8555.947408] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 11:49:48 (1782488988)
[ 8596.086448] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[ 8597.271632] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 11:50:29 (1782489029)
[ 8642.560992] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 11:51:14 (1782489074)
[ 8643.624427] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[ 8650.909747] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 11:51:23 (1782489083)
[ 8687.776297] LustreError: 214812:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[ 8743.695193] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 11:52:55 (1782489175)
[ 8765.154178] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[ 8766.692981] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[ 8949.598658] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 11:56:21 (1782489381)
[ 9006.661881] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 11:57:17 (1782489437)
[ 9045.494593] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 11:57:56 (1782489476)
[ 9047.390442] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[ 9049.878994] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 11:58:01 (1782489481)
[ 9052.116854] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[ 9054.556110] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 11:58:06 (1782489486)
[ 9072.362604] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[ 9089.194978] Lustre: DEBUG MARKER: Write...
[ 9091.670691] Lustre: DEBUG MARKER: Write out of block quota ...
[ 9155.446493] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 11:59:46 (1782489586)
[ 9186.590538] Lustre: DEBUG MARKER: set to use default quota
[ 9188.545956] Lustre: DEBUG MARKER: set default quota
[ 9190.685584] Lustre: DEBUG MARKER: get default quota
[ 9198.242439] Lustre: DEBUG MARKER: Test not out of quota
[ 9202.712503] Lustre: DEBUG MARKER: Test out of quota
[ 9216.725299] Lustre: DEBUG MARKER: Increase default quota
[ 9252.545791] Lustre: DEBUG MARKER: Set quota to override default quota
[ 9252.668117] LustreError: 191447:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1783094485 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 9265.962940] Lustre: DEBUG MARKER: Set to use default quota again
[ 9292.012630] Lustre: DEBUG MARKER: Cleanup
[ 9372.425104] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 12:03:22 (1782489802)
[ 9401.355194] Lustre: DEBUG MARKER: set default quota for qpool1
[ 9402.938298] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[ 9467.695176] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 12:04:58 (1782489898)
[ 9562.429900] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 12:06:33 (1782489993)
[ 9642.027465] Lustre: DEBUG MARKER: Write...
[ 9645.588146] Lustre: DEBUG MARKER: Write out of block quota ...
[ 9767.986684] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 12:09:59 (1782490199)
[ 9812.086443] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 12:10:43 (1782490243)
[ 9818.933081] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 12:10:50 (1782490250)
[ 9820.204660] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate
[ 9822.060852] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 12:10:53 (1782490253)
[ 9824.221937] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate
[ 9826.131451] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 12:10:57 (1782490257)
[ 9852.847112] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 12:11:24 (1782490284)
[ 9854.335834] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[ 9857.000286] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 12:11:28 (1782490288)
[ 9872.478716] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[ 9879.029841] LustreError: 239824:0:(qmt_pool.c:1413:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[ 9883.106817] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9883.110239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9883.128969] Lustre: Skipped 1 previous similar message
[ 9884.682906] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.40@tcp (stopping)
[ 9884.689762] Lustre: Skipped 1 previous similar message
[ 9888.226823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9888.232357] Lustre: Skipped 1 previous similar message
[ 9889.127138] LustreError: 239824:0:(qmt_pool.c:1413:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[ 9889.453618] Lustre: server umount lustre-MDT0000 complete
[ 9897.799886] LustreError: MGC192.168.201.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9898.709424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9899.086976] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[ 9899.091116] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[ 9905.022543] Lustre: DEBUG MARKER: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9912.803986] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 9912.812557] Lustre: Skipped 1 previous similar message
[ 9943.169378] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 12:12:54 (1782490374)
[ 9982.874470] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 12:13:34 (1782490414)
[10017.480835] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 12:14:09 (1782490449)
[10071.420156] Lustre: *** cfs_fail_loc=a08, val=0***
[10071.422976] Lustre: Skipped 11 previous similar messages
[10071.430677] Lustre: *** cfs_fail_loc=a08, val=0***
[10071.432371] Lustre: Skipped 9 previous similar messages
[10193.451362] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 12:17:04 (1782490624)
[10290.298831] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 12:18:42 (1782490722)
[10361.860630] LustreError: 240361:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1783095595 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[10500.729299] LustreError: 240361:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1783095734 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[10510.991102] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed