[ 5800.674864] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5805.758905] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5809.162278] LustreError: 125596:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5810.248789] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5812.038264] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5814.308115] Lustre: Failing over lustre-MDT0000
[ 5814.604356] Lustre: server umount lustre-MDT0000 complete
[ 5832.056782] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5832.397786] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5832.412368] Lustre: Skipped 1 previous similar message
[ 5832.635613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5832.692970] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5832.937530] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5832.996149] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5833.038332] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:628 to 0x280000400:673)
[ 5833.045752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:631 to 0x240000400:673)
[ 5833.503271] Lustre: 3308:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789503903/real 1789503903]  req@ffff98fbc19e0e00 x1876424853289088/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789503919 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5833.519908] Lustre: 3308:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5837.611254] Lustre: 3308:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789503908/real 1789503908]  req@ffff98fbca991f80 x1876424853289600/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789503924 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5837.650583] Lustre: 3308:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5837.805141] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5837.988130] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5842.917910] Lustre: 3309:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789503913/real 1789503913]  req@ffff98fbca993480 x1876424853289984/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789503929 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5842.957611] Lustre: 3309:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5848.843511] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5850.320439] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5855.292957] Lustre: DEBUG MARKER: (dd_pid=117115, time=0, timeout=600)
[ 5898.483352] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5903.717498] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5906.905421] LustreError: 128552:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5907.925561] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5909.738351] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5911.946566] Lustre: Failing over lustre-MDT0000
[ 5912.439080] Lustre: server umount lustre-MDT0000 complete
[ 5929.952197] Lustre: 3308:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789504000/real 1789504000]  req@ffff98fc06150700 x1876424853322880/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1789504016 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5929.973856] Lustre: 3308:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5929.980658] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5930.923316] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5931.019991] LustreError: 129210:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5931.043817] LustreError: 129210:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages
[ 5931.264130] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.51@tcp (not set up)
[ 5931.515520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5931.577570] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5933.210403] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5933.417325] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5933.476591] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:628 to 0x280000400:705)
[ 5933.476758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:675 to 0x240000400:705)
[ 5935.459337] LustreError: 3306:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-lwp-OST0001: namespace resource [0x200000006:0x20000:0x0].0x0 (ffff98fce0233c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5935.480447] LustreError: 3306:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 5 previous similar messages
[ 5935.495639] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5935.505492] Lustre: Skipped 1 previous similar message
[ 5936.111090] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5945.183375] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5946.780582] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5946.847962] Lustre: 3310:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789504017/real 1789504017]  req@ffff98fbc2491180 x1876424853324800/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789504033 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5946.882893] Lustre: 3310:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 5957.026456] Lustre: DEBUG MARKER: (dd_pid=119590, time=4, timeout=600)
[ 6020.512169] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 16:28:26 (1789504106)
[ 6042.398984] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6043.716614] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6048.744874] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6050.068926] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6084.213779] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 16:29:30 (1789504170)
[ 6110.083450] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 6126.193479] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 6127.769532] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 6129.527431] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 6131.181568] Lustre: DEBUG MARKER: Set quota for 1 times
[ 6134.312533] Lustre: DEBUG MARKER: Set quota for 2 times
[ 6137.553601] Lustre: DEBUG MARKER: Set quota for 3 times
[ 6140.577854] Lustre: DEBUG MARKER: Set quota for 4 times
[ 6144.446713] Lustre: DEBUG MARKER: Set quota for 5 times
[ 6149.167434] Lustre: DEBUG MARKER: Set quota for 6 times
[ 6153.920650] Lustre: DEBUG MARKER: Set quota for 7 times
[ 6157.526988] Lustre: DEBUG MARKER: Set quota for 8 times
[ 6201.321329] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 16:31:27 (1789504287)
[ 6217.191039] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6217.218160] Lustre: Skipped 1 previous similar message
[ 6217.234559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6218.899645] Lustre: server umount lustre-MDT0000 complete
[ 6222.664219] LustreError: 20189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789504309 with bad export cookie 14513500780225995154
[ 6222.685258] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6222.814730] Lustre: server umount lustre-OST0000 complete
[ 6226.685738] Lustre: server umount lustre-OST0001 complete
[ 6241.281092] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local
[ 6250.254200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6254.632654] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6257.171707] Lustre: 138336:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6263.058161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6268.668317] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6275.302373] LustreError: 138710:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6275.340580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:707 to 0x240000400:737)
[ 6276.380740] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6281.739674] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:708 to 0x280000400:737)
[ 6283.095389] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6290.078169] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6293.461335] Lustre: 140258:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6311.539927] Lustre: server umount lustre-MDT0000 complete
[ 6315.374877] LustreError: 140260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789504402 with bad export cookie 14513500780226004632
[ 6315.396911] LustreError: 140260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 6315.404374] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6325.910027] Lustre: server umount lustre-OST0000 complete
[ 6328.608853] Lustre: 3307:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789504399/real 1789504399]  req@ffff98fbc19e3800 x1876424853484544/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789504415 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6329.987854] Lustre: server umount lustre-OST0001 complete
[ 6345.392992] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local
[ 6355.881969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6360.695553] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6363.942568] Lustre: 142912:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6377.522064] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6380.131157] LustreError: 143278:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6380.181820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:707 to 0x240000400:769)
[ 6386.013810] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6386.019387] Lustre: Skipped 1 previous similar message
[ 6391.316809] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:708 to 0x280000400:769)
[ 6393.286181] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6402.104667] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6405.690602] Lustre: 144830:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6417.649426] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 16:35:03 (1789504503)
[ 6419.219954] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6421.386716] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 16:35:07 (1789504507)
[ 6481.857544] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 16:36:07 (1789504567)
[ 6531.920872] Lustre: DEBUG MARKER: Write...
[ 6534.203948] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6587.446663] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 16:37:53 (1789504673)
[ 6593.607348] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 16:38:00 (1789504680)
[ 6603.604376] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 16:38:09 (1789504689)
[ 6612.536958] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 16:38:18 (1789504698)
[ 6620.945295] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 16:38:27 (1789504707)
[ 6684.098855] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6859.191951] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 7061.314891] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 16:45:46 (1789505146)
[ 7131.040367] Lustre: DEBUG MARKER: Restart...
[ 7138.785731] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7138.787135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7138.787143] Lustre: Skipped 1 previous similar message
[ 7138.840138] Lustre: Skipped 3 previous similar messages
[ 7142.125948] Lustre: server umount lustre-MDT0000 complete
[ 7146.731037] LustreError: 147367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789505233 with bad export cookie 14513500780226006165
[ 7146.741458] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7146.742087] LustreError: 147367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 7146.869647] Lustre: server umount lustre-OST0000 complete
[ 7151.761817] Lustre: server umount lustre-OST0001 complete
[ 7169.957532] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local
[ 7181.120076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7185.909427] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7188.749976] Lustre: 163420:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7196.173824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7203.343781] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7205.356113] LustreError: 163788:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7205.381124] LustreError: 163788:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message
[ 7205.421387] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:778 to 0x240000400:801)
[ 7210.491962] LustreError: 164244:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7213.152149] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7218.178387] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:778 to 0x280000400:801)
[ 7221.120875] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7231.050489] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7234.889220] Lustre: 165344:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7314.211879] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 16:49:59 (1789505399)
[ 7392.251110] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 16:51:17 (1789505477)
[ 8987.377555] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 17:17:53 (1789507073)
[ 9001.945726] Lustre: server umount lustre-MDT0000 complete
[ 9006.061694] LustreError: 169807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789507093 with bad export cookie 14513500780226012654
[ 9006.076616] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9016.477932] Lustre: server umount lustre-OST0000 complete
[ 9019.375663] Lustre: 3309:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789507090/real 1789507090]  req@ffff98fc0c1f6300 x1876424856674432/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789507106 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 9019.403402] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9021.848873] Lustre: server umount lustre-OST0001 complete
[ 9039.596654] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local
[ 9054.781316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9059.567268] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9062.996665] Lustre: 173516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9069.538154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9076.218933] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9079.779259] LustreError: 173877:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9079.805189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5804 to 0x240000400:5825)
[ 9084.913549] LustreError: 173879:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9086.462946] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9091.599895] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5802 to 0x280000400:5825)
[ 9092.868699] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9100.679355] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9105.268903] Lustre: 175433:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9138.885477] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 17:20:24 (1789507224)
[ 9185.359185] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 17:21:11 (1789507271)
[ 9231.392875] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 17:21:57 (1789507317)
[ 9233.102110] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9235.035973] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 17:22:01 (1789507321)
[ 9236.583813] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9238.215804] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 17:22:04 (1789507324)
[ 9323.480355] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 17:23:29 (1789507409)
[ 9377.437784] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 17:24:23 (1789507463)
[ 9466.208137] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 17:25:52 (1789507552)
[ 9477.685204] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9477.690563] Lustre: Skipped 2 previous similar messages
[ 9479.710318] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9479.713799] Lustre: Skipped 85 previous similar messages
[ 9483.738593] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9483.741588] Lustre: Skipped 215 previous similar messages
[ 9491.750524] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9491.752277] Lustre: Skipped 383 previous similar messages
[ 9507.750723] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9507.752251] Lustre: Skipped 739 previous similar messages
[ 9539.764631] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9539.767349] Lustre: Skipped 1513 previous similar messages
[ 9603.771577] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9603.780377] Lustre: Skipped 2959 previous similar messages
[10013.668503] Lustre: *** cfs_fail_loc=a08, val=0***
[10013.672664] Lustre: *** cfs_fail_loc=a08, val=0***
[10013.675811] Lustre: Skipped 2100 previous similar messages
[10228.198477] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10228.200601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10228.213173] Lustre: Skipped 1 previous similar message
[10228.220420] Lustre: Skipped 1 previous similar message
[10233.260644] Lustre: server umount lustre-MDT0000 complete
[10237.298417] LustreError: 178830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789508324 with bad export cookie 14513500780227787721
[10237.311547] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10237.662609] Lustre: server umount lustre-OST0000 complete
[10242.063713] Lustre: server umount lustre-OST0001 complete
[10248.052924] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid
[10258.170868] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local
[10271.792408]  vdc: vdc1 vdc9
[10282.359515]  vde: vde1 vde9
[10293.087260]  vdf: vdf1 vdf9
[10313.440522] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local
[10326.066250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[10326.529136] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[10326.648254] Lustre: lustre-MDT0000: new disk, initializing
[10327.148597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10327.197183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[10332.846910] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10338.515363] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[10346.398724] Lustre: lustre-OST0000: new disk, initializing
[10346.411886] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[10346.420236] Lustre: Skipped 1 previous similar message
[10346.519324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10348.430128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[10348.441185] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[10348.597744] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[10353.607438] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10365.243262] Lustre: lustre-OST0001: new disk, initializing
[10365.246510] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[10365.460871] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10367.069109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[10367.084592] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[10367.302799] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[10373.878110] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10384.901897] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10389.055775] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[10430.835649] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 17:41:56 (1789508516)
[10437.911330] Lustre: DEBUG MARKER: == sanity-quota test 49c: lfs quota long options don't consume an extra argument ========================================================== 17:42:04 (1789508524)
[10445.533540] Lustre: DEBUG MARKER: == sanity-quota test 49d: lfs quota -d and --delimiter both work ========================================================== 17:42:11 (1789508531)
[10451.589408] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 17:42:17 (1789508537)
[10502.906859] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 17:43:08 (1789508588)
[10572.372823] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 17:44:17 (1789508657)
[10605.024352] Lustre: DEBUG MARKER: rename directory return 255
[10656.890948] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 17:45:43 (1789508743)
[10690.499083] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 17:46:16 (1789508776)
[10737.771403] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 17:47:03 (1789508823)
[10847.734760] LustreError: 200739:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[10887.632864] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 17:49:33 (1789508973)
[10929.507499] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 17:50:15 (1789509015)
[10981.890823] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 17:51:07 (1789509067)
[11175.006089] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 17:54:21 (1789509261)
[11176.651429] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[11178.396824] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 17:54:24 (1789509264)
[11240.678799] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[11242.712940] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 17:55:28 (1789509328)
[11275.136606] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[11277.045757] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 17:56:03 (1789509363)
[11331.570645] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[11334.689866] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 17:56:59 (1789509419)
[11384.760544] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 17:57:50 (1789509470)
[11386.156533] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[11388.139991] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 17:57:54 (1789509474)
[11419.233410] LustreError: 213638:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[11465.631453] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 17:59:12 (1789509552)
[11494.761320] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[11497.030310] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[11681.507278] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 18:02:47 (1789509767)
[11736.851805] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 18:03:42 (1789509822)
[11771.104865] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 18:04:17 (1789509857)
[11772.687762] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[11774.481256] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 18:04:20 (1789509860)
[11775.844988] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[11777.666896] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 18:04:23 (1789509863)
[11794.768776] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[11812.349432] Lustre: DEBUG MARKER: Write...
[11815.466456] Lustre: DEBUG MARKER: Write out of block quota ...
[11881.934875] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 18:06:07 (1789509967)
[11912.122753] Lustre: DEBUG MARKER: set to use default quota
[11914.231777] Lustre: DEBUG MARKER: set default quota
[11916.564280] Lustre: DEBUG MARKER: get default quota
[11923.914178] Lustre: DEBUG MARKER: Test not out of quota
[11928.386534] Lustre: DEBUG MARKER: Test out of quota
[11942.542657] Lustre: DEBUG MARKER: Increase default quota
[11972.070327] Lustre: DEBUG MARKER: Set quota to override default quota
[11972.122656] LustreError: 196347:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1790114859 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11985.068848] Lustre: DEBUG MARKER: Set to use default quota again
[12015.190965] Lustre: DEBUG MARKER: Cleanup
[12102.753953] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 18:09:48 (1789510188)
[12127.306795] Lustre: DEBUG MARKER: set default quota for qpool1
[12129.875339] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[12187.148096] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 18:11:13 (1789510273)
[12269.459768] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 18:12:35 (1789510355)
[12349.536108] Lustre: DEBUG MARKER: Write...
[12353.474744] Lustre: DEBUG MARKER: Write out of block quota ...
[12478.309972] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 18:16:04 (1789510564)
[12525.421358] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 18:16:51 (1789510611)
[12533.836468] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 18:16:59 (1789510619)
[12536.674319] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A fallocate not supported
[12538.639494] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 18:17:04 (1789510624)
[12541.127273] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a fallocate not supported
[12543.095207] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 18:17:09 (1789510629)
[12563.286965] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 18:17:29 (1789510649)
[12564.936359] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[12566.997235] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 18:17:33 (1789510653)
[12585.348696] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[12592.291349] LustreError: 239141:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[12592.304113] LustreError: 239141:0:(qmt_pool.c:1407:qmt_pool_recalc()) Skipped 4 previous similar messages
[12597.997950] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.51@tcp (stopping)
[12598.007213] Lustre: Skipped 1 previous similar message
[12599.264827] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[12599.273262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[12599.275820] Lustre: Skipped 1 previous similar message
[12600.287524] LustreError: 3309:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff98fcfc6f6d80 x1876424862486656/t0(0) o601->lustre-MDT0000-lwp-MDT0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[12600.324444] LustreError: 3309:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-MDT0000 qtype:prj lqe: ffff98fc0ab24f00 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 374 qunit:0 qtune:0 edquot:0 default:no revoke:0
[12600.340889] LustreError: 3309:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 4 previous similar messages
[12602.295164] LustreError: 239141:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[12602.695879] Lustre: server umount lustre-MDT0000 complete
[12613.015120] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[12613.761295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[12613.950779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[12613.956175] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[12618.673931] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[12619.766685] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[12619.778171] Lustre: Skipped 1 previous similar message
[12654.826468] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 18:19:00 (1789510740)
[12690.744874] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 18:19:36 (1789510776)
[12724.515454] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 18:20:10 (1789510810)
[12776.745104] Lustre: *** cfs_fail_loc=a08, val=0***
[12776.748607] Lustre: Skipped 14 previous similar messages
[12776.755599] Lustre: *** cfs_fail_loc=a08, val=0***
[12776.760954] Lustre: Skipped 15 previous similar messages
[12898.806761] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 18:23:05 (1789510985)
[12990.603712] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 18:24:36 (1789511076)
[13068.277040] LustreError: 241844:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1790115955 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13077.438546] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed