[ 5676.953044] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5682.464743] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5685.818042] LustreError: 120464:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5686.799404] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5688.695092] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5690.723764] Lustre: Failing over lustre-MDT0000
[ 5690.906716] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 5690.914659] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5690.933246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5691.088112] Lustre: server umount lustre-MDT0000 complete
[ 5707.563933] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5707.745534] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5707.760485] LustreError: 121065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5707.774844] LustreError: 121065:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message
[ 5708.150993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5708.237641] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5709.024122] Lustre: 3290:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784191725/real 1784191725]  req@ffff9655ad899880 x1870854713840640/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784191741 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5709.069027] Lustre: 3290:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5709.102545] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5709.303343] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5709.468765] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5709.529443] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:656 to 0x280000400:673)
[ 5709.530503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:659 to 0x240000400:705)
[ 5712.037129] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5713.620171] Lustre: 3289:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784191730/real 1784191730]  req@ffff965591783480 x1870854713841536/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784191746 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5718.499664] Lustre: 3290:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784191735/real 1784191735]  req@ffff9655b4220a80 x1870854713842048/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784191751 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5720.589809] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5722.201793] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5728.090462] Lustre: DEBUG MARKER: (dd_pid=114081, time=1, timeout=600)
[ 5775.372095] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5782.099640] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5785.806328] LustreError: 123374:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5786.759075] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5788.566511] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5790.685244] Lustre: Failing over lustre-MDT0000
[ 5791.114105] Lustre: server umount lustre-MDT0000 complete
[ 5807.583199] Lustre: 3289:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784191824/real 1784191824]  req@ffff9654c8acce00 x1870854713879168/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784191840 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5807.586228] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5807.633264] Lustre: 3289:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5807.662290] Lustre: Skipped 1 previous similar message
[ 5807.666285] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5816.287207] Lustre: 123686:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784191833/real 1784191833]  req@ffff965495660700 x1870854713879808/t0(0) o101->lustre-MDT0000-lwp-OST0001@0@lo:23/10 lens 456/496 e 0 to 1 dl 1784191849 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'qsd_reint_0.lus.0' uid:0 gid:0 projid:4294967295
[ 5816.325581] Lustre: 123686:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 5817.828188] LustreError: 3286:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff9655b5294000 x1870854713881984/t0(0) o250->MGC192.168.202.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5818.470104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5818.540838] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5822.268447] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5822.616287] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5822.744882] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:656 to 0x280000400:705)
[ 5822.745825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:707 to 0x240000400:737)
[ 5824.362832] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5832.771353] LustreError: 3286:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-lwp-OST0001: namespace resource [0x200000006:0x20000:0x0].0x0 (ffff9654810c3000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5832.797794] LustreError: 3286:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 5 previous similar messages
[ 5832.829777] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5832.844315] Lustre: Skipped 1 previous similar message
[ 5836.439156] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5838.181947] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5854.828071] Lustre: DEBUG MARKER: (dd_pid=116559, time=7, timeout=600)
[ 5917.591256] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 04:52:29 (1784191949)
[ 5941.219493] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5943.074639] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5949.499623] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5951.081764] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5984.424330] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 04:53:36 (1784192016)
[ 6012.515833] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 6031.534936] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 6033.249158] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 6034.975119] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 6037.703846] Lustre: DEBUG MARKER: Set quota for 1 times
[ 6042.131943] Lustre: DEBUG MARKER: Set quota for 2 times
[ 6046.327930] Lustre: DEBUG MARKER: Set quota for 3 times
[ 6050.848413] Lustre: DEBUG MARKER: Set quota for 4 times
[ 6054.709641] Lustre: DEBUG MARKER: Set quota for 5 times
[ 6058.838981] Lustre: DEBUG MARKER: Set quota for 6 times
[ 6062.899437] Lustre: DEBUG MARKER: Set quota for 7 times
[ 6125.454136] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 04:55:57 (1784192157)
[ 6141.040262] Lustre: server umount lustre-MDT0000 complete
[ 6145.786366] LustreError: 8286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784192178 with bad export cookie 4925528266730946619
[ 6145.805554] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6155.337173] Lustre: server umount lustre-OST0000 complete
[ 6158.113119] Lustre: 3287:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784192175/real 1784192175]  req@ffff965591722a00 x1870854714005888/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784192191 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6158.156882] Lustre: 3287:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 6158.180180] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6159.566194] Lustre: server umount lustre-OST0001 complete
[ 6175.424418] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local
[ 6184.133989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6188.266528] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6191.035393] Lustre: 133050:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6196.415478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6202.677810] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6209.541220] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6210.549189] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:708 to 0x280000400:737)
[ 6210.554890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:739 to 0x240000400:769)
[ 6216.193707] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6223.822490] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6227.161505] Lustre: 134871:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6246.139974] Lustre: server umount lustre-MDT0000 complete
[ 6250.398995] LustreError: 132596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784192283 with bad export cookie 4925528266730953304
[ 6250.405841] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6260.798671] Lustre: server umount lustre-OST0000 complete
[ 6262.753201] Lustre: 3289:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784192279/real 1784192279]  req@ffff9655c19f1180 x1870854714047360/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784192295 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6265.137483] Lustre: server umount lustre-OST0001 complete
[ 6282.366445] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local
[ 6291.523506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6295.701781] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6298.625364] Lustre: 137473:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6311.513709] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6316.135228] LustreError: 137789:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6316.164872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:739 to 0x240000400:801)
[ 6321.713276] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6321.724744] Lustre: Skipped 1 previous similar message
[ 6326.847889] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:708 to 0x280000400:769)
[ 6329.057927] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6336.697873] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6341.755462] Lustre: 139291:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6353.195503] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 04:59:45 (1784192385)
[ 6355.244694] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6357.121410] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 04:59:49 (1784192389)
[ 6411.270459] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 05:00:43 (1784192443)
[ 6459.905810] Lustre: DEBUG MARKER: Write...
[ 6463.036794] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6517.404574] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 05:02:29 (1784192549)
[ 6525.220855] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 05:02:36 (1784192556)
[ 6535.121882] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 05:02:46 (1784192566)
[ 6544.011663] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 05:02:56 (1784192576)
[ 6552.696828] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 05:03:04 (1784192584)
[ 6615.838454] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6782.211182] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 6982.881686] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 05:10:14 (1784193014)
[ 7049.304938] Lustre: DEBUG MARKER: Restart...
[ 7053.623838] Lustre: server umount lustre-MDT0000 complete
[ 7058.257495] LustreError: 137028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784193091 with bad export cookie 4925528266730954837
[ 7058.273543] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7068.756965] Lustre: server umount lustre-OST0000 complete
[ 7069.991735] Lustre: 3287:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784193086/real 1784193086]  req@ffff9655c1ca8000 x1870854714269824/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784193102 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7070.051970] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7070.082171] Lustre: Skipped 1 previous similar message
[ 7073.166870] Lustre: server umount lustre-OST0001 complete
[ 7089.219822] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local
[ 7098.590375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7102.736275] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7106.038599] Lustre: 157759:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7111.547459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7120.238789] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7123.746055] LustreError: 158085:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7123.772973] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:810 to 0x240000400:833)
[ 7129.058772] LustreError: 158085:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7132.886058] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7137.824350] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:778 to 0x280000400:801)
[ 7141.174399] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7152.851971] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7158.373238] Lustre: 159592:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7263.800446] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 05:14:55 (1784193295)
[ 7339.781731] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 05:16:11 (1784193371)
[ 9077.547420] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 05:45:09 (1784195109)
[ 9094.631643] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9094.634746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9094.654490] Lustre: Skipped 1 previous similar message
[ 9094.669479] Lustre: Skipped 1 previous similar message
[ 9096.163761] Lustre: server umount lustre-MDT0000 complete
[ 9101.178192] LustreError: 164368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784195134 with bad export cookie 4925528266730961326
[ 9101.198896] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9101.200344] LustreError: 164368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 9101.355534] Lustre: server umount lustre-OST0000 complete
[ 9106.710824] Lustre: server umount lustre-OST0001 complete
[ 9125.203547] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local
[ 9134.872536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9139.637941] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9143.158506] Lustre: 167980:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9149.716606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9157.524573] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9159.906913] LustreError: 168305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9159.957487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5835 to 0x240000400:5857)
[ 9165.301880] LustreError: 168305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9165.997469] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9170.974358] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5803 to 0x280000400:5825)
[ 9173.332134] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9181.503434] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9185.218361] Lustre: 169807:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9229.424442] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 05:47:41 (1784195261)
[ 9274.806163] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 05:48:27 (1784195307)
[ 9321.345968] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 05:49:13 (1784195353)
[ 9323.176747] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9324.756729] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 05:49:17 (1784195357)
[ 9326.370188] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9328.160756] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 05:49:20 (1784195360)
[ 9419.097368] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 05:50:51 (1784195451)
[ 9466.397327] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 05:51:38 (1784195498)
[ 9559.666868] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 05:53:11 (1784195591)
[ 9570.574370] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9570.583416] Lustre: Skipped 2 previous similar messages
[ 9572.599846] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9572.601664] Lustre: Skipped 77 previous similar messages
[ 9576.613804] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9576.626254] Lustre: Skipped 193 previous similar messages
[ 9584.642102] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9584.645255] Lustre: Skipped 387 previous similar messages
[ 9600.654603] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9600.656523] Lustre: Skipped 785 previous similar messages
[ 9632.704736] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9632.710018] Lustre: Skipped 1481 previous similar messages
[ 9696.761148] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9696.772319] Lustre: Skipped 3283 previous similar messages
[ 9953.393180] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9953.403382] Lustre: Skipped 1790 previous similar messages
[ 9953.414092] Lustre: *** cfs_fail_loc=a08, val=0***
[10098.809462] Lustre: *** cfs_fail_loc=a08, val=0***
[10098.815481] Lustre: Skipped 15 previous similar messages
[10246.112731] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10246.132446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10246.140543] Lustre: Skipped 1 previous similar message
[10251.234229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10251.242473] Lustre: Skipped 1 previous similar message
[10251.694585] Lustre: server umount lustre-MDT0000 complete
[10255.491986] LustreError: 167535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784196288 with bad export cookie 4925528266732738584
[10255.505780] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10255.845671] Lustre: server umount lustre-OST0000 complete
[10260.007774] Lustre: server umount lustre-OST0001 complete
[10266.907951] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid
[10275.549374] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local
[10284.403552]  vdc: vdc1 vdc9
[10294.531312]  vde: vde1 vde9
[10304.125769]  vdf: vdf1 vdf9
[10319.988287] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local
[10331.643111] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[10332.194656] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[10332.354410] Lustre: lustre-MDT0000: new disk, initializing
[10333.119375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10333.211241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[10338.638395] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10343.699644] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[10350.736090] Lustre: lustre-OST0000: new disk, initializing
[10350.738481] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[10350.741862] Lustre: Skipped 1 previous similar message
[10350.842364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10358.248554] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10358.302110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[10358.313993] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[10358.485473] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[10368.005210] Lustre: lustre-OST0001: new disk, initializing
[10368.008603] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[10368.130910] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10369.957158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[10369.966510] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[10370.100806] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[10375.452572] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10387.868082] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10392.809545] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[10432.032111] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 06:07:44 (1784196464)
[10438.956750] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 06:07:50 (1784196470)
[10490.835511] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 06:08:42 (1784196522)
[10556.730701] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 06:09:48 (1784196588)
[10594.374046] Lustre: DEBUG MARKER: rename directory return 255
[10644.420780] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 06:11:16 (1784196676)
[10676.700617] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 06:11:48 (1784196708)
[10725.816690] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 06:12:37 (1784196757)
[10880.648323] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 06:15:12 (1784196912)
[10921.285546] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 06:15:53 (1784196953)
[10968.508359] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 06:16:40 (1784197000)
[11138.711914] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 06:19:30 (1784197170)
[11139.988445] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[11141.985863] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 06:19:34 (1784197174)
[11194.352766] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[11195.790187] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 06:20:28 (1784197228)
[11224.065415] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[11225.593804] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 06:20:57 (1784197257)
[11274.149223] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[11275.413559] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 06:21:47 (1784197307)
[11321.074627] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 06:22:33 (1784197353)
[11322.542521] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[11324.382597] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 06:22:36 (1784197356)
[11349.633278] LustreError: 207818:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[11390.230629] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 06:23:42 (1784197422)
[11413.584887] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[11414.849830] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[11544.223954] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 06:26:16 (1784197576)
[11586.751656] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 06:26:59 (1784197619)
[11615.737736] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 06:27:28 (1784197648)
[11616.805089] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[11618.035265] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 06:27:30 (1784197650)
[11619.103743] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[11620.499198] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 06:27:32 (1784197652)
[11634.749832] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[11648.584133] Lustre: DEBUG MARKER: Write...
[11650.710716] Lustre: DEBUG MARKER: Write out of block quota ...
[11699.856946] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 06:28:52 (1784197732)
[11721.680833] Lustre: DEBUG MARKER: set to use default quota
[11723.155250] Lustre: DEBUG MARKER: set default quota
[11724.509103] Lustre: DEBUG MARKER: get default quota
[11729.035574] Lustre: DEBUG MARKER: Test not out of quota
[11732.423241] Lustre: DEBUG MARKER: Test out of quota
[11740.611166] Lustre: DEBUG MARKER: Increase default quota
[11764.435575] Lustre: DEBUG MARKER: Set quota to override default quota
[11764.539020] LustreError: 185600:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1784802597 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11772.602696] Lustre: DEBUG MARKER: Set to use default quota again
[11787.739986] Lustre: DEBUG MARKER: Cleanup
[11848.599282] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 06:31:21 (1784197881)
[11867.786987] Lustre: DEBUG MARKER: set default quota for qpool1
[11868.945607] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[11907.519985] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 06:32:20 (1784197940)
[11974.649488] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 06:33:27 (1784198007)
[12029.555948] Lustre: DEBUG MARKER: Write...
[12031.660403] Lustre: DEBUG MARKER: Write out of block quota ...
[12124.731327] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 06:35:57 (1784198157)
[12157.772925] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 06:36:30 (1784198190)
[12162.147960] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 06:36:34 (1784198194)
[12163.092899] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate
[12164.297070] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 06:36:36 (1784198196)
[12165.321172] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate
[12206.387995] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 06:37:18 (1784198238)
[12219.943690] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 06:37:32 (1784198252)
[12220.720301] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[12221.803300] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 06:37:34 (1784198254)
[12234.366926] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[12240.977024] LustreError: 233140:0:(qmt_pool.c:1413:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[12240.982110] LustreError: 233140:0:(qmt_pool.c:1413:qmt_pool_recalc()) Skipped 3 previous similar messages
[12243.936376] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[12243.936939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[12243.943898] Lustre: Skipped 2 previous similar messages
[12249.055875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[12249.058661] Lustre: Skipped 3 previous similar messages
[12251.023364] LustreError: 233140:0:(qmt_pool.c:1413:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[12251.224239] Lustre: server umount lustre-MDT0000 complete
[12255.353773] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[12255.740606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[12255.809448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[12255.810047] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[12258.279604] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[12274.082117] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[12274.093794] Lustre: Skipped 1 previous similar message
[12296.647159] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 06:38:49 (1784198329)
[12321.566781] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 06:39:14 (1784198354)
[12345.997309] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 06:39:38 (1784198378)
[12380.978645] Lustre: *** cfs_fail_loc=a08, val=0***
[12380.981921] Lustre: Skipped 27 previous similar messages
[12380.989923] Lustre: *** cfs_fail_loc=a08, val=0***
[12380.994125] Lustre: Skipped 15 previous similar messages
[12461.643725] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 06:41:34 (1784198494)
[12527.279664] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 06:42:39 (1784198559)
[12582.865587] LustreError: 233661:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1784803415 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12590.337929] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed