[ 5500.966518] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5507.257189] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5516.309845] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5518.243435] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5520.528093] Lustre: Failing over lustre-MDT0000
[ 5521.020111] Lustre: server umount lustre-MDT0000 complete
[ 5521.380884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 5521.390224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5521.403823] Lustre: Skipped 1 previous similar message
[ 5521.407341] LustreError: 6557:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5521.443552] LustreError: 6557:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages
[ 5539.296363] Lustre: 3645:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788780646/real 1788780646]  req@ffff9dc3f5c35880 x1875666771745024/t0(0) o400->MGC192.168.203.143@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788780662 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5539.326132] Lustre: 3645:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 5539.335727] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5542.371186] LDISKFS-fs (dm-0): 7 truncates cleaned up
[ 5542.378000] LDISKFS-fs (dm-0): recovery complete
[ 5542.390790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5549.538045] LustreError: 3642:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff9dc2ce12c380 x1875666771753088/t0(0) o250->MGC192.168.203.143@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5549.834723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5549.877689] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5551.245750] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 5554.208566] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5555.175669] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5555.288543] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 5555.329537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:679 to 0x280000401:705)
[ 5555.336519] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:611 to 0x2c0000401:641)
[ 5562.976687] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5564.376915] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5569.918397] Lustre: DEBUG MARKER: (dd_pid=130687, time=0, timeout=600)
[ 5596.627453] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5603.239753] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5613.057453] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5614.900853] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5617.704235] Lustre: Failing over lustre-MDT0000
[ 5618.039511] LustreError: 6553:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5618.075110] LustreError: 6553:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 31 previous similar messages
[ 5618.201949] Lustre: server umount lustre-MDT0000 complete
[ 5638.111558] Lustre: 3646:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788780744/real 1788780744]  req@ffff9dc3ff248000 x1875666771807232/t0(0) o400->MGC192.168.203.143@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788780760 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5638.146775] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5641.425521] LDISKFS-fs (dm-0): 3 truncates cleaned up
[ 5641.428519] LDISKFS-fs (dm-0): recovery complete
[ 5641.437344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5647.402711] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb25bfe15d90ddff9
[ 5647.754735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5647.826775] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5648.728377] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 5652.971902] LustreError: 3642:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-lwp-OST0001: namespace resource [0x200000006:0x20000:0x0].0x0 (ffff9dc3c4c70900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5653.010804] LustreError: 3642:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 5 previous similar messages
[ 5653.055780] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted.
[ 5653.096899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:707 to 0x280000401:737)
[ 5653.097538] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:611 to 0x2c0000401:673)
[ 5653.638428] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5663.520380] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5665.193160] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5671.029553] Lustre: DEBUG MARKER: (dd_pid=133128, time=0, timeout=600)
[ 5708.846984] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 07:33:49 (1788780829)
[ 5760.430672] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 07:34:41 (1788780881)
[ 5786.166536] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 5798.158377] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 5799.685668] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 5801.154957] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 5803.203498] Lustre: DEBUG MARKER: Set quota for 1 times
[ 5806.294095] Lustre: DEBUG MARKER: Set quota for 2 times
[ 5809.790695] Lustre: DEBUG MARKER: Set quota for 3 times
[ 5813.677662] Lustre: DEBUG MARKER: Set quota for 4 times
[ 5817.173028] Lustre: DEBUG MARKER: Set quota for 5 times
[ 5821.248166] Lustre: DEBUG MARKER: Set quota for 6 times
[ 5825.140395] Lustre: DEBUG MARKER: Set quota for 7 times
[ 5828.621025] Lustre: DEBUG MARKER: Set quota for 8 times
[ 5832.188773] Lustre: DEBUG MARKER: Set quota for 9 times
[ 5864.526687] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 07:36:25 (1788780985)
[ 5883.365836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 5883.366452] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5883.374704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5883.390791] Lustre: Skipped 8 previous similar messages
[ 5885.176546] Lustre: server umount lustre-MDT0000 complete
[ 5888.494422] LustreError: 8778:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5888.519694] LustreError: 8778:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 37 previous similar messages
[ 5888.781877] LustreError: 34673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788781011 with bad export cookie 12852145331421175801
[ 5888.783315] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5888.798017] LustreError: 34673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 5889.102236] Lustre: server umount lustre-MDT0001 complete
[ 5902.923908] Lustre: server umount lustre-OST0000 complete
[ 5916.985307] Lustre: server umount lustre-OST0001 complete
[ 5932.950473] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local
[ 5945.394979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5946.062604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5950.487771] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5959.440922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5959.989424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 5964.947483] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5968.169634] Lustre: 157833:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5975.415147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5975.949412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5983.190210] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5990.328996] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:739 to 0x280000401:769)
[ 5992.384690] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5992.714204] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5998.104458] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:97)
[ 5998.107513] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:676 to 0x2c0000401:705)
[ 5999.932463] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6007.350996] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6011.352772] Lustre: 159709:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6028.768844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6028.775680] Lustre: Skipped 6 previous similar messages
[ 6033.900720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6033.908992] Lustre: Skipped 3 previous similar messages
[ 6034.426942] Lustre: server umount lustre-MDT0000 complete
[ 6038.748790] LustreError: 156659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788781161 with bad export cookie 12852145331421186749
[ 6038.753217] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6038.774831] LustreError: 156659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 6039.092855] Lustre: server umount lustre-MDT0001 complete
[ 6046.130636] Lustre: server umount lustre-OST0000 complete
[ 6050.776030] Lustre: server umount lustre-OST0001 complete
[ 6066.666255] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local
[ 6077.422229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6077.995199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6081.647913] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6089.775458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6094.950281] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6097.501522] Lustre: 163446:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6104.529603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6112.246114] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6122.450182] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6122.470960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:739 to 0x280000401:801)
[ 6122.787710] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6122.794104] Lustre: Skipped 2 previous similar messages
[ 6128.109940] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:676 to 0x2c0000401:737)
[ 6128.157142] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:129)
[ 6128.971248] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6136.475819] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6140.000352] Lustre: 165315:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6149.467905] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 07:41:10 (1788781270)
[ 6151.103693] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 6144
[ 6152.722539] Lustre: DEBUG MARKER: run for 4MB test file
[ 6163.098678] Lustre: DEBUG MARKER: User quota (limit: 4 MB)
[ 6169.036980] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 6170.437628] Lustre: DEBUG MARKER: Write half of file
[ 6172.223898] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6174.075816] Lustre: DEBUG MARKER: Step1: done
[ 6175.780700] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 6177.352976] Lustre: DEBUG MARKER: Step2: done
[ 6202.774560] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 61440
[ 6204.158775] Lustre: DEBUG MARKER: run for 40MB test file
[ 6215.314318] Lustre: DEBUG MARKER: User quota (limit: 40 MB)
[ 6221.756608] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 6223.162739] Lustre: DEBUG MARKER: Write half of file
[ 6226.393625] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6229.122790] Lustre: DEBUG MARKER: Step1: done
[ 6230.631491] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 6232.104528] Lustre: DEBUG MARKER: Step2: done
[ 6278.097900] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 07:43:17 (1788781397)
[ 6321.335518] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 07:44:02 (1788781442)
[ 6367.447975] Lustre: DEBUG MARKER: Write...
[ 6369.858685] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6418.717794] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 07:45:39 (1788781539)
[ 6425.395616] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 07:45:45 (1788781545)
[ 6435.373026] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 07:45:56 (1788781556)
[ 6444.490179] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 07:46:05 (1788781565)
[ 6454.060872] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 07:46:14 (1788781574)
[ 6507.874239] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6636.875842] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 6778.741756] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 07:51:39 (1788781899)
[ 6830.629618] Lustre: DEBUG MARKER: Restart...
[ 6837.730645] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 6837.736538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6837.753395] Lustre: Skipped 8 previous similar messages
[ 6837.762528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6839.779459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6841.788472] Lustre: server umount lustre-MDT0000 complete
[ 6844.911151] LustreError: 162301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6844.936116] LustreError: 162301:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 21 previous similar messages
[ 6845.820761] LustreError: 166050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788781968 with bad export cookie 12852145331421189045
[ 6845.826922] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6845.847235] LustreError: 166050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages
[ 6846.201457] Lustre: server umount lustre-MDT0001 complete
[ 6859.541442] Lustre: server umount lustre-OST0000 complete
[ 6874.083663] LustreError: 3643:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff9dc2c77f8a80 x1875666772542848/t0(0) o601->lustre-MDT0000-lwp-OST0001@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 6874.129953] LustreError: 3643:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0001 qtype:usr lqe: ffff9dc4007af680 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 1596 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 6875.069875] Lustre: server umount lustre-OST0001 complete
[ 6891.540545] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local
[ 6903.041897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6903.360669] LustreError: 188562:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6903.485051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6908.435669] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6918.899923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6919.712747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 6925.861636] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6929.787270] Lustre: 189707:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6937.118444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6937.510517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6944.478955] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6948.784517] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:813 to 0x280000401:833)
[ 6952.928219] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6958.581923] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:745 to 0x2c0000401:769)
[ 6958.596620] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:161)
[ 6960.264676] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6968.224322] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6972.142293] Lustre: 191580:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7031.171553] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 07:55:52 (1788782152)
[ 7089.502887] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 07:56:50 (1788782210)
[ 8631.227700] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 08:22:32 (1788783752)
[ 8640.981820] Lustre: server umount lustre-MDT0000 complete
[ 8640.998613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8641.005418] LustreError: 189937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8641.010784] Lustre: Skipped 4 previous similar messages
[ 8641.049922] LustreError: 189937:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages
[ 8644.770243] LustreError: 195940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788783767 with bad export cookie 12852145331421198159
[ 8644.777319] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8644.785532] LustreError: 195940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 8645.231621] Lustre: server umount lustre-MDT0001 complete
[ 8658.614345] Lustre: server umount lustre-OST0000 complete
[ 8661.474454] Lustre: 3644:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788783768/real 1788783768]  req@ffff9dc2c6b01880 x1875666775964928/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788783784 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 8662.657871] Lustre: server umount lustre-OST0001 complete
[ 8678.894908] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local
[ 8688.318798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8688.721610] LustreError: 199184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8688.863432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8688.875239] Lustre: Skipped 1 previous similar message
[ 8693.097638] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8701.096543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8701.480342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 8705.346771] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8708.178583] Lustre: 200329:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 8714.852964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 8715.184628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 8721.019900] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8726.438808] LustreError: 200682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8726.462139] LustreError: 200682:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages
[ 8729.445270] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 8733.949544] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5770 to 0x2c0000401:5793)
[ 8733.951353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5836 to 0x280000401:5857)
[ 8733.954341] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:193)
[ 8737.163216] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8744.582710] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8748.831698] Lustre: 202202:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 8778.174802] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 08:24:58 (1788783898)
[ 8806.509612] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 08:25:27 (1788783927)
[ 8836.175716] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 08:25:57 (1788783957)
[ 8874.147828] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 08:26:34 (1788783994)
[ 8915.003894] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 08:27:15 (1788784035)
[ 8996.938171] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 08:28:37 (1788784117)
[ 9032.312338] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 08:29:12 (1788784152)
[ 9100.994381] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 08:30:21 (1788784221)
[ 9106.517904] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9106.521295] Lustre: Skipped 2 previous similar messages
[ 9107.518526] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9107.520541] Lustre: Skipped 41 previous similar messages
[ 9109.519880] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9109.524979] Lustre: Skipped 72 previous similar messages
[ 9113.560844] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9113.567362] Lustre: Skipped 186 previous similar messages
[ 9121.573588] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9121.579658] Lustre: Skipped 381 previous similar messages
[ 9137.595552] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9137.606489] Lustre: Skipped 651 previous similar messages
[ 9169.598054] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9169.601129] Lustre: Skipped 1558 previous similar messages
[ 9233.599537] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9233.602882] Lustre: Skipped 2884 previous similar messages
[ 9544.550183] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9544.553991] Lustre: Skipped 2219 previous similar messages
[ 9544.578319] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9628.130138] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 9628.139583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9628.151735] Lustre: Skipped 3 previous similar messages
[ 9628.162923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9628.169133] Lustre: Skipped 3 previous similar messages
[ 9630.184414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9634.887123] Lustre: server umount lustre-MDT0000 complete
[ 9635.301861] LustreError: 199185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9635.322231] LustreError: 199185:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages
[ 9638.256101] LustreError: 199166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788784761 with bad export cookie 12852145331422974661
[ 9638.262397] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9638.273612] LustreError: 199166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 9640.418277] LustreError: 201509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9640.423737] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9640.428605] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 9640.428614] Lustre: Skipped 3 previous similar messages
[ 9640.453451] LustreError: 201509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message
[ 9640.497021] Lustre: Skipped 4 previous similar messages
[ 9643.424731] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 9643.433688] Lustre: Skipped 2 previous similar messages
[ 9644.804354] Lustre: server umount lustre-MDT0001 complete
[ 9649.437325] Lustre: server umount lustre-OST0000 complete
[ 9654.067987] Lustre: server umount lustre-OST0001 complete
[ 9661.291527] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid
[ 9670.041878] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local
[ 9716.085884] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local
[ 9726.877833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9727.143844] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 9727.178268] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 9727.309373] Lustre: lustre-MDT0000: new disk, initializing
[ 9727.410129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9727.419448] Lustre: Skipped 1 previous similar message
[ 9727.435716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 9732.108671] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9742.249990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9742.402459] Lustre: 219142:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17
[ 9742.446393] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
[ 9742.454657] Lustre: Skipped 1 previous similar message
[ 9742.566829] Lustre: lustre-MDT0001: new disk, initializing
[ 9742.657133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 9742.703266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt
[ 9742.707263] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt]
[ 9747.086996] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9751.861454] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 9758.656394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 9758.949786] Lustre: lustre-OST0000: new disk, initializing
[ 9758.954823] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 9758.964716] Lustre: 220770:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0]
[ 9759.053246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9760.262774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost
[ 9760.273482] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost]
[ 9760.351454] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401
[ 9765.432851] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9776.409178] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 9776.550930] Lustre: lustre-OST0001: new disk, initializing
[ 9776.555175] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 9776.560770] Lustre: 221641:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0]
[ 9776.626860] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9778.553722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost
[ 9778.562913] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost]
[ 9778.661085] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401
[ 9783.330691] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9792.670974] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9796.544855] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 9821.748527] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 08:42:22 (1788784942)
[ 9829.159193] Lustre: DEBUG MARKER: == sanity-quota test 49c: lfs quota long options don't consume an extra argument ========================================================== 08:42:29 (1788784949)
[ 9837.144984] Lustre: DEBUG MARKER: == sanity-quota test 49d: lfs quota -d and --delimiter both work ========================================================== 08:42:38 (1788784958)
[ 9844.730503] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 08:42:45 (1788784965)
[ 9875.751128] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 08:43:16 (1788784996)
[ 9922.031951] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 08:44:02 (1788785042)
[ 9941.188623] Lustre: DEBUG MARKER: rename directory return 255
[ 9982.261875] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 08:45:03 (1788785103)
[10002.662880] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 08:45:23 (1788785123)
[10031.304362] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 08:45:52 (1788785152)
[10133.974584] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 08:47:35 (1788785255)
[10161.053533] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 08:48:01 (1788785281)
[10193.732327] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 08:48:34 (1788785314)
[10326.558323] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 08:50:47 (1788785447)
[10331.618836] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10331.620915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[10331.632640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10331.657508] Lustre: Skipped 3 previous similar messages
[10336.738954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10336.743558] Lustre: Skipped 5 previous similar messages
[10338.917830] Lustre: server umount lustre-MDT0000 complete
[10341.858136] LustreError: 219152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10341.882202] LustreError: 219152:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages
[10342.087144] LustreError: 222406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788785464 with bad export cookie 12852145331423346851
[10342.097057] LustreError: 222406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message
[10342.102611] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10342.550982] Lustre: server umount lustre-MDT0001 complete
[10356.289953] Lustre: server umount lustre-OST0000 complete
[10370.972732] Lustre: server umount lustre-OST0001 complete
[10391.595141] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local
[10401.978624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[10402.422419] LustreError: 239314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10402.516513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10402.534466] LustreError: lustre-MDT0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[10407.904766] LustreError: 239315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10408.008379] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10413.029943] LustreError: 239314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10416.316658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[10416.726981] LustreError: lustre-MDT0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[10421.151128] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10423.753678] Lustre: 240458:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[10429.648787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[10429.963154] LustreError: lustre-OST0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[10435.740819] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10441.200771] LustreError: 240810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10443.706363] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[10443.861369] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10443.866158] Lustre: Skipped 2 previous similar messages
[10443.870113] LustreError: lustre-OST0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[10447.983403] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:65)
[10447.987302] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:20 to 0x2c0000401:65)
[10449.386721] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10456.539422] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10459.633783] Lustre: 242330:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[10466.488304] LustreError: 239310:0:(osd_handler.c:3425:osd_quota_transfer()) lustre-MDT0000: quota transfer failed. Is project enforcement enabled on the ldiskfs filesystem? rc = -95
[10472.928159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[10472.933704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10472.947979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10476.808228] Lustre: server umount lustre-MDT0000 complete
[10479.072450] LustreError: 239315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10479.086580] LustreError: 239315:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages
[10480.504198] LustreError: 239296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788785603 with bad export cookie 12852145331423395704
[10480.511741] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10480.526817] LustreError: 239296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages
[10480.877884] Lustre: server umount lustre-MDT0001 complete
[10490.849846] LustreError: 3646:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff9dc2c93adf80 x1875666783067136/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[10490.882873] LustreError: 3646:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0000 qtype:usr lqe: ffff9dc2ce125c80 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 1516 qunit:0 qtune:0 edquot:0 default:no revoke:0
[10495.248575] Lustre: server umount lustre-OST0000 complete
[10507.231517] LustreError: 3646:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff9dc2ce4b4e00 x1875666783068032/t0(0) o601->lustre-MDT0000-lwp-OST0001@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[10507.288418] LustreError: 3645:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0001 qtype:usr lqe: ffff9dc2cb709380 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 1516 qunit:0 qtune:0 edquot:0 default:no revoke:0
[10507.329066] LustreError: 3645:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 1 previous similar message
[10509.961715] Lustre: server umount lustre-OST0001 complete
[10529.044616] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local
[10539.529032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[10539.952355] LustreError: 244721:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10540.035585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10544.027280] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10552.376698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[10556.977385] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10559.824311] Lustre: 245867:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[10565.930316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[10574.143817] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10583.383972] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[10584.791169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97)
[10584.805359] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:20 to 0x2c0000401:97)
[10590.759991] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10598.491671] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10602.293920] Lustre: 247744:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[10632.883325] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 08:55:53 (1788785753)
[10683.466304] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[10685.299665] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 08:56:46 (1788785806)
[10705.575244] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[10707.393279] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 08:57:08 (1788785828)
[10741.853050] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[10744.000204] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 08:57:44 (1788785864)
[10772.406854] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 08:58:13 (1788785893)
[10782.629673] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[10787.643478] Lustre: 254306:0:(qsd_reint.c:249:qsd_reint_index()) lustre-OST0000: index version for fid [0x200000005:0x1008:0x0] is 0, but index isn't empty (1)
[10787.676438] Lustre: 254306:0:(qsd_reint.c:249:qsd_reint_index()) Skipped 1 previous similar message
[10792.030282] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[10796.978175] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[10800.083102] Lustre: DEBUG MARKER: Write...
[10818.472572] LustreError: 255652:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[10824.098864] Lustre: DEBUG MARKER: Write...
[10836.591623] Lustre: DEBUG MARKER: Write...
[10913.432119] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 09:00:34 (1788786034)
[10941.476713] LustreError: 259725:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[10988.286702] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 09:01:48 (1788786108)
[11014.913752] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[11017.073336] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[11104.122186] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 09:03:44 (1788786224)
[11151.732167] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 09:04:32 (1788786272)
[11190.584301] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 09:05:11 (1788786311)
[11203.838520] Lustre: DEBUG MARKER: User quota (block hardlimit:100 MB)
[11228.748294] Lustre: DEBUG MARKER: Write...
[11230.805568] Lustre: DEBUG MARKER: Write out of block quota ...
[11320.108312] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 09:07:20 (1788786440)
[11333.345816] Lustre: DEBUG MARKER: User quota (block hardlimit:1000 MB)
[11415.690290] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 09:08:56 (1788786536)
[11427.737786] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[11446.823843] Lustre: DEBUG MARKER: Write...
[11449.111722] Lustre: DEBUG MARKER: Write out of block quota ...
[11449.635478] LustreError: 244719:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:10240 soft:0 granted:10240 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11504.148555] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 09:10:24 (1788786624)
[11530.799507] Lustre: DEBUG MARKER: set to use default quota
[11532.470318] Lustre: DEBUG MARKER: set default quota
[11534.294474] Lustre: DEBUG MARKER: get default quota
[11540.715301] Lustre: DEBUG MARKER: Test not out of quota
[11544.425106] Lustre: DEBUG MARKER: Test out of quota
[11554.504740] Lustre: DEBUG MARKER: Increase default quota
[11580.744528] Lustre: DEBUG MARKER: Set quota to override default quota
[11580.822936] LustreError: 247016:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45056 time:1789391503 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11594.231855] Lustre: DEBUG MARKER: Set to use default quota again
[11613.651704] Lustre: DEBUG MARKER: Cleanup
[11672.901720] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 09:13:13 (1788786793)
[11694.005207] Lustre: DEBUG MARKER: set default quota for qpool1
[11695.582854] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[11739.392287] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 09:14:19 (1788786859)
[11822.650378] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 09:15:43 (1788786943)
[11891.030709] Lustre: DEBUG MARKER: Write...
[11894.682547] Lustre: DEBUG MARKER: Write out of block quota ...
[11999.064073] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 09:18:40 (1788787120)
[12035.211548] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 09:19:15 (1788787155)
[12045.181025] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 09:19:25 (1788787165)
[12088.662442] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 09:20:09 (1788787209)
[12128.794257] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 09:20:49 (1788787249)
[12153.357838] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 09:21:13 (1788787273)
[12182.935536] Lustre: *** cfs_fail_loc=a06, val=0***
[12182.949189] Lustre: Skipped 15 previous similar messages
[12183.514231] LustreError: 244735:0:(qmt_lock.c:479:qmt_lvbo_update()) $$$ failed to release quota space on glimpse 0!=2048 : rc = -11
                 qmt:lustre-QMT0000 pool:dt-0x0 id:60000 enforced:1 hard:102400 soft:0 granted:25600 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no
[12186.474268] Lustre: DEBUG MARKER: Files for user (quota_usr), count=5:
[12188.920324] Lustre: DEBUG MARKER: sanity-quota test_80: @@@@@@ FAIL: write failed