[ 5566.468711] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5574.308651] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5583.725907] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5586.339106] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5588.800226] Lustre: Failing over lustre-MDT0000
[ 5588.913029] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.56@tcp (stopping)
[ 5588.927700] Lustre: Skipped 1 previous similar message
[ 5589.515867] Lustre: server umount lustre-MDT0000 complete
[ 5605.346964] Lustre: 3648:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789137409/real 1789137409]  req@ffff9f7713e80780 x1876040795828352/t0(0) o400->MGC192.168.204.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789137425 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5605.405308] Lustre: 3648:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 5605.419082] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5615.794201] LDISKFS-fs (dm-0): 7 truncates cleaned up
[ 5615.802658] LDISKFS-fs (dm-0): recovery complete
[ 5615.823520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5630.945730] LustreError: 3646:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff9f78268d0b40 x1876040795840256/t0(0) o250->MGC192.168.204.156@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5631.222125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5631.300809] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5632.112621] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 5636.181440] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5636.664243] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 5636.693698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:664 to 0x280000401:705)
[ 5636.694043] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:628 to 0x2c0000401:673)
[ 5645.179173] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5646.537740] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5651.643248] Lustre: DEBUG MARKER: (dd_pid=127609, time=0, timeout=600)
[ 5687.149614] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5695.050261] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5704.416773] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5706.216219] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5708.024153] Lustre: Failing over lustre-MDT0000
[ 5708.260920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 5708.262100] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5708.269686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5708.296478] Lustre: Skipped 7 previous similar messages
[ 5708.305902] LustreError: 18315:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5708.328225] LustreError: 18315:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 53 previous similar messages
[ 5708.578637] Lustre: server umount lustre-MDT0000 complete
[ 5729.231894] Lustre: 3650:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789137533/real 1789137533]  req@ffff9f77135aad00 x1876040795901952/t0(0) o400->MGC192.168.204.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789137549 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5729.282735] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5732.148437] LDISKFS-fs (dm-0): 3 truncates cleaned up
[ 5732.151336] LDISKFS-fs (dm-0): recovery complete
[ 5732.160983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5739.489753] LustreError: 3646:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff9f7713f03480 x1876040795909888/t0(0) o250->MGC192.168.204.156@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5739.864583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5739.939802] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5741.118234] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 5745.119797] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5745.133433] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5745.147447] Lustre: Skipped 5 previous similar messages
[ 5745.232097] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 5745.316967] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:707 to 0x280000401:737)
[ 5745.332265] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:628 to 0x2c0000401:705)
[ 5755.647358] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5757.554607] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5763.914443] Lustre: DEBUG MARKER: (dd_pid=130044, time=0, timeout=600)
[ 5808.089356] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 10:40:24 (1789137624)
[ 5859.994811] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 10:41:16 (1789137676)
[ 5887.161735] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 5898.328856] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 5899.954510] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 5901.945339] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 5904.207382] Lustre: DEBUG MARKER: Set quota for 1 times
[ 5907.187770] Lustre: DEBUG MARKER: Set quota for 2 times
[ 5910.684658] Lustre: DEBUG MARKER: Set quota for 3 times
[ 5914.586372] Lustre: DEBUG MARKER: Set quota for 4 times
[ 5918.112764] Lustre: DEBUG MARKER: Set quota for 5 times
[ 5921.746895] Lustre: DEBUG MARKER: Set quota for 6 times
[ 5925.322383] Lustre: DEBUG MARKER: Set quota for 7 times
[ 5928.550794] Lustre: DEBUG MARKER: Set quota for 8 times
[ 5932.083487] Lustre: DEBUG MARKER: Set quota for 9 times
[ 5975.077757] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 10:43:12 (1789137792)
[ 5990.889140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5990.903599] Lustre: Skipped 4 previous similar messages
[ 5995.489735] Lustre: server umount lustre-MDT0000 complete
[ 5999.391461] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137819 with bad export cookie 16269092797325041289
[ 5999.394699] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5999.407612] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 5999.716224] Lustre: server umount lustre-MDT0001 complete
[ 6014.225196] Lustre: server umount lustre-OST0000 complete
[ 6016.288841] Lustre: 3648:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789137820/real 1789137820]  req@ffff9f783bef6d00 x1876040796101504/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789137836 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6019.062920] Lustre: server umount lustre-OST0001 complete
[ 6035.491346] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local
[ 6045.644680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6046.343669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6051.006353] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6059.754748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6060.144445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 6064.890551] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6068.194929] Lustre: 154249:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6076.404714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6076.804336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6084.035625] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6084.986970] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97)
[ 6091.236980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:739 to 0x280000401:769)
[ 6093.361437] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6093.597791] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6098.937370] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:708 to 0x2c0000401:737)
[ 6101.148930] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6111.432875] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6116.573452] Lustre: 156129:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6139.889811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6139.912251] Lustre: Skipped 2 previous similar messages
[ 6143.320866] Lustre: server umount lustre-MDT0000 complete
[ 6147.539396] LustreError: 153072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137967 with bad export cookie 16269092797325048114
[ 6147.540367] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6147.553096] LustreError: 153072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 6147.773447] Lustre: server umount lustre-MDT0001 complete
[ 6161.667075] Lustre: server umount lustre-OST0000 complete
[ 6165.471193] Lustre: 3647:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789137969/real 1789137969]  req@ffff9f783a33c000 x1876040796186112/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789137985 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6166.393565] Lustre: server umount lustre-OST0001 complete
[ 6188.431657] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local
[ 6199.248709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6199.791062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6204.259625] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6213.556147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6218.388297] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6221.411205] Lustre: 159872:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6228.616506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6235.358359] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6238.476889] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:129)
[ 6244.013069] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6244.250435] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6244.260773] Lustre: Skipped 2 previous similar messages
[ 6245.313871] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:708 to 0x2c0000401:769)
[ 6245.313991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:739 to 0x280000401:801)
[ 6250.442165] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6258.072086] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6261.585218] Lustre: 161745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6272.425360] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 10:48:09 (1789138089)
[ 6274.433801] Lustre: DEBUG MARKER: OST0_SIZE: 3605392 required: 6144
[ 6276.154266] Lustre: DEBUG MARKER: run for 4MB test file
[ 6289.204934] Lustre: DEBUG MARKER: User quota (limit: 4 MB)
[ 6296.610803] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 6298.847735] Lustre: DEBUG MARKER: Write half of file
[ 6301.028717] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6303.504795] Lustre: DEBUG MARKER: Step1: done
[ 6305.356522] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 6307.508688] Lustre: DEBUG MARKER: Step2: done
[ 6334.456509] Lustre: DEBUG MARKER: OST0_SIZE: 3605392 required: 61440
[ 6336.568622] Lustre: DEBUG MARKER: run for 40MB test file
[ 6350.550588] Lustre: DEBUG MARKER: User quota (limit: 40 MB)
[ 6359.195987] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 6361.495074] Lustre: DEBUG MARKER: Write half of file
[ 6365.758656] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6369.802779] Lustre: DEBUG MARKER: Step1: done
[ 6371.870864] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 6374.200035] Lustre: DEBUG MARKER: Step2: done
[ 6432.484229] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 10:50:49 (1789138249)
[ 6476.501817] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 10:51:33 (1789138293)
[ 6517.212664] Lustre: DEBUG MARKER: Write...
[ 6519.650923] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6575.194936] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 10:53:11 (1789138391)
[ 6583.119634] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 10:53:20 (1789138400)
[ 6593.662544] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 10:53:30 (1789138410)
[ 6603.986382] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 10:53:40 (1789138420)
[ 6614.390951] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 10:53:51 (1789138431)
[ 6668.271690] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6802.442599] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 6942.142875] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 10:59:19 (1789138759)
[ 6996.243651] Lustre: DEBUG MARKER: Restart...
[ 7002.593030] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 7002.605690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7002.621433] Lustre: Skipped 11 previous similar messages
[ 7002.636638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7007.167280] Lustre: server umount lustre-MDT0000 complete
[ 7008.232134] LustreError: 158724:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7008.274422] LustreError: 158724:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 61 previous similar messages
[ 7011.220289] LustreError: 158710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138831 with bad export cookie 16269092797325050417
[ 7011.229689] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7011.238947] LustreError: 158710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages
[ 7011.638533] Lustre: server umount lustre-MDT0001 complete
[ 7026.193783] Lustre: server umount lustre-OST0000 complete
[ 7029.727508] Lustre: 3647:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789138833/real 1789138833]  req@ffff9f783fb3ad00 x1876040796651392/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789138849 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7030.097102] Lustre: server umount lustre-OST0001 complete
[ 7049.160236] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local
[ 7061.161376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7061.757245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7066.613230] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7076.080117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7076.669282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 7081.591841] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7085.381390] Lustre: 185966:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7092.881781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 7093.374156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7101.532623] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7101.545798] LustreError: 186319:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7101.571444] LustreError: 186319:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages
[ 7101.573894] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:161)
[ 7106.679461] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:813 to 0x280000401:833)
[ 7112.438256] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 7118.338494] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:777 to 0x2c0000401:801)
[ 7121.086248] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7129.944787] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7134.653801] Lustre: 187843:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7200.766365] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 11:03:37 (1789139017)
[ 7255.378096] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 11:04:32 (1789139072)
[ 9140.358458] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 11:35:57 (1789140957)
[ 9150.451239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 9150.458598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9150.479529] Lustre: Skipped 4 previous similar messages
[ 9150.510950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9150.523482] Lustre: Skipped 4 previous similar messages
[ 9151.459773] LustreError: 3647:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff9f7711b0f480 x1876040800201728/t0(0) o601->lustre-MDT0000-lwp-MDT0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 9151.507849] LustreError: 3647:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-MDT0000 qtype:usr lqe: ffff9f770b215680 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2544 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 9154.019375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9154.030760] Lustre: Skipped 3 previous similar messages
[ 9156.941185] Lustre: server umount lustre-MDT0000 complete
[ 9159.139705] LustreError: 184818:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9159.159549] LustreError: 184818:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages
[ 9160.879951] LustreError: 184802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140980 with bad export cookie 16269092797325059552
[ 9160.885717] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9160.894686] LustreError: 184802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 9161.420394] Lustre: server umount lustre-MDT0001 complete
[ 9175.965525] Lustre: server umount lustre-OST0000 complete
[ 9190.185780] Lustre: server umount lustre-OST0001 complete
[ 9207.256597] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local
[ 9217.205800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9217.683084] LustreError: 195404:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9217.799104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9217.807322] Lustre: Skipped 1 previous similar message
[ 9222.704807] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9232.635140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9233.195206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 9238.488248] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9242.353337] Lustre: 196550:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9252.115605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 9252.676237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9257.840713] LustreError: 196904:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9257.875302] LustreError: 196904:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages
[ 9257.911791] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:193)
[ 9261.539416] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9262.957310] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5835 to 0x280000401:5857)
[ 9272.865798] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 9273.282271] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9278.467448] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5803 to 0x2c0000401:5825)
[ 9280.563327] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9289.579208] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9294.870418] Lustre: 198430:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9330.307733] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 11:39:07 (1789141147)
[ 9362.065856] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 11:39:39 (1789141179)
[ 9392.755759] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 11:40:09 (1789141209)
[ 9434.299663] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 11:40:51 (1789141251)
[ 9474.843651] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 11:41:31 (1789141291)
[ 9567.307904] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 11:43:04 (1789141384)
[ 9604.728525] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 11:43:41 (1789141421)
[ 9685.433196] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 11:45:02 (1789141502)
[ 9693.701298] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9693.708407] Lustre: Skipped 1 previous similar message
[ 9695.732601] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9695.736128] Lustre: Skipped 65 previous similar messages
[ 9699.744535] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9699.747444] Lustre: Skipped 171 previous similar messages
[ 9707.768957] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9707.772419] Lustre: Skipped 361 previous similar messages
[ 9723.800109] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9723.802114] Lustre: Skipped 615 previous similar messages
[ 9755.865398] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9755.867478] Lustre: Skipped 1215 previous similar messages
[ 9819.889118] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9819.891828] Lustre: Skipped 2409 previous similar messages
[10328.035900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10328.040633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10328.044568] Lustre: Skipped 4 previous similar messages
[10333.158512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10333.164080] Lustre: Skipped 6 previous similar messages
[10333.724136] Lustre: server umount lustre-MDT0000 complete
[10337.272984] LustreError: 195386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789142157 with bad export cookie 16269092797326841311
[10337.291506] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10337.294865] LustreError: 195386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[10338.273067] LustreError: 195401:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10338.273768] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10338.277243] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[10338.301594] LustreError: 195401:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages
[10338.331682] Lustre: Skipped 3 previous similar messages
[10342.307066] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[10342.318973] Lustre: Skipped 2 previous similar messages
[10343.740552] Lustre: server umount lustre-MDT0001 complete
[10348.791975] Lustre: server umount lustre-OST0000 complete
[10353.376088] Lustre: server umount lustre-OST0001 complete
[10361.071540] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid
[10370.410924] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local
[10422.121193] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local
[10432.308576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[10432.509792] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[10432.534181] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[10432.606105] Lustre: lustre-MDT0000: new disk, initializing
[10432.674687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10432.701411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[10437.422125] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10450.139691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[10450.346156] Lustre: 215070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17
[10450.385621] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
[10450.389816] Lustre: Skipped 1 previous similar message
[10450.494925] Lustre: lustre-MDT0001: new disk, initializing
[10450.605304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[10450.669046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt
[10450.696759] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt]
[10456.561314] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10462.250699] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[10470.096668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[10470.290995] Lustre: lustre-OST0000: new disk, initializing
[10470.296176] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[10470.299886] Lustre: 216701:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0]
[10470.349578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10471.637574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost
[10471.664253] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost]
[10471.775227] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401
[10477.430363] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10488.836866] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[10488.951666] Lustre: lustre-OST0001: new disk, initializing
[10488.958293] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[10488.963985] Lustre: 217572:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0]
[10489.043415] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10490.597140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost
[10490.609617] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost]
[10490.684779] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401
[10495.775979] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10506.330636] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10510.304415] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[10538.920289] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 11:59:15 (1789142355)
[10545.732768] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 11:59:23 (1789142363)
[10579.274249] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 11:59:56 (1789142396)
[10628.483698] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 12:00:45 (1789142445)
[10652.600222] Lustre: DEBUG MARKER: rename directory return 255
[10692.317346] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 12:01:49 (1789142509)
[10714.641472] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 12:02:11 (1789142531)
[10747.831776] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 12:02:45 (1789142565)
[10874.950924] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 12:04:52 (1789142692)
[10903.097756] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 12:05:20 (1789142720)
[10938.457756] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 12:05:55 (1789142755)
[11080.343423] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 12:08:17 (1789142897)
[11084.780454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[11084.783534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[11084.799098] Lustre: Skipped 1 previous similar message
[11084.817727] Lustre: Skipped 3 previous similar messages
[11089.894820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[11089.902768] Lustre: Skipped 3 previous similar messages
[11090.932978] Lustre: server umount lustre-MDT0000 complete
[11094.802439] LustreError: 230010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789142914 with bad export cookie 16269092797327213487
[11094.811533] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[11094.812957] LustreError: 230010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[11095.009369] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[11095.009864] LustreError: 216321:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[11095.027839] Lustre: Skipped 3 previous similar messages
[11095.042165] LustreError: 216321:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages
[11095.187239] Lustre: server umount lustre-MDT0001 complete
[11105.184182] Lustre: server umount lustre-OST0000 complete
[11109.186406] Lustre: server umount lustre-OST0001 complete
[11127.393837] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local
[11137.400750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[11137.778574] LustreError: 234819:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[11137.851840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[11137.858996] LustreError: lustre-MDT0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[11142.612698] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11143.143745] LustreError: 234820:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[11151.610693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[11151.977355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[11152.001293] LustreError: lustre-MDT0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[11157.079300] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11160.364810] Lustre: 235962:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[11168.237129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[11168.660166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[11168.674432] LustreError: lustre-OST0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[11174.461272] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11176.866273] LustreError: 236316:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[11176.892205] LustreError: 236316:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message
[11183.097375] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:65)
[11184.072595] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[11184.272343] LustreError: lustre-OST0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[11189.744537] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:20 to 0x2c0000401:65)
[11190.445466] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11198.291393] Lustre: DEBUG MARKER: Using TIMEOUT=20
[11202.309803] Lustre: 237838:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[11209.214900] LustreError: 234816:0:(osd_handler.c:3425:osd_quota_transfer()) lustre-MDT0000: quota transfer failed. Is project enforcement enabled on the ldiskfs filesystem? rc = -95
[11213.283680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[11213.296699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[11213.323813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[11215.330669] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[11215.331665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[11215.350158] Lustre: Skipped 1 previous similar message
[11215.387749] Lustre: Skipped 3 previous similar messages
[11219.592347] Lustre: server umount lustre-MDT0000 complete
[11220.449528] LustreError: 234815:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[11220.476225] LustreError: 234815:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages
[11223.012215] LustreError: 234802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789143042 with bad export cookie 16269092797327262382
[11223.015773] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[11223.025065] LustreError: 234802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message
[11223.401638] Lustre: server umount lustre-MDT0001 complete
[11232.223833] LustreError: 3648:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff9f7717bc9680 x1876040807405696/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[11232.244161] LustreError: 3648:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0000 qtype:grp lqe: ffff9f7707949680 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 1520 qunit:0 qtune:0 edquot:0 default:no revoke:0
[11232.259079] LustreError: 3648:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 1 previous similar message
[11237.672436] Lustre: server umount lustre-OST0000 complete
[11251.783888] Lustre: server umount lustre-OST0001 complete
[11272.499581] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local
[11281.428837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[11281.755464] LustreError: 240230:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[11281.840578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[11281.853385] Lustre: Skipped 1 previous similar message
[11285.594531] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11293.773285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[11298.160343] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11301.376806] Lustre: 241380:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[11308.450409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[11315.708910] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11324.385695] LustreError: 241736:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[11324.413673] LustreError: 241736:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages
[11325.980359] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[11326.184784] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[11326.190785] Lustre: Skipped 2 previous similar messages
[11327.217796] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97)
[11327.237318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:97)
[11332.529459] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11341.001977] Lustre: DEBUG MARKER: Using TIMEOUT=20
[11344.640782] Lustre: 243255:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[11374.538449] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 12:13:11 (1789143191)
[11419.465898] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[11421.076447] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 12:13:58 (1789143238)
[11440.127474] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[11441.652814] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 12:14:19 (1789143259)
[11470.019690] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[11471.408413] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 12:14:48 (1789143288)
[11504.740523] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 12:15:21 (1789143321)
[11517.086862] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[11524.748528] Lustre: 249717:0:(qsd_reint.c:249:qsd_reint_index()) lustre-OST0000: index version for fid [0x200000005:0x1007:0x0] is 0, but index isn't empty (1)
[11530.929442] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[11538.547987] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[11542.600191] Lustre: DEBUG MARKER: Write...
[11565.739650] LustreError: 251062:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[11571.012362] Lustre: DEBUG MARKER: Write...
[11583.095284] Lustre: DEBUG MARKER: Write...
[11649.044727] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 12:17:46 (1789143466)
[11673.802809] LustreError: 255125:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[11720.447957] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 12:18:57 (1789143537)
[11748.529677] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[11750.321502] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[11829.672393] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 12:20:46 (1789143646)
[11873.416758] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 12:21:30 (1789143690)
[11916.835908] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 12:22:12 (1789143732)
[11931.998539] Lustre: DEBUG MARKER: User quota (block hardlimit:100 MB)
[11955.138519] Lustre: DEBUG MARKER: Write...
[11957.499171] Lustre: DEBUG MARKER: Write out of block quota ...
[12045.410714] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 12:24:22 (1789143862)
[12056.503590] Lustre: DEBUG MARKER: User quota (block hardlimit:1000 MB)
[12137.307635] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 12:25:54 (1789143954)
[12150.474642] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[12172.016389] Lustre: DEBUG MARKER: Write...
[12174.116540] Lustre: DEBUG MARKER: Write out of block quota ...
[12174.559843] LustreError: 240229:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:10240 soft:0 granted:10240 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12224.379294] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 12:27:21 (1789144041)
[12251.376648] Lustre: DEBUG MARKER: set to use default quota
[12253.392955] Lustre: DEBUG MARKER: set default quota
[12255.204754] Lustre: DEBUG MARKER: get default quota
[12261.675450] Lustre: DEBUG MARKER: Test not out of quota
[12265.514982] Lustre: DEBUG MARKER: Test out of quota
[12276.289807] Lustre: DEBUG MARKER: Increase default quota
[12299.713763] Lustre: DEBUG MARKER: Set quota to override default quota
[12299.770896] LustreError: 257995:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45056 time:1789748919 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12311.701846] Lustre: DEBUG MARKER: Set to use default quota again
[12334.214615] Lustre: DEBUG MARKER: Cleanup
[12397.040899] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 12:30:14 (1789144214)
[12420.985544] Lustre: DEBUG MARKER: set default quota for qpool1
[12422.973486] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[12467.435479] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 12:31:23 (1789144283)
[12561.500891] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 12:32:58 (1789144378)
[12628.777756] Lustre: DEBUG MARKER: Write...
[12631.179984] Lustre: DEBUG MARKER: Write out of block quota ...
[12735.889247] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 12:35:52 (1789144552)
[12775.484841] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 12:36:32 (1789144592)
[12783.951805] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 12:36:40 (1789144600)
[12829.680982] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 12:37:26 (1789144646)
[12880.043394] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 12:38:16 (1789144696)
[12906.121394] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 12:38:43 (1789144723)
[12938.311753] Lustre: *** cfs_fail_loc=a06, val=0***
[12938.315532] Lustre: Skipped 3157 previous similar messages
[12938.821952] LustreError: 240244:0:(qmt_lock.c:479:qmt_lvbo_update()) $$$ failed to release quota space on glimpse 0!=1024 : rc = -11
                 qmt:lustre-QMT0000 pool:dt-0x0 id:60000 enforced:1 hard:102400 soft:0 granted:24576 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no
[12944.242331] Lustre: Failing over lustre-OST0001
[12944.490461] Lustre: server umount lustre-OST0001 complete
[12945.314051] LustreError: 241736:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[12945.376943] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107
[12945.387507] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[12945.404048] Lustre: Skipped 2 previous similar messages
[12955.189416] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[12955.630357] Lustre: lustre-OST0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[12955.664324] Lustre: lustre-OST0001: in recovery but waiting for the first client to connect
[12957.112594] Lustre: lustre-OST0001: Will be in recovery for at least 1:00, or until 3 clients reconnect
[12957.599067] Lustre: lustre-OST0001: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted.
[12957.600249] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo)
[12957.631387] Lustre: Skipped 3 previous similar messages
[12963.960556] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[13008.438215] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 12:40:25 (1789144825)
[13021.589211] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[13029.491753] LustreError: 293903:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[13029.500889] LustreError: 293903:0:(qmt_pool.c:1407:qmt_pool_recalc()) Skipped 3 previous similar messages
[13034.975862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[13034.981190] LustreError: Skipped 1 previous similar message
[13034.983844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[13035.005961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[13037.460834] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.56@tcp (stopping)
[13037.537238] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[13037.548899] Lustre: Skipped 2 previous similar messages
[13039.503142] LustreError: 293903:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[13039.775582] Lustre: server umount lustre-MDT0000 complete
[13042.565193] LustreError: 241270:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[13042.582708] LustreError: 241270:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages
[13048.412606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[13048.563588] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[13048.898499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[13048.960689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:112 to 0x280000401:129)
[13048.966924] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:113 to 0x2c0000401:129)
[13053.945727] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo)
[13053.953926] Lustre: Skipped 1 previous similar message
[13054.437383] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[13087.308320] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 12:41:44 (1789144904)
[13095.874640] LustreError: 240982:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-MDT0001: Delaying reintegration for qtype:2 until pending updates are flushed.
[13117.287666] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 12:42:14 (1789144934)
[13144.842973] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 12:42:41 (1789144961)
[13194.725866] Lustre: *** cfs_fail_loc=a08, val=0***
[13194.734780] Lustre: Skipped 9 previous similar messages
[13194.752026] Lustre: *** cfs_fail_loc=a08, val=0***
[13194.765679] Lustre: Skipped 1 previous similar message
[13298.715892] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 12:45:15 (1789145115)
[13329.452294] LustreError: 240228:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:3072 soft:0 granted:3072 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13382.087959] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 12:46:38 (1789145198)
[13450.783815] LustreError: 240225:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:16384 time:1789750070 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13606.787763] LustreError: 240225:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:12308 time:1789750226 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13771.884290] LustreError: 240226:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:1000 enforced:1 hard:2048 soft:2048 granted:16384 time:1789750391 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13781.292884] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed