[ 5784.794331] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5791.887684] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5795.363538] LustreError: 121530:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5796.505475] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5798.288799] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5800.294209] Lustre: Failing over lustre-MDT0000
[ 5800.487290] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 5800.495702] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5800.505565] Lustre: Skipped 1 previous similar message
[ 5800.513679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5800.655592] Lustre: server umount lustre-MDT0000 complete
[ 5817.761542] Lustre: 3299:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786980770/real 1786980770]  req@ffff8eb5c6919500 x1873779139752704/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786980786 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5817.830092] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5822.240233] Lustre: 3299:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786980775/real 1786980775]  req@ffff8eb5c7bd6a00 x1873779139753856/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786980791 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5822.282318] Lustre: 3299:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5828.001915] LustreError: 3296:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff8eb5fc086300 x1873779139755776/t0(0) o250->MGC192.168.202.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5828.002632] Lustre: 3300:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786980780/real 1786980780]  req@ffff8eb5c6919c00 x1873779139754368/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786980796 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5828.664269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5828.760100] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5830.499669] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5830.693617] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5830.731230] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:627 to 0x280000400:673)
[ 5830.732061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:631 to 0x240000400:673)
[ 5833.239743] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5836.789638] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 5836.803347] Lustre: Skipped 1 previous similar message
[ 5843.504600] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5844.975227] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5869.739728] Lustre: DEBUG MARKER: (dd_pid=115056, time=16, timeout=600)
[ 5921.981203] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5928.127284] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5931.950753] LustreError: 124510:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5933.075997] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5934.676865] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5936.659512] Lustre: Failing over lustre-MDT0000
[ 5937.052355] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 5937.057538] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5937.064615] Lustre: Skipped 1 previous similar message
[ 5937.116645] Lustre: server umount lustre-MDT0000 complete
[ 5955.576610] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5955.994165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5956.117434] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5957.860806] Lustre: 3300:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786980910/real 1786980910]  req@ffff8eb5ec29fb80 x1873779139797120/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786980926 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5957.885636] Lustre: 3300:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5960.532728] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5961.205433] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5961.212796] Lustre: Skipped 1 previous similar message
[ 5964.138490] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5964.302668] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5964.338672] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:627 to 0x280000400:705)
[ 5964.344261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:675 to 0x240000400:705)
[ 5970.068796] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5971.881370] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5982.696125] Lustre: DEBUG MARKER: (dd_pid=117568, time=5, timeout=600)
[ 6043.608565] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 11:36:51 (1786981011)
[ 6067.968135] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6069.925416] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6076.062976] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6077.741887] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6113.148768] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 11:38:01 (1786981081)
[ 6138.458770] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 6155.584551] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 6157.425468] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 6159.239929] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 6161.698034] Lustre: DEBUG MARKER: Set quota for 1 times
[ 6166.280898] Lustre: DEBUG MARKER: Set quota for 2 times
[ 6171.205293] Lustre: DEBUG MARKER: Set quota for 3 times
[ 6175.213979] Lustre: DEBUG MARKER: Set quota for 4 times
[ 6178.683177] Lustre: DEBUG MARKER: Set quota for 5 times
[ 6183.632686] Lustre: DEBUG MARKER: Set quota for 6 times
[ 6187.656614] Lustre: DEBUG MARKER: Set quota for 7 times
[ 6226.627544] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 11:39:54 (1786981194)
[ 6238.690796] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6238.691460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6238.730954] Lustre: Skipped 2 previous similar messages
[ 6243.816083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6243.826325] Lustre: Skipped 2 previous similar messages
[ 6244.195038] Lustre: server umount lustre-MDT0000 complete
[ 6248.756164] LustreError: 5812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786981217 with bad export cookie 4788240267056159006
[ 6248.759274] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6248.765031] LustreError: 5812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 6248.868236] Lustre: server umount lustre-OST0000 complete
[ 6253.170425] Lustre: server umount lustre-OST0001 complete
[ 6268.072985] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local
[ 6279.101340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6283.615538] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6286.366233] Lustre: 134245:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6293.651928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6300.542866] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6303.778300] LustreError: 134613:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6303.808794] LustreError: 134613:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message
[ 6303.838486] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:707 to 0x240000400:737)
[ 6308.838478] LustreError: 134613:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6311.768328] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6317.082137] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:708 to 0x280000400:737)
[ 6320.297323] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6328.877457] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6332.744971] Lustre: 136160:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6352.883698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6355.361489] Lustre: server umount lustre-MDT0000 complete
[ 6360.764257] LustreError: 136409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786981329 with bad export cookie 4788240267056165481
[ 6360.769476] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6360.774423] LustreError: 136409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 6360.879515] Lustre: server umount lustre-OST0000 complete
[ 6365.603136] Lustre: server umount lustre-OST0001 complete
[ 6379.774412] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local
[ 6389.702347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6394.250421] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6397.258506] Lustre: 138827:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6409.996917] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6414.690287] LustreError: 139199:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6414.760438] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:707 to 0x240000400:769)
[ 6419.949429] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6419.967406] Lustre: Skipped 1 previous similar message
[ 6425.105370] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:708 to 0x280000400:769)
[ 6429.568888] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6438.611318] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6442.469678] Lustre: 140749:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6454.040949] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 11:43:41 (1786981421)
[ 6455.804467] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6457.626770] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 11:43:45 (1786981425)
[ 6519.503373] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 11:44:46 (1786981486)
[ 6572.027602] Lustre: DEBUG MARKER: Write...
[ 6574.976300] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6639.229400] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 11:46:46 (1786981606)
[ 6647.040770] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 11:46:54 (1786981614)
[ 6658.081668] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 11:47:05 (1786981625)
[ 6668.317290] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 11:47:15 (1786981635)
[ 6677.440591] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 11:47:24 (1786981644)
[ 6750.915687] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6921.797146] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 7124.898466] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 11:54:52 (1786982092)
[ 7190.390471] Lustre: DEBUG MARKER: Restart...
[ 7194.159290] Lustre: server umount lustre-MDT0000 complete
[ 7198.817399] LustreError: 138381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786982167 with bad export cookie 4788240267056167014
[ 7198.835529] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7208.575762] Lustre: server umount lustre-OST0000 complete
[ 7214.561396] Lustre: 3299:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786982167/real 1786982167]  req@ffff8eb50b7cca80 x1873779140186496/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786982183 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7214.626898] Lustre: 3299:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 7214.647274] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7214.689691] Lustre: Skipped 2 previous similar messages
[ 7217.378635] Lustre: server umount lustre-OST0001 complete
[ 7240.699977] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local
[ 7250.941668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7256.503299] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7260.410869] Lustre: 159373:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7266.973035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7273.671501] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7275.109728] LustreError: 159740:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7275.155244] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:779 to 0x240000400:801)
[ 7280.610029] LustreError: 160182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7283.319759] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7288.369202] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:777 to 0x280000400:801)
[ 7289.357124] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7297.212132] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7301.353254] Lustre: 161283:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7365.315190] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 11:58:52 (1786982332)
[ 7438.809511] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 12:00:06 (1786982406)
[ 9279.268759] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 12:30:45 (1786984245)
[ 9295.329216] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9295.341390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9295.343478] Lustre: Skipped 1 previous similar message
[ 9299.616399] Lustre: server umount lustre-MDT0000 complete
[ 9304.283048] LustreError: 158924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786984273 with bad export cookie 4788240267056173510
[ 9304.290965] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9304.297441] LustreError: 158924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 9304.428806] Lustre: server umount lustre-OST0000 complete
[ 9309.029527] Lustre: server umount lustre-OST0001 complete
[ 9326.674643] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local
[ 9338.992585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9344.481387] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9347.578511] Lustre: 169747:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9354.453394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9360.922466] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9363.627077] LustreError: 170098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9363.660260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5804 to 0x240000400:5825)
[ 9369.069472] LustreError: 170587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9371.664847] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9376.815953] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5802 to 0x280000400:5825)
[ 9380.526713] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9390.288221] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9395.131797] Lustre: 171656:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9432.752281] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 12:33:19 (1786984399)
[ 9486.244535] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 12:34:13 (1786984453)
[ 9535.289144] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 12:35:02 (1786984502)
[ 9536.813485] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9538.961178] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 12:35:06 (1786984506)
[ 9540.631811] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9542.399851] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 12:35:10 (1786984510)
[ 9645.249336] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 12:36:52 (1786984612)
[ 9700.648498] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 12:37:47 (1786984667)
[ 9802.743062] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 12:39:29 (1786984769)
[ 9815.622609] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9815.624383] Lustre: Skipped 4 previous similar messages
[ 9817.624770] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9817.627384] Lustre: Skipped 97 previous similar messages
[ 9821.632496] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9821.634178] Lustre: Skipped 181 previous similar messages
[ 9829.637568] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9829.641324] Lustre: Skipped 365 previous similar messages
[ 9845.672546] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9845.676569] Lustre: Skipped 669 previous similar messages
[ 9877.704818] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9877.708403] Lustre: Skipped 1341 previous similar messages
[ 9941.721586] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9941.725529] Lustre: Skipped 3025 previous similar messages
[10342.563189] Lustre: *** cfs_fail_loc=a08, val=0***
[10342.565121] Lustre: Skipped 2315 previous similar messages
[10342.593378] Lustre: *** cfs_fail_loc=a08, val=0***
[10342.601254] Lustre: Skipped 4 previous similar messages
[10618.048859] Lustre: *** cfs_fail_loc=a08, val=0***
[10618.054158] Lustre: Skipped 15 previous similar messages
[10618.078234] Lustre: *** cfs_fail_loc=a08, val=0***
[10618.088317] Lustre: Skipped 12 previous similar messages
[10622.132658] Lustre: *** cfs_fail_loc=a08, val=0***
[10622.144784] Lustre: Skipped 65 previous similar messages
[10641.376809] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10641.378181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10641.383592] Lustre: Skipped 2 previous similar messages
[10641.390845] Lustre: Skipped 2 previous similar messages
[10643.715763] Lustre: server umount lustre-MDT0000 complete
[10648.741541] LustreError: 169303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786985617 with bad export cookie 4788240267057952441
[10648.764725] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10649.112407] Lustre: server umount lustre-OST0000 complete
[10654.574483] Lustre: server umount lustre-OST0001 complete
[10663.403666] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid
[10671.835875] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local
[10683.430623]  vdc: vdc1 vdc9
[10694.435605]  vde: vde1 vde9
[10694.460237]  vde: vde1 vde9
[10694.485701]  vde: vde1 vde9
[10705.596758]  vdf: vdf1 vdf9
[10725.142167] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local
[10736.368766] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[10736.727144] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[10736.840617] Lustre: lustre-MDT0000: new disk, initializing
[10737.255388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10737.320487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[10742.746500] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10748.693879] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[10756.392402] Lustre: lustre-OST0000: new disk, initializing
[10756.399332] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[10756.408255] Lustre: Skipped 1 previous similar message
[10756.493714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10757.841975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[10757.859856] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[10758.083362] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[10764.055418] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10778.282668] Lustre: lustre-OST0001: new disk, initializing
[10778.301342] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[10778.483808] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10779.952170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[10779.962705] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[10780.138820] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[10786.215658] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10796.377158] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10801.060807] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[10846.912280] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 12:56:53 (1786985813)
[10856.510453] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 12:57:03 (1786985823)
[10914.594377] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 12:58:01 (1786985881)
[10976.880803] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 12:59:04 (1786985944)
[11008.573809] Lustre: DEBUG MARKER: rename directory return 255
[11055.836000] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 13:00:23 (1786986023)
[11089.064949] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 13:00:56 (1786986056)
[11138.863443] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 13:01:46 (1786986106)
[11261.376557] LustreError: 196497:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11300.282405] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 13:04:27 (1786986267)
[11345.550074] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 13:05:12 (1786986312)
[11391.490854] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 13:05:58 (1786986358)
[11590.678465] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 13:09:18 (1786986558)
[11592.334203] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[11594.032416] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 13:09:21 (1786986561)
[11657.117304] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[11659.050208] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 13:10:26 (1786986626)
[11693.063629] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[11694.974644] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 13:11:02 (1786986662)
[11756.042443] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[11758.687405] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 13:12:05 (1786986725)
[11813.859817] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 13:13:00 (1786986780)
[11816.250820] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[11818.371683] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 13:13:05 (1786986785)
[11850.084225] LustreError: 209251:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[11903.230341] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 13:14:30 (1786986870)
[11937.282180] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[11939.318967] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[12145.736127] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 13:18:32 (1786987112)
[12205.343253] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 13:19:32 (1786987172)
[12241.375477] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 13:20:09 (1786987209)
[12243.147626] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[12245.440795] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 13:20:12 (1786987212)
[12247.570764] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[12249.560382] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 13:20:17 (1786987217)
[12269.403848] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[12291.724706] Lustre: DEBUG MARKER: Write...
[12296.083904] Lustre: DEBUG MARKER: Write out of block quota ...
[12356.583952] LustreError: 3298:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-OST0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[12360.722895] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 13:22:07 (1786987327)
[12391.992770] Lustre: DEBUG MARKER: set to use default quota
[12394.396663] Lustre: DEBUG MARKER: set default quota
[12396.166128] Lustre: DEBUG MARKER: get default quota
[12403.696632] Lustre: DEBUG MARKER: Test not out of quota
[12408.637776] Lustre: DEBUG MARKER: Test out of quota
[12421.795479] Lustre: DEBUG MARKER: Increase default quota
[12455.923390] Lustre: DEBUG MARKER: Set quota to override default quota
[12456.009766] LustreError: 186402:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1787592225 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12469.125552] Lustre: DEBUG MARKER: Set to use default quota again
[12491.446718] Lustre: DEBUG MARKER: Cleanup
[12568.461457] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 13:25:36 (1786987536)
[12595.179318] Lustre: DEBUG MARKER: set default quota for qpool1
[12596.972744] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[12653.567526] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 13:27:00 (1786987620)
[12746.527279] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 13:28:33 (1786987713)
[12831.771147] Lustre: DEBUG MARKER: Write...
[12838.956633] Lustre: DEBUG MARKER: Write out of block quota ...
[12984.088936] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 13:32:31 (1786987951)
[13035.616676] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 13:33:22 (1786988002)
[13045.619581] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 13:33:33 (1786988013)
[13047.760974] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate
[13050.450362] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 13:33:37 (1786988017)
[13052.225236] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate
[13054.554454] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 13:33:41 (1786988021)
[13080.285584] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 13:34:07 (1786988047)
[13081.921758] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[13084.365043] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 13:34:11 (1786988051)
[13106.054320] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[13113.500906] LustreError: 234315:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[13118.345088] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.52@tcp (stopping)
[13119.969303] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[13119.969755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[13119.984253] Lustre: Skipped 1 previous similar message
[13119.996962] Lustre: Skipped 1 previous similar message
[13123.424915] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.52@tcp (stopping)
[13123.555296] LustreError: 234315:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[13124.102637] Lustre: server umount lustre-MDT0000 complete
[13136.697976] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[13137.506687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[13137.604453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[13137.604820] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[13140.523836] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[13140.529913] Lustre: Skipped 1 previous similar message
[13144.524609] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[13188.276153] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 13:35:55 (1786988155)
[13224.791702] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 13:36:32 (1786988192)
[13260.742960] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 13:37:08 (1786988228)
[13313.111596] Lustre: *** cfs_fail_loc=a08, val=0***
[13313.113184] Lustre: Skipped 234 previous similar messages
[13313.137098] Lustre: *** cfs_fail_loc=a08, val=0***
[13313.139832] Lustre: Skipped 166 previous similar messages
[13450.998717] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 13:40:18 (1786988418)
[13546.060506] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 13:41:53 (1786988513)
[13640.964162] LustreError: 235784:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1787593409 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13806.930549] LustreError: 234900:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1787593575 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13816.532643] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed