[ 3177.801692] Lustre: DEBUG MARKER: User quota (limit: 200) [ 3179.174328] Lustre: DEBUG MARKER: Write 100M (buffered) ... [ 3180.238660] LustreError: 13939:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 3180.542398] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3181.007254] Lustre: DEBUG MARKER: Fail mds for 40 seconds [ 3181.657743] Lustre: Failing over lustre-MDT0000 [ 3181.778973] LustreError: 14313:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 3181.782527] LustreError: 14313:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3181.851376] Lustre: server umount lustre-MDT0000 complete [ 3194.306712] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3194.497742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3194.533688] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3195.293615] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3195.352830] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3195.372495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:864 to 0x240000400:897) [ 3195.372546] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:828 to 0x280000400:865) [ 3195.536809] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3197.933483] Lustre: 3305:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746159685/real 1746159685] req@ffff880092591880 x1830977850471424/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1746159701 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 [ 3197.949853] Lustre: 3305:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3197.956423] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3197.964337] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.204.104@tcp (at 0@lo) [ 3198.811891] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3199.433571] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3199.962614] Lustre: 3303:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746159687/real 1746159687] req@ffff88008fa19c00 x1830977850471808/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746159703 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 3199.983567] LustreError: 20076:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-OST0000: Delaying reintegration for qtype:0 until pending updates are flushed. [ 3199.992227] LustreError: 20076:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 6 previous similar messages [ 3204.966584] Lustre: 3304:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746159692/real 1746159692] req@ffff88008fa18380 x1830977850472192/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746159708 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 3204.979979] Lustre: 3304:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3206.359174] Lustre: DEBUG MARKER: (dd_pid=2430, time=4, timeout=600) [ 3210.045915] Lustre: 3306:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746159697/real 1746159697] req@ffff88013168d500 x1830977850472576/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746159713 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 3210.069665] Lustre: 3306:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3235.668687] Lustre: DEBUG MARKER: User quota (limit: 200) [ 3237.470405] Lustre: DEBUG MARKER: Write 100M (directio) ... [ 3238.794993] LustreError: 18380:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 3239.261299] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3239.990237] Lustre: DEBUG MARKER: Fail mds for 40 seconds [ 3241.075838] Lustre: Failing over lustre-MDT0000 [ 3241.239061] LustreError: 19176:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 3241.245567] LustreError: 19176:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3241.373225] Lustre: server umount lustre-MDT0000 complete [ 3254.059154] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3254.236485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3254.261451] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3255.204132] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3255.389520] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3255.444159] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3255.465189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:899 to 0x240000400:929) [ 3255.467038] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:828 to 0x280000400:897) [ 3257.169867] Lustre: 3303:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746159744/real 1746159744] req@ffff8800a66cd500 x1830977850501504/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1746159760 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 [ 3257.183381] Lustre: 3303:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3258.296592] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3258.841997] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3265.215623] Lustre: 3305:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746159751/real 1746159751] req@ffff88009bacbb80 x1830977850502144/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746159767 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 3265.225598] Lustre: 3305:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 3265.520918] Lustre: DEBUG MARKER: (dd_pid=4786, time=4, timeout=600) [ 3305.035684] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 00:23:28 (1746159808) [ 3313.727626] LustreError: 24460:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:1 until pending updates are flushed. [ 3313.733842] LustreError: 24460:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 1 previous similar message [ 3317.013971] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1: [ 3317.630918] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2). [ 3319.325970] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1: [ 3319.869822] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2). [ 3342.896743] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 00:24:06 (1746159846) [ 3351.742211] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing [ 3360.785103] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr [ 3361.546308] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr [ 3362.291220] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000 [ 3363.140896] Lustre: DEBUG MARKER: Set quota for 1 times [ 3365.021677] Lustre: DEBUG MARKER: Set quota for 2 times [ 3366.867139] Lustre: DEBUG MARKER: Set quota for 3 times [ 3368.726130] Lustre: DEBUG MARKER: Set quota for 4 times [ 3370.656087] Lustre: DEBUG MARKER: Set quota for 5 times [ 3372.728374] Lustre: DEBUG MARKER: Set quota for 6 times [ 3374.739251] Lustre: DEBUG MARKER: Set quota for 7 times [ 3376.613977] Lustre: DEBUG MARKER: Set quota for 8 times [ 3378.510402] Lustre: DEBUG MARKER: Set quota for 9 times [ 3380.388516] Lustre: DEBUG MARKER: Set quota for 10 times [ 3382.319684] Lustre: DEBUG MARKER: Set quota for 11 times [ 3384.244594] Lustre: DEBUG MARKER: Set quota for 12 times [ 3386.235565] Lustre: DEBUG MARKER: Set quota for 13 times [ 3388.235194] Lustre: DEBUG MARKER: Set quota for 14 times [ 3390.096217] Lustre: DEBUG MARKER: Set quota for 15 times [ 3392.239986] Lustre: DEBUG MARKER: Set quota for 16 times [ 3419.025394] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 00:25:22 (1746159922) [ 3426.042454] LustreError: 1058:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 3426.044777] LustreError: 1058:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3426.135798] Lustre: server umount lustre-MDT0000 complete [ 3428.057890] LustreError: 17918:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746159931 with bad export cookie 919254851129693294 [ 3428.063035] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3438.097017] Lustre: server umount lustre-OST0000 complete [ 3450.199153] Lustre: server umount lustre-OST0001 complete [ 3456.168056] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 3460.609859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3462.419015] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3464.056683] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 3466.540188] Lustre: 4091:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.204.104@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3466.722216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3469.318315] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3470.699710] LustreError: 4095:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3470.719716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:932 to 0x240000400:961) [ 3473.411238] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3474.899181] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:899 to 0x280000400:929) [ 3476.555267] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3480.737513] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3482.421914] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 3498.383379] LustreError: 7220:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3498.385405] LustreError: 7220:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3498.471502] Lustre: server umount lustre-MDT0000 complete [ 3500.696223] LustreError: 3017:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746160003 with bad export cookie 919254851129707294 [ 3500.700138] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3510.754806] Lustre: server umount lustre-OST0000 complete [ 3522.959448] Lustre: server umount lustre-OST0001 complete [ 3529.722006] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 3534.303576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3536.214289] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3537.947035] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 3541.053519] Lustre: 10197:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.204.104@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3541.059325] Lustre: 10197:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 1 previous similar message [ 3541.348541] mount.lustre (10197) used greatest stack depth: 9488 bytes left [ 3544.248291] LustreError: 10201:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3544.251151] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3544.288506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:932 to 0x240000400:993) [ 3550.281992] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:899 to 0x280000400:961) [ 3551.830384] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3556.576528] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3558.625531] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 3568.684339] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 00:27:51 (1746160071) [ 3569.376226] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS [ 3570.160506] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 00:27:53 (1746160073) [ 3603.167822] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 00:28:26 (1746160106) [ 3631.222017] Lustre: DEBUG MARKER: Write... [ 3632.590403] Lustre: DEBUG MARKER: Write out of block quota ... [ 3670.600248] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 00:29:33 (1746160173) [ 3673.547922] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 00:29:36 (1746160176) [ 3678.958594] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 00:29:41 (1746160181) [ 3683.506426] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 00:29:46 (1746160186) [ 3687.293004] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 00:29:50 (1746160190) [ 3729.172126] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user [ 3806.080984] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user [ 3913.926766] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 00:33:37 (1746160417) [ 3946.993414] Lustre: DEBUG MARKER: Restart... [ 3949.318932] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3949.319169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3949.326418] Lustre: Skipped 4 previous similar messages [ 3954.034858] LustreError: 3201:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3954.036584] LustreError: 3201:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3954.149729] Lustre: server umount lustre-MDT0000 complete [ 3955.908913] LustreError: 9164:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746160459 with bad export cookie 919254851129708827 [ 3955.911379] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3955.922463] LustreError: 9164:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3955.948804] Lustre: server umount lustre-OST0000 complete [ 3957.770618] Lustre: server umount lustre-OST0001 complete [ 3962.674534] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 3966.135811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3966.138399] Lustre: Skipped 2 previous similar messages [ 3967.554610] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3968.776539] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 3970.384349] Lustre: 6200:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.204.104@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3970.389192] Lustre: 6200:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 1 previous similar message [ 3972.148128] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3975.780546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1003 to 0x240000400:1025) [ 3975.794097] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:969 to 0x280000400:993) [ 3976.574772] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3979.488095] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3980.721059] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 4015.250711] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 00:35:18 (1746160518) [ 4064.124969] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 00:36:07 (1746160567) [ 4741.215552] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 00:47:24 (1746161244) [ 4751.426905] LustreError: 26675:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4751.429726] LustreError: 26675:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 4751.496142] Lustre: server umount lustre-MDT0000 complete [ 4753.281854] LustreError: 8327:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746161256 with bad export cookie 919254851129715204 [ 4753.289080] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4763.337713] Lustre: server umount lustre-OST0000 complete [ 4774.932021] Lustre: server umount lustre-OST0001 complete [ 4779.678580] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 4783.277055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4783.279595] Lustre: Skipped 2 previous similar messages [ 4784.893335] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4786.277480] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 4789.040064] Lustre: 29651:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.204.104@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4789.043673] Lustre: 29651:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 1 previous similar message [ 4789.226850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4791.742744] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4793.200631] LustreError: 30328:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4793.215644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6028 to 0x240000400:6049) [ 4794.942828] Lustre: 30726:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.204.104@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4797.098224] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5994 to 0x280000400:6017) [ 4797.710691] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4801.873322] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4803.553266] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 4829.482792] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 00:48:52 (1746161332) [ 4856.942396] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 00:49:19 (1746161359) [ 4884.375425] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 00:49:47 (1746161387) [ 4885.117755] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs [ 4885.923970] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 00:49:48 (1746161388) [ 4886.760110] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs [ 4887.786335] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 00:49:50 (1746161390) [ 4897.446069] LustreError: 3153:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed. [ 4897.452180] LustreError: 3153:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 3 previous similar messages [ 4929.654262] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 00:50:32 (1746161432) [ 4958.530592] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 00:51:01 (1746161461) [ 5029.055492] Lustre: DEBUG MARKER: == sanity-quota test 49: lfs quota -a prints the quota usage for all quota IDs ========================================================== 00:52:12 (1746161532) [ 5036.663514] Lustre: *** cfs_fail_loc=a09, val=0*** [ 5036.665267] Lustre: Skipped 1 previous similar message [ 5037.676703] Lustre: *** cfs_fail_loc=a09, val=0*** [ 5037.678769] Lustre: Skipped 85 previous similar messages [ 5039.684883] Lustre: *** cfs_fail_loc=a09, val=0*** [ 5039.688425] Lustre: Skipped 217 previous similar messages [ 5043.695986] Lustre: *** cfs_fail_loc=a09, val=0*** [ 5043.698700] Lustre: Skipped 383 previous similar messages [ 5051.705012] Lustre: *** cfs_fail_loc=a09, val=0*** [ 5051.706314] Lustre: Skipped 849 previous similar messages [ 5067.715675] Lustre: *** cfs_fail_loc=a09, val=0*** [ 5067.717323] Lustre: Skipped 1729 previous similar messages [ 5099.716060] Lustre: *** cfs_fail_loc=a09, val=0*** [ 5099.717803] Lustre: Skipped 3774 previous similar messages [ 5361.050434] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5361.064100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5361.064102] Lustre: Skipped 1 previous similar message [ 5361.078968] Lustre: Skipped 1 previous similar message [ 5365.797070] LustreError: 21354:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5365.799830] LustreError: 21354:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5365.871084] Lustre: server umount lustre-MDT0000 complete [ 5367.859216] LustreError: 28632:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746161871 with bad export cookie 919254851131468102 [ 5367.866926] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5368.041857] Lustre: server umount lustre-OST0000 complete [ 5370.122352] Lustre: server umount lustre-OST0001 complete [ 5372.505747] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 5375.440827] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 5377.681580] hrtimer: interrupt took 4111205 ns [ 5379.954503] vdc: vdc1 vdc9 [ 5384.599859] vde: vde1 vde9 [ 5389.093275] vdf: vdf1 vdf9 [ 5395.903087] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 5399.438447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5399.552184] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5399.591909] Lustre: lustre-MDT0000: new disk, initializing [ 5399.822500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5399.825762] Lustre: Skipped 1 previous similar message [ 5399.853166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5401.377229] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5406.178027] Lustre: 27618:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.204.104@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5406.191431] Lustre: lustre-OST0000: new disk, initializing [ 5406.198203] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5406.201210] Lustre: Skipped 1 previous similar message [ 5407.357841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 5407.362163] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 5407.429216] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 5408.240229] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5412.432748] Lustre: lustre-OST0001: new disk, initializing [ 5412.436551] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5414.214095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 5414.220834] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 5414.294470] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 5414.432022] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5419.241385] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5420.718078] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 5420.720493] Lustre: Skipped 1 previous similar message [ 5446.697306] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 00:59:09 (1746161949) [ 5478.438104] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 00:59:41 (1746161981) [ 5515.169660] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 01:00:17 (1746162017) [ 5530.759932] Lustre: DEBUG MARKER: rename directory return 255 [ 5560.467569] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 01:01:03 (1746162063) [ 5579.061827] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 01:01:22 (1746162082) [ 5608.015664] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 01:01:51 (1746162111) [ 5667.824601] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 01:02:50 (1746162170) [ 5689.111157] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 01:03:12 (1746162192) [ 5719.486265] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 01:03:42 (1746162222) [ 5817.355642] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 01:05:20 (1746162320) [ 5818.124907] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test [ 5818.965558] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 01:05:21 (1746162321) [ 5828.352587] LustreError: 23617:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed. [ 5828.361958] LustreError: 23617:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 4 previous similar messages [ 5857.582353] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61 [ 5858.406084] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 01:06:01 (1746162361) [ 5877.365988] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63 [ 5878.075896] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non dir/files should succeed ========================================================== 01:06:21 (1746162381) [ 5910.764393] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65 [ 5911.491906] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 01:06:54 (1746162414) [ 5943.283110] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 01:07:26 (1746162446) [ 5944.143347] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode [ 5945.363934] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 01:07:28 (1746162448) [ 5963.132958] LustreError: 30475:0:(mgs_handler.c:848:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1 [ 5990.027961] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 01:08:12 (1746162492) [ 6001.249267] LustreError: 530:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 6001.273243] LustreError: 530:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 5 previous similar messages [ 6008.808518] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB) [ 6009.657200] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB) [ 6124.209354] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 01:10:27 (1746162627) [ 6157.174044] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 01:11:00 (1746162660) [ 6175.808587] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 01:11:18 (1746162678) [ 6176.562521] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode [ 6177.398291] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 01:11:20 (1746162680) [ 6178.268672] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode [ 6179.225232] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 01:11:22 (1746162682) [ 6189.226896] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB) [ 6198.336613] Lustre: DEBUG MARKER: Write... [ 6199.647425] Lustre: DEBUG MARKER: Write out of block quota ... [ 6235.892020] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 01:12:18 (1746162738) [ 6253.383721] Lustre: DEBUG MARKER: set to use default quota [ 6254.243759] Lustre: DEBUG MARKER: set default quota [ 6255.047676] Lustre: DEBUG MARKER: get default quota [ 6257.225908] Lustre: DEBUG MARKER: Test not out of quota [ 6259.434011] Lustre: DEBUG MARKER: Test out of quota [ 6265.806897] Lustre: DEBUG MARKER: Increase default quota [ 6279.416112] Lustre: DEBUG MARKER: Set quota to override default quota [ 6279.454103] LustreError: 26368:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1746767582 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [ 6285.941406] Lustre: DEBUG MARKER: Set to use default quota again [ 6299.891761] Lustre: DEBUG MARKER: Cleanup [ 6348.845122] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 01:14:11 (1746162851) [ 6365.026040] Lustre: DEBUG MARKER: set default quota for qpool1 [ 6365.806155] Lustre: DEBUG MARKER: Write from user that hasn't lqe [ 6401.847780] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 01:15:04 (1746162904) [ 6453.804705] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 01:15:56 (1746162956) [ 6539.712588] Lustre: DEBUG MARKER: Write... [ 6541.562135] Lustre: DEBUG MARKER: Write out of block quota ... [ 6650.267879] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 01:19:13 (1746163153) [ 6681.559959] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 01:19:44 (1746163184) [ 6685.031079] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 01:19:47 (1746163187) [ 6685.759167] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate [ 6686.550857] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 01:19:49 (1746163189) [ 6687.268227] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate [ 6688.072339] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 01:19:51 (1746163191) [ 6699.185876] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 01:20:02 (1746163202) [ 6699.743247] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode [ 6700.378783] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 01:20:03 (1746163203) [ 6710.029200] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB) [ 6713.049790] LustreError: 1463:0:(qmt_pool.c:1295:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms [ 6713.058775] LustreError: 1463:0:(qmt_pool.c:1295:qmt_pool_recalc()) Skipped 5 previous similar messages [ 6716.487620] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6716.488361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6716.488362] Lustre: Skipped 1 previous similar message [ 6716.506332] Lustre: Skipped 1 previous similar message [ 6718.847747] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.4@tcp (stopping) [ 6718.851209] Lustre: Skipped 1 previous similar message [ 6721.495938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6721.500885] Lustre: Skipped 1 previous similar message [ 6723.066707] LustreError: 1463:0:(qmt_pool.c:1295:qmt_pool_recalc()) cfs_fail_timeout id a07 awake [ 6723.150177] LustreError: 1644:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 6723.161396] LustreError: 1644:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6723.249154] Lustre: server umount lustre-MDT0000 complete [ 6726.541711] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6726.823990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6726.827638] Lustre: Skipped 2 previous similar messages [ 6726.939929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65) [ 6726.945148] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65) [ 6728.321974] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6745.798325] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 6745.800631] Lustre: Skipped 3 previous similar messages [ 6756.251388] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 01:20:59 (1746163259) [ 6774.988614] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 01:21:18 (1746163278) [ 6795.407566] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 01:21:38 (1746163298) [ 6812.382083] LustreError: 27791:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-OST0000: Delaying reintegration for qtype:0 until pending updates are flushed. [ 6812.388554] LustreError: 27791:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 2 previous similar messages [ 6826.991924] Lustre: *** cfs_fail_loc=a08, val=0*** [ 6826.993539] Lustre: Skipped 956 previous similar messages [ 6826.998174] Lustre: *** cfs_fail_loc=a08, val=0*** [ 6890.307358] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 01:23:13 (1746163393) [ 6945.030626] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 01:24:08 (1746163448) [ 7062.246681] LustreError: 6117:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1746768365 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [ 7222.602250] LustreError: 6117:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1746768525 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [ 7377.135562] LustreError: 2329:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:md-0x0 id:1000 enforced:1 hard:2048 soft:2048 granted:65536 time:1746768680 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [ 7424.965279] Lustre: DEBUG MARKER: == sanity-quota test 87: lfs quota -a should print default quota setting ========================================================== 01:32:08 (1746163928) [ 7431.626686] Lustre: *** cfs_fail_loc=a09, val=0*** [ 7460.059766] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.4@tcp (stopping) [ 7461.959031] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7461.959725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7461.965739] Lustre: Skipped 1 previous similar message [ 7463.829514] LustreError: 4076:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 7463.832124] LustreError: 4076:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7463.888650] Lustre: server umount lustre-MDT0000 complete [ 7466.427841] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7466.641808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7466.698261] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 7466.699973] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 7467.950204] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7471.626288] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 7471.629075] Lustre: Skipped 1 previous similar message [ 7488.364836] Lustre: DEBUG MARKER: == sanity-quota test 88: Writing over quota should not hang ========================================================== 01:33:11 (1746163991) [ 7488.779385] Lustre: DEBUG MARKER: SKIP: sanity-quota test_88 require client with >4k pages [ 7489.283506] Lustre: DEBUG MARKER: == sanity-quota test 89: Show default quota with squash_uid ========================================================== 01:33:12 (1746163992) [ 7526.136957] Lustre: DEBUG MARKER: == sanity-quota test 90a: lfs quota should work without mount point ========================================================== 01:33:49 (1746164029) [ 7533.066205] Lustre: DEBUG MARKER: sanity-quota test_90a: @@@@@@ FAIL: quota info not Disk usr quotas