[ 2996.605340] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 2997.612545] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 2998.335664] LustreError: 6410:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 2998.569130] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2998.917635] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 2999.415317] Lustre: Failing over lustre-MDT0000
[ 2999.536636] Lustre: server umount lustre-MDT0000 complete
[ 3011.127563] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3011.242851] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3011.248435] Lustre: Skipped 1 previous similar message
[ 3011.281988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3011.303812] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3011.440675] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 3011.471024] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 3011.485950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2184 to 0x240000400:2209)
[ 3011.485951] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2181 to 0x280000400:2209)
[ 3012.039264] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3014.464043] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3014.816761] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3016.280831] Lustre: 3308:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1741900716/real 1741900716]  req@ffff88008ddcb800 x1826512191456384/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1741900732 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0
[ 3016.290972] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to  (at 0@lo)
[ 3018.453661] LustreError: 20337:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-OST0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 3018.458136] LustreError: 20337:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 6 previous similar messages
[ 3019.287933] Lustre: 3307:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1741900719/real 1741900719]  req@ffff880134dc0e00 x1826512191456768/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1741900735 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 3023.960452] Lustre: DEBUG MARKER: (dd_pid=4616, time=7, timeout=600)
[ 3024.297905] Lustre: 3307:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1741900724/real 1741900724]  req@ffff880134dc2a00 x1826512191457152/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1741900740 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 3024.305818] Lustre: 3307:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3051.775610] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 3052.975336] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 3053.809568] LustreError: 11021:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 3054.071033] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3054.445443] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 3054.966953] Lustre: Failing over lustre-MDT0000
[ 3055.124617] Lustre: server umount lustre-MDT0000 complete
[ 3066.767726] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3066.897735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3066.916210] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3067.696503] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3069.473628] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 3069.504552] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 3069.522015] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2211 to 0x240000400:2241)
[ 3069.522088] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2181 to 0x280000400:2241)
[ 3070.348537] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3070.753853] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3071.369921] Lustre: 3309:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1741900771/real 1741900771]  req@ffff8800a1f75c00 x1826512191483520/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1741900787 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0
[ 3078.955485] Lustre: DEBUG MARKER: (dd_pid=7071, time=6, timeout=600)
[ 3081.578942] Lustre: 3309:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1741900782/real 1741900782]  req@ffff88008bce1500 x1826512191484544/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1741900798 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 3081.588664] Lustre: 3309:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 3115.990977] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 17:20:32 (1741900832)
[ 3123.989389] LustreError: 16933:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:1 until pending updates are flushed.
[ 3123.992385] LustreError: 16933:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 1 previous similar message
[ 3126.604569] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 3126.965578] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 3128.025691] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 3128.401146] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 3149.434959] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 17:21:05 (1741900865)
[ 3155.462553] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 3162.816553] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 3163.166411] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 3163.533246] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 3163.911654] Lustre: DEBUG MARKER: Set quota for 1 times
[ 3165.340904] Lustre: DEBUG MARKER: Set quota for 2 times
[ 3166.787589] Lustre: DEBUG MARKER: Set quota for 3 times
[ 3168.250299] Lustre: DEBUG MARKER: Set quota for 4 times
[ 3169.704568] Lustre: DEBUG MARKER: Set quota for 5 times
[ 3171.154110] Lustre: DEBUG MARKER: Set quota for 6 times
[ 3172.615955] Lustre: DEBUG MARKER: Set quota for 7 times
[ 3174.072046] Lustre: DEBUG MARKER: Set quota for 8 times
[ 3175.563295] Lustre: DEBUG MARKER: Set quota for 9 times
[ 3177.009152] Lustre: DEBUG MARKER: Set quota for 10 times
[ 3178.419521] Lustre: DEBUG MARKER: Set quota for 11 times
[ 3179.864953] Lustre: DEBUG MARKER: Set quota for 12 times
[ 3181.299380] Lustre: DEBUG MARKER: Set quota for 13 times
[ 3182.720249] Lustre: DEBUG MARKER: Set quota for 14 times
[ 3184.142031] Lustre: DEBUG MARKER: Set quota for 15 times
[ 3185.558286] Lustre: DEBUG MARKER: Set quota for 16 times
[ 3186.957705] Lustre: DEBUG MARKER: Set quota for 17 times
[ 3188.339065] Lustre: DEBUG MARKER: Set quota for 18 times
[ 3189.761119] Lustre: DEBUG MARKER: Set quota for 19 times
[ 3191.162513] Lustre: DEBUG MARKER: Set quota for 20 times
[ 3192.567823] Lustre: DEBUG MARKER: Set quota for 21 times
[ 3215.343186] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 17:22:11 (1741900931)
[ 3222.146333] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3222.146583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3222.152732] Lustre: Skipped 3 previous similar messages
[ 3225.154943] Lustre: server umount lustre-MDT0000 complete
[ 3226.441389] LustreError: 11006:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1741900942 with bad export cookie 5552341199115680633
[ 3226.443727] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3226.450417] LustreError: 11006:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 3226.476486] Lustre: server umount lustre-OST0000 complete
[ 3227.857352] Lustre: server umount lustre-OST0001 complete
[ 3231.836702] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local
[ 3234.470753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3235.656735] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3236.728930] Lustre: Modifying parameter general.debug_raw_pointers in log params
[ 3238.701352] Lustre: 29338:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.122@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 3238.865022] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 3240.844181] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3243.597598] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 3244.599650] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2243 to 0x280000400:2273)
[ 3244.603409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2244 to 0x240000400:2273)
[ 3245.226160] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3248.694102] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3250.152085] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 3264.626461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3264.629884] Lustre: Skipped 2 previous similar messages
[ 3269.635560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3269.638753] Lustre: Skipped 1 previous similar message
[ 3269.966309] Lustre: server umount lustre-MDT0000 complete
[ 3271.735758] LustreError: 31757:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1741900988 with bad export cookie 5552341199115703978
[ 3271.735845] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3271.744007] LustreError: 31757:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 3271.769945] Lustre: server umount lustre-OST0000 complete
[ 3273.453501] Lustre: server umount lustre-OST0001 complete
[ 3278.246920] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local
[ 3281.230475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3282.437314] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3283.573178] Lustre: Modifying parameter general.debug_raw_pointers in log params
[ 3285.431519] Lustre: 3144:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.122@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 3285.436900] Lustre: 3144:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message
[ 3285.606461] mount.lustre (3144) used greatest stack depth: 9488 bytes left
[ 3287.112295] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3291.221428] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3291.442803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2244 to 0x240000400:2305)
[ 3291.446964] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2243 to 0x280000400:2305)
[ 3294.161588] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3295.204291] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 3303.352736] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 17:23:39 (1741901019)
[ 3303.830486] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 3304.420134] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 17:23:40 (1741901020)
[ 3334.719007] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 17:24:10 (1741901050)
[ 3351.650759] Lustre: DEBUG MARKER: Write...
[ 3352.359882] Lustre: DEBUG MARKER: Write out of block quota ...
[ 3384.603680] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 17:25:00 (1741901100)
[ 3386.138521] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 17:25:02 (1741901102)
[ 3389.771365] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 17:25:05 (1741901105)
[ 3392.433946] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 17:25:08 (1741901108)
[ 3394.893327] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 17:25:11 (1741901111)
[ 3435.134519] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 3502.806567] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 3604.598186] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 17:28:40 (1741901320)
[ 3633.955870] Lustre: DEBUG MARKER: Restart...
[ 3636.994234] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3636.997125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3637.000162] Lustre: Skipped 3 previous similar messages
[ 3640.766759] Lustre: server umount lustre-MDT0000 complete
[ 3641.840224] LustreError: 2122:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1741901358 with bad export cookie 5552341199115705511
[ 3641.841209] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3641.849183] LustreError: 2122:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 3641.872254] Lustre: server umount lustre-OST0000 complete
[ 3643.099172] Lustre: server umount lustre-OST0001 complete
[ 3646.145816] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local
[ 3648.108707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3648.110580] Lustre: Skipped 2 previous similar messages
[ 3648.865643] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3649.587134] Lustre: Modifying parameter general.debug_raw_pointers in log params
[ 3650.828973] Lustre: 32585:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.122@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 3650.833461] Lustre: 32585:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message
[ 3652.180336] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3655.249906] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3657.989031] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2314 to 0x240000400:2337)
[ 3657.990530] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2314 to 0x280000400:2337)
[ 3662.843653] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3663.819331] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 3696.017151] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 17:30:12 (1741901412)
[ 3740.640586] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 17:30:56 (1741901456)
[ 4096.554225] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 17:36:52 (1741901812)
[ 4103.698507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4103.700927] Lustre: Skipped 2 previous similar messages
[ 4109.258205] Lustre: server umount lustre-MDT0000 complete
[ 4110.291629] LustreError: 31567:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1741901826 with bad export cookie 5552341199115711755
[ 4110.292715] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4110.299668] LustreError: 31567:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 4110.320923] Lustre: server umount lustre-OST0000 complete
[ 4111.312052] Lustre: server umount lustre-OST0001 complete
[ 4114.226672] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local
[ 4116.074945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4116.076792] Lustre: Skipped 2 previous similar messages
[ 4116.828637] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4117.568144] Lustre: Modifying parameter general.debug_raw_pointers in log params
[ 4118.784421] Lustre: 17913:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.122@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 4118.787488] Lustre: 17913:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message
[ 4120.044574] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4123.007629] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4125.825479] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:7339 to 0x240000400:7361)
[ 4125.834383] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:7339 to 0x280000400:7361)
[ 4130.533410] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4154.450157] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 17:37:50 (1741901870)
[ 4178.713514] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 17:38:14 (1741901894)
[ 4204.172654] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 17:38:40 (1741901920)
[ 4204.528179] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 4204.938973] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 17:38:41 (1741901921)
[ 4205.270808] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 4205.652307] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 17:38:41 (1741901921)
[ 4214.157081] LustreError: 24194:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 4214.163949] LustreError: 24194:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 3 previous similar messages
[ 4238.777350] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 17:39:14 (1741901954)
[ 4261.607751] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 17:39:37 (1741901977)
[ 4317.731889] Lustre: DEBUG MARKER: SKIP: sanity-quota test_49 skipping ALWAYS excluded test 49
[ 4318.125304] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 17:40:34 (1741902034)
[ 4344.613113] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 17:41:00 (1741902060)
[ 4374.706815] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 17:41:30 (1741902090)
[ 4387.256564] Lustre: DEBUG MARKER: rename directory return 255
[ 4387.600529] LustreError: 19014:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:1001 enforced:1 hard:81920 soft:0 granted:81920 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 4409.008995] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 17:42:05 (1741902125)
[ 4423.862849] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 17:42:20 (1741902140)
[ 4443.364487] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 17:42:39 (1741902159)
[ 4470.445824] LustreError: 19014:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 4489.705401] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 17:43:25 (1741902205)
[ 4506.071383] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 17:43:42 (1741902222)
[ 4534.222891] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 17:44:10 (1741902250)
[ 4604.248148] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 17:45:20 (1741902320)
[ 4604.577705] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[ 4604.974925] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 17:45:21 (1741902321)
[ 4634.188967] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[ 4634.550669] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 17:45:50 (1741902350)
[ 4649.189564] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[ 4649.572602] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non dir/files should succeed ========================================================== 17:46:05 (1741902365)
[ 4679.235799] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[ 4679.636553] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 17:46:35 (1741902395)
[ 4704.970879] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 17:47:01 (1741902421)
[ 4705.306627] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[ 4705.715740] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 17:47:01 (1741902421)
[ 4718.983531] LustreError: 30713:0:(mgs_handler.c:844:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[ 4720.848187] Lustre: DEBUG MARKER: sanity-quota test_68: @@@@@@ FAIL: Slave number 1 for qpool1 != 2