[ 2825.369602] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 2826.797413] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 2827.718667] LustreError: 20579:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 2827.988092] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2828.412665] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 2829.033320] Lustre: Failing over lustre-MDT0000
[ 2829.035789] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2829.043712] Lustre: Skipped 1 previous similar message
[ 2829.120694] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.44@tcp (stopping)
[ 2829.183295] Lustre: server umount lustre-MDT0000 complete
[ 2841.436051] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2841.656918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2841.687479] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 2842.746359] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2848.623150] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to  (at 0@lo)
[ 2849.148308] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 2849.180309] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 2849.198871] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1584 to 0x280000400:1601)
[ 2849.199060] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1588 to 0x240000400:1633)
[ 2850.178847] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2850.507040] LustreError: 8412:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-OST0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 2850.511806] LustreError: 8412:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 6 previous similar messages
[ 2850.670343] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2859.109158] Lustre: DEBUG MARKER: (dd_pid=4685, time=6, timeout=600)
[ 2885.338923] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 2886.635542] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 2887.480545] LustreError: 25132:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 2887.744752] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2888.150808] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 2888.703895] Lustre: Failing over lustre-MDT0000
[ 2888.836829] Lustre: server umount lustre-MDT0000 complete
[ 2900.522506] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2900.638393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2900.656004] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 2901.454011] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2903.198685] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 2903.223385] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 2903.240206] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1584 to 0x280000400:1633)
[ 2903.240222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1635 to 0x240000400:1665)
[ 2904.070672] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2904.488829] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2904.637311] Lustre: 3299:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1735101666/real 1735101666]  req@ffff8801338b8a80 x1819382989074176/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1735101682 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0
[ 2909.627349] Lustre: 3298:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1735101671/real 1735101671]  req@ffff880089de8e00 x1819382989074560/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1735101687 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 2910.711818] Lustre: DEBUG MARKER: (dd_pid=7085, time=4, timeout=600)
[ 2914.651336] Lustre: 3300:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1735101676/real 1735101676]  req@ffff880089dea680 x1819382989074816/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1735101692 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 2914.659500] Lustre: 3300:0:(client.c:2364:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 2945.166342] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 23:42:01 (1735101721)
[ 2952.658522] LustreError: 30689:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:1 until pending updates are flushed.
[ 2955.505611] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 2955.952189] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 2957.316431] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 2957.794517] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 2978.607983] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 23:42:35 (1735101755)
[ 2985.539393] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 2993.508775] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 2993.957543] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 2994.379978] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 2994.909908] Lustre: DEBUG MARKER: Set quota for 1 times
[ 2996.525589] Lustre: DEBUG MARKER: Set quota for 2 times
[ 2998.124808] Lustre: DEBUG MARKER: Set quota for 3 times
[ 2999.691803] Lustre: DEBUG MARKER: Set quota for 4 times
[ 3001.298792] Lustre: DEBUG MARKER: Set quota for 5 times
[ 3002.911157] Lustre: DEBUG MARKER: Set quota for 6 times
[ 3004.563822] Lustre: DEBUG MARKER: Set quota for 7 times
[ 3006.188520] Lustre: DEBUG MARKER: Set quota for 8 times
[ 3007.846434] Lustre: DEBUG MARKER: Set quota for 9 times
[ 3009.593985] Lustre: DEBUG MARKER: Set quota for 10 times
[ 3011.312715] Lustre: DEBUG MARKER: Set quota for 11 times
[ 3012.945724] Lustre: DEBUG MARKER: Set quota for 12 times
[ 3014.614917] Lustre: DEBUG MARKER: Set quota for 13 times
[ 3016.344728] Lustre: DEBUG MARKER: Set quota for 14 times
[ 3018.129943] Lustre: DEBUG MARKER: Set quota for 15 times
[ 3019.773462] Lustre: DEBUG MARKER: Set quota for 16 times
[ 3021.459769] Lustre: DEBUG MARKER: Set quota for 17 times
[ 3023.066519] Lustre: DEBUG MARKER: Set quota for 18 times
[ 3049.257964] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 23:43:46 (1735101826)
[ 3055.883969] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3055.887674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3055.891807] Lustre: Skipped 3 previous similar messages
[ 3059.430197] Lustre: server umount lustre-MDT0000 complete
[ 3060.570926] LustreError: 17606:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1735101838 with bad export cookie 8335466551091319899
[ 3060.572102] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3060.579290] LustreError: 17606:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 3060.604762] Lustre: server umount lustre-OST0000 complete
[ 3061.764773] Lustre: server umount lustre-OST0001 complete
[ 3064.787499] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local
[ 3066.788982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3067.627461] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3068.446561] Lustre: Modifying parameter general.debug_raw_pointers in log params
[ 3069.865256] Lustre: 10553:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.144@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 3069.960744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 3071.226579] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3074.537369] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3077.184882] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1636 to 0x280000400:1665)
[ 3077.184914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1667 to 0x240000400:1697)
[ 3082.253944] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3083.201714] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 3096.731421] Lustre: server umount lustre-MDT0000 complete
[ 3097.947911] LustreError: 13033:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1735101875 with bad export cookie 8335466551091340003
[ 3097.954451] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3107.977213] Lustre: server umount lustre-OST0000 complete
[ 3119.121359] Lustre: server umount lustre-OST0001 complete
[ 3122.386138] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local
[ 3124.539315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3124.542150] Lustre: Skipped 1 previous similar message
[ 3125.410285] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3126.274489] Lustre: Modifying parameter general.debug_raw_pointers in log params
[ 3127.687032] Lustre: 16756:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.144@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 3127.690702] Lustre: 16756:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message
[ 3129.164867] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3131.114677] mount.lustre (17783) used greatest stack depth: 9504 bytes left
[ 3132.522061] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3135.080217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1667 to 0x240000400:1729)
[ 3135.080219] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1636 to 0x280000400:1697)
[ 3140.148013] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3141.007130] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 3148.203022] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 23:45:25 (1735101925)
[ 3148.549927] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 3148.949770] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 23:45:25 (1735101925)
[ 3178.272807] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 23:45:55 (1735101955)
[ 3193.147433] Lustre: DEBUG MARKER: Write...
[ 3193.710055] Lustre: DEBUG MARKER: Write out of block quota ...
[ 3223.016900] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 23:46:39 (1735101999)
[ 3224.354051] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 23:46:41 (1735102001)
[ 3227.931885] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 23:46:44 (1735102004)
[ 3230.576956] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 23:46:47 (1735102007)
[ 3232.989818] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 23:46:49 (1735102009)
[ 3272.722635] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 3336.001725] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 3444.196853] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 23:50:21 (1735102221)
[ 3471.801330] Lustre: DEBUG MARKER: Restart...
[ 3475.627720] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3475.628173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3475.628175] Lustre: Skipped 1 previous similar message
[ 3475.634765] Lustre: Skipped 1 previous similar message
[ 3478.554564] Lustre: server umount lustre-MDT0000 complete
[ 3479.560776] LustreError: 15745:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1735102257 with bad export cookie 8335466551091341536
[ 3479.561987] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3479.566844] LustreError: 15745:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 3479.584282] Lustre: server umount lustre-OST0000 complete
[ 3480.618998] Lustre: server umount lustre-OST0001 complete
[ 3483.363612] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local
[ 3485.305857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3485.308296] Lustre: Skipped 2 previous similar messages
[ 3486.100041] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3486.847056] Lustre: Modifying parameter general.debug_raw_pointers in log params
[ 3488.169372] Lustre: 13784:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.144@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 3488.173837] Lustre: 13784:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message
[ 3489.564144] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3492.573172] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3495.306835] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1739 to 0x240000400:1761)
[ 3495.307106] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1705 to 0x280000400:1729)
[ 3500.123704] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3501.021075] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 3533.490067] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 23:51:50 (1735102310)
[ 3578.792332] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 23:52:35 (1735102355)
[ 3888.469188] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 23:57:45 (1735102665)
[ 3895.948339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3895.951230] Lustre: Skipped 2 previous similar messages
[ 3897.234321] Lustre: server umount lustre-MDT0000 complete
[ 3898.301918] LustreError: 12775:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1735102675 with bad export cookie 8335466551091347738
[ 3898.303149] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3898.310795] LustreError: 12775:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 3900.321121] Lustre: server umount lustre-OST0000 complete
[ 3901.377605] Lustre: server umount lustre-OST0001 complete
[ 3904.216755] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local
[ 3906.090918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3906.092731] Lustre: Skipped 2 previous similar messages
[ 3906.877632] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3907.623812] Lustre: Modifying parameter general.debug_raw_pointers in log params
[ 3908.906754] Lustre: 30310:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.144@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 3908.911045] Lustre: 30310:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message
[ 3910.252336] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3913.332756] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3916.070400] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6764 to 0x240000400:6785)
[ 3916.081626] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6730 to 0x280000400:6753)
[ 3920.864162] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3944.647042] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 23:58:41 (1735102721)
[ 3968.558350] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 23:59:05 (1735102745)
[ 3993.637274] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 23:59:30 (1735102770)
[ 3993.959454] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 3994.336950] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 23:59:31 (1735102771)
[ 3994.656749] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 3995.005688] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 23:59:31 (1735102771)
[ 4003.329182] LustreError: 4151:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 4003.335433] LustreError: 4151:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 3 previous similar messages
[ 4027.904481] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 00:00:04 (1735102804)
[ 4050.544590] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 00:00:27 (1735102827)
[ 4108.162977] Lustre: DEBUG MARKER: == sanity-quota test 49: lfs quota -a prints the quota usage for all quota IDs ========================================================== 00:01:25 (1735102885)
[ 4114.233850] Lustre: *** cfs_fail_loc=a09, val=0***
[ 4114.235016] Lustre: Skipped 1 previous similar message
[ 4115.234364] Lustre: *** cfs_fail_loc=a09, val=0***
[ 4115.235670] Lustre: Skipped 326 previous similar messages
[ 4117.236203] Lustre: *** cfs_fail_loc=a09, val=0***
[ 4117.237476] Lustre: Skipped 658 previous similar messages
[ 4121.238862] Lustre: *** cfs_fail_loc=a09, val=0***
[ 4121.240016] Lustre: Skipped 1333 previous similar messages
[ 4129.240038] Lustre: *** cfs_fail_loc=a09, val=0***
[ 4129.241587] Lustre: Skipped 2629 previous similar messages
[ 4246.603716] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4246.603928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4246.612511] Lustre: Skipped 3 previous similar messages
[ 4248.576923] Lustre: 22182:0:(qsd_writeback.c:661:qsd_cleanup_deferred()) lustre-MDT0000: Free global deferred upd: ID:3, ver:2126/2124
[ 4248.636624] Lustre: server umount lustre-MDT0000 complete
[ 4249.762074] LustreError: 29284:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1735103027 with bad export cookie 8335466551093077690
[ 4249.765242] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4249.881476] Lustre: server umount lustre-OST0000 complete
[ 4251.077825] Lustre: server umount lustre-OST0001 complete
[ 4252.577199] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid
[ 4254.121327] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local
[ 4256.346775]  vdc: vdc1 vdc9
[ 4258.600645]  vde: vde1 vde9
[ 4260.959624]  vdf: vdf1 vdf9
[ 4264.298073] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local
[ 4266.234720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000
[ 4266.236778] Lustre: Skipped 1 previous similar message
[ 4266.286266] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 4266.325172] Lustre: lustre-MDT0000: new disk, initializing
[ 4266.389817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4266.391477] Lustre: Skipped 2 previous similar messages
[ 4266.405244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 4267.211710] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4270.759587] Lustre: 28330:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.144@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 4270.764883] Lustre: 28330:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message
[ 4270.774400] Lustre: lustre-OST0000: new disk, initializing
[ 4270.777294] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 4270.780780] Lustre: Skipped 1 previous similar message
[ 4272.119333] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4272.689901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[ 4272.693204] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[ 4272.719353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[ 4275.301481] Lustre: lustre-OST0001: new disk, initializing
[ 4275.303363] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 4276.694837] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4276.986907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[ 4276.991826] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[ 4277.030951] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[ 4280.634009] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4303.023468] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 00:04:39 (1735103079)
[ 4330.141018] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 00:05:07 (1735103107)
[ 4360.107974] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 00:05:37 (1735103137)
[ 4369.089497] Lustre: DEBUG MARKER: rename directory return 255
[ 4390.301087] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 00:06:07 (1735103167)
[ 4405.186734] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 00:06:22 (1735103182)
[ 4422.973215] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 00:06:39 (1735103199)
[ 4448.718424] LustreError: 7215:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 4469.911548] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 00:07:26 (1735103246)
[ 4486.990974] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 00:07:43 (1735103263)
[ 4515.765447] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 00:08:12 (1735103292)
[ 4580.938853] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 00:09:17 (1735103357)
[ 4581.289875] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[ 4581.695801] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 00:09:18 (1735103358)
[ 4589.677201] LustreError: 23736:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 4589.680292] LustreError: 23736:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 4 previous similar messages
[ 4610.673998] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[ 4611.094667] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 00:09:48 (1735103388)
[ 4626.001558] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[ 4626.418007] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non dir/files should succeed ========================================================== 00:10:03 (1735103403)
[ 4655.146509] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[ 4655.573134] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 00:10:32 (1735103432)
[ 4686.444504] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 00:11:03 (1735103463)
[ 4686.835789] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[ 4687.297722] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 00:11:04 (1735103464)
[ 4700.516783] LustreError: 30862:0:(mgs_handler.c:844:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[ 4702.347938] Lustre: DEBUG MARKER: sanity-quota test_68: @@@@@@ FAIL: Slave number 1 for qpool1 != 2