[ 3238.217130] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 3240.516670] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 3242.137500] LustreError: 5114:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 3242.565996] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3243.319884] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 3244.361936] Lustre: Failing over lustre-MDT0000
[ 3244.536331] LustreError: 5495:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup
[ 3244.543237] LustreError: 5495:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message
[ 3244.698427] Lustre: server umount lustre-MDT0000 complete
[ 3258.150892] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3258.344388] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3258.452785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3258.494945] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3258.499550] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 3258.580961] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 3258.602617] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:589 to 0x280000400:609)
[ 3258.602668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:593 to 0x240000400:609)
[ 3260.113923] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3261.354591] Lustre: 3300:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746141719/real 1746141719]  req@ffff880094c55c00 x1830958938753792/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1746141735 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0
[ 3263.366003] Lustre: 3298:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746141721/real 1746141721]  req@ffff880092359180 x1830958938754176/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746141737 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 3263.422725] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.202.149@tcp (at 0@lo)
[ 3264.568478] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3265.317864] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3266.542081] LustreError: 7165:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-OST0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 3266.554489] LustreError: 7165:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 6 previous similar messages
[ 3268.419661] Lustre: 3300:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746141726/real 1746141726]  req@ffff880092359500 x1830958938754432/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746141742 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 3268.434460] Lustre: 3300:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 3273.479958] Lustre: DEBUG MARKER: (dd_pid=1361, time=5, timeout=600)
[ 3274.435636] Lustre: 3301:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746141731/real 1746141731]  req@ffff880094b21500 x1830958938754816/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746141747 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 3274.475145] Lustre: 3301:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 3308.386410] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 3311.017950] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 3312.722524] LustreError: 9666:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 3313.253283] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3314.137227] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 3315.506581] Lustre: Failing over lustre-MDT0000
[ 3315.818668] LustreError: 10365:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup
[ 3315.825839] LustreError: 10365:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3315.952772] Lustre: server umount lustre-MDT0000 complete
[ 3330.406876] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3330.910846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3330.958992] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3332.716867] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3332.770991] Lustre: 3299:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746141790/real 1746141790]  req@ffff880092b36680 x1830958938783616/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1746141806 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0
[ 3332.783474] Lustre: 3299:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 3333.670353] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 3333.757226] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 3333.783583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:611 to 0x240000400:641)
[ 3333.784004] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:589 to 0x280000400:641)
[ 3337.558682] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3337.866083] LustreError: 7165:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-OST0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 3338.497142] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3343.810071] Lustre: 3298:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746141802/real 1746141802]  req@ffff8800920c2300 x1830958938784768/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746141818 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 3343.829408] Lustre: 3298:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 3348.199303] Lustre: DEBUG MARKER: (dd_pid=3833, time=6, timeout=600)
[ 3391.712374] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 19:24:25 (1746141865)
[ 3405.178925] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 3405.936536] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 3408.385254] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 3409.199766] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 3431.232320] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 19:25:05 (1746141905)
[ 3440.368853] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 3449.823272] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 3450.574675] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 3451.225882] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 3451.842908] Lustre: DEBUG MARKER: Set quota for 1 times
[ 3453.809859] Lustre: DEBUG MARKER: Set quota for 2 times
[ 3455.778218] Lustre: DEBUG MARKER: Set quota for 3 times
[ 3457.682114] Lustre: DEBUG MARKER: Set quota for 4 times
[ 3459.540353] Lustre: DEBUG MARKER: Set quota for 5 times
[ 3461.491426] Lustre: DEBUG MARKER: Set quota for 6 times
[ 3463.696970] Lustre: DEBUG MARKER: Set quota for 7 times
[ 3465.665101] Lustre: DEBUG MARKER: Set quota for 8 times
[ 3467.676005] Lustre: DEBUG MARKER: Set quota for 9 times
[ 3469.719164] Lustre: DEBUG MARKER: Set quota for 10 times
[ 3471.764125] Lustre: DEBUG MARKER: Set quota for 11 times
[ 3473.893416] Lustre: DEBUG MARKER: Set quota for 12 times
[ 3475.962402] Lustre: DEBUG MARKER: Set quota for 13 times
[ 3478.170652] Lustre: DEBUG MARKER: Set quota for 14 times
[ 3480.352430] Lustre: DEBUG MARKER: Set quota for 15 times
[ 3510.713916] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 19:26:24 (1746141984)
[ 3512.412221] LustreError: 23172:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-OST0001: Delaying reintegration for qtype:2 until pending updates are flushed.
[ 3512.427033] LustreError: 23172:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 1 previous similar message
[ 3519.045346] LustreError: 23758:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup
[ 3519.053683] LustreError: 23758:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3519.143691] Lustre: server umount lustre-MDT0000 complete
[ 3520.974032] LustreError: 27122:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746141995 with bad export cookie 18375103072182635473
[ 3520.979863] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3531.023777] Lustre: server umount lustre-OST0000 complete
[ 3543.197229] Lustre: server umount lustre-OST0001 complete
[ 3549.300939] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local
[ 3553.233620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3554.611266] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3555.851731] Lustre: Modifying parameter general.debug_raw_pointers in log params
[ 3558.173291] Lustre: 26765:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.149@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 3558.328735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 3558.378709] mount.lustre (26765) used greatest stack depth: 9616 bytes left
[ 3560.762832] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3563.311496] LustreError: 26768:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3563.338658] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:644 to 0x240000400:673)
[ 3564.101980] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 3565.293878] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:643 to 0x280000400:673)
[ 3566.344917] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3570.355367] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3572.155301] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 3588.811451] LustreError: 30013:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup
[ 3588.817057] LustreError: 30013:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages
[ 3588.921125] Lustre: server umount lustre-MDT0000 complete
[ 3591.104415] LustreError: 25742:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746142065 with bad export cookie 18375103072182648388
[ 3591.109907] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3601.143582] Lustre: server umount lustre-OST0000 complete
[ 3612.883153] Lustre: server umount lustre-OST0001 complete
[ 3618.362585] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local
[ 3622.197225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3623.692185] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3624.968966] Lustre: Modifying parameter general.debug_raw_pointers in log params
[ 3627.487742] Lustre: 559:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.149@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 3627.492335] Lustre: 559:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 1 previous similar message
[ 3630.020096] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3632.616131] LustreError: 563:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3632.630378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:644 to 0x240000400:705)
[ 3635.072998] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:643 to 0x280000400:705)
[ 3635.840997] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3640.261402] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3641.807143] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 3651.102094] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 19:28:45 (1746142125)
[ 3651.730806] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 3652.444951] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 19:28:46 (1746142126)
[ 3687.430067] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 19:29:21 (1746142161)
[ 3716.235428] Lustre: DEBUG MARKER: Write...
[ 3717.876289] Lustre: DEBUG MARKER: Write out of block quota ...
[ 3754.976272] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 19:30:29 (1746142229)
[ 3757.954592] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 19:30:32 (1746142232)
[ 3763.573878] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 19:30:37 (1746142237)
[ 3768.375178] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 19:30:42 (1746142242)
[ 3773.007254] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 19:30:46 (1746142246)
[ 3819.515817] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 3898.654117] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 4009.269649] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 19:34:43 (1746142483)
[ 4043.784672] Lustre: DEBUG MARKER: Restart...
[ 4049.132559] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4049.139602] Lustre: Skipped 3 previous similar messages
[ 4049.144099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4049.146883] Lustre: Skipped 1 previous similar message
[ 4051.115910] LustreError: 25702:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup
[ 4051.122370] LustreError: 25702:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages
[ 4051.257040] Lustre: server umount lustre-MDT0000 complete
[ 4052.984752] LustreError: 2631:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746142527 with bad export cookie 18375103072182649921
[ 4052.985862] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4052.996771] LustreError: 2631:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 4053.021108] Lustre: server umount lustre-OST0000 complete
[ 4055.021594] Lustre: server umount lustre-OST0001 complete
[ 4061.240889] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local
[ 4065.057768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4065.060860] Lustre: Skipped 2 previous similar messages
[ 4066.704460] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4068.174714] Lustre: Modifying parameter general.debug_raw_pointers in log params
[ 4071.048532] Lustre: 28675:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.149@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 4071.053685] Lustre: 28675:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 1 previous similar message
[ 4073.755137] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4075.244034] LustreError: 28679:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4075.276861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:715 to 0x240000400:737)
[ 4078.932417] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:713 to 0x280000400:737)
[ 4079.947447] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4083.686277] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4084.936439] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 4119.804830] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 19:36:33 (1746142593)
[ 4170.127425] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 19:37:24 (1746142644)
[ 5213.257197] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 19:54:46 (1746143686)
[ 5224.542104] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5224.542666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5224.560302] Lustre: Skipped 2 previous similar messages
[ 5229.338837] LustreError: 21423:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup
[ 5229.343042] LustreError: 21423:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages
[ 5229.451803] Lustre: server umount lustre-MDT0000 complete
[ 5231.930412] LustreError: 27638:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746143706 with bad export cookie 18375103072182656277
[ 5231.935537] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5231.987681] Lustre: server umount lustre-OST0000 complete
[ 5234.673571] Lustre: server umount lustre-OST0001 complete
[ 5241.154984] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local
[ 5245.235654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5245.239510] Lustre: Skipped 2 previous similar messages
[ 5246.889898] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5248.502663] Lustre: Modifying parameter general.debug_raw_pointers in log params
[ 5251.477126] Lustre: 24403:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.149@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 5251.481709] Lustre: 24403:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 1 previous similar message
[ 5251.708496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5254.469291] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5255.683101] LustreError: 24407:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5255.706653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5740 to 0x240000400:5761)
[ 5258.419521] Lustre: 25442:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.149@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 5258.689340] mount.lustre (25442) used greatest stack depth: 9552 bytes left
[ 5260.571010] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5738 to 0x280000400:5761)
[ 5261.336642] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5265.950999] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5267.705890] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 5298.288069] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 19:56:12 (1746143772)
[ 5324.514732] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 19:56:38 (1746143798)
[ 5352.745802] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 19:57:06 (1746143826)
[ 5353.345957] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 5354.225729] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 19:57:08 (1746143828)
[ 5354.949371] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 5355.818775] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 19:57:09 (1746143829)
[ 5365.546395] LustreError: 30605:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 5365.557026] LustreError: 30605:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 2 previous similar messages
[ 5398.048830] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 19:57:52 (1746143872)
[ 5426.502315] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 19:58:20 (1746143900)
[ 5495.470033] Lustre: DEBUG MARKER: == sanity-quota test 49: lfs quota -a prints the quota usage for all quota IDs ========================================================== 19:59:29 (1746143969)
[ 5503.234127] Lustre: *** cfs_fail_loc=a09, val=0***
[ 5503.237484] Lustre: Skipped 1 previous similar message
[ 5504.238248] Lustre: *** cfs_fail_loc=a09, val=0***
[ 5504.239622] Lustre: Skipped 91 previous similar messages
[ 5506.249653] Lustre: *** cfs_fail_loc=a09, val=0***
[ 5506.251765] Lustre: Skipped 225 previous similar messages
[ 5510.275330] Lustre: *** cfs_fail_loc=a09, val=0***
[ 5510.277475] Lustre: Skipped 405 previous similar messages
[ 5518.287545] Lustre: *** cfs_fail_loc=a09, val=0***
[ 5518.289650] Lustre: Skipped 905 previous similar messages
[ 5534.300113] Lustre: *** cfs_fail_loc=a09, val=0***
[ 5534.301956] Lustre: Skipped 1709 previous similar messages
[ 5566.301489] Lustre: *** cfs_fail_loc=a09, val=0***
[ 5566.308253] Lustre: Skipped 2962 previous similar messages
[ 5884.637705] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5884.639460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5884.639462] Lustre: Skipped 1 previous similar message
[ 5884.653205] Lustre: Skipped 1 previous similar message
[ 5886.418596] LustreError: 14637:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup
[ 5886.421568] LustreError: 14637:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages
[ 5886.505807] Lustre: server umount lustre-MDT0000 complete
[ 5888.284485] LustreError: 23382:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746144362 with bad export cookie 18375103072184422636
[ 5888.289471] LustreError: 23382:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 5888.295876] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5888.452065] Lustre: server umount lustre-OST0000 complete
[ 5890.481743] Lustre: server umount lustre-OST0001 complete
[ 5893.155567] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid
[ 5895.974376] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local
[ 5900.172587]  vdc: vdc1 vdc9
[ 5905.178553]  vde: vde1 vde9
[ 5910.811789]  vdf: vdf1 vdf9
[ 5918.862772] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local
[ 5923.041836] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000
[ 5923.129425] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 5923.188002] Lustre: lustre-MDT0000: new disk, initializing
[ 5923.366926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5923.370103] Lustre: Skipped 1 previous similar message
[ 5923.403808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 5924.813301] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5927.569952] Lustre: Modifying parameter general.debug_raw_pointers in log params
[ 5930.785731] Lustre: 20906:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.149@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 5930.801616] Lustre: lustre-OST0000: new disk, initializing
[ 5930.803783] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 5930.805926] Lustre: Skipped 1 previous similar message
[ 5930.861870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5932.230555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[ 5932.234836] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[ 5932.321763] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[ 5934.080382] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5939.226000] Lustre: 22223:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.149@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 5939.238581] Lustre: lustre-OST0001: new disk, initializing
[ 5939.240598] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 5939.271543] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5940.937449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[ 5940.942365] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[ 5941.005348] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[ 5941.712057] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5947.640993] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5949.420437] Lustre: Setting parameter general.lod.*.mdt_hash in log params
[ 5977.509842] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 20:07:31 (1746144451)
[ 6005.214730] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 20:07:59 (1746144479)
[ 6035.369585] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 20:08:29 (1746144509)
[ 6048.429801] Lustre: DEBUG MARKER: rename directory return 255
[ 6075.526951] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 20:09:09 (1746144549)
[ 6093.926348] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 20:09:28 (1746144568)
[ 6122.161535] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 20:09:56 (1746144596)
[ 6158.252374] LustreError: 31821:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 6180.074819] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 20:10:54 (1746144654)
[ 6201.945645] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 20:11:16 (1746144676)
[ 6231.270852] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 20:11:45 (1746144705)
[ 6315.321115] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 20:13:09 (1746144789)
[ 6315.988944] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[ 6316.834104] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 20:13:10 (1746144790)
[ 6325.452125] LustreError: 15751:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 6325.460545] LustreError: 15751:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 4 previous similar messages
[ 6357.414207] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[ 6358.230977] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 20:13:52 (1746144832)
[ 6375.839200] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[ 6376.553745] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non dir/files should succeed ========================================================== 20:14:10 (1746144850)
[ 6407.682179] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[ 6408.650158] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 20:14:42 (1746144882)
[ 6442.040524] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 20:15:15 (1746144915)
[ 6442.772414] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[ 6443.680368] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 20:15:17 (1746144917)
[ 6460.883067] LustreError: 22614:0:(mgs_handler.c:848:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[ 6487.852462] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 20:16:01 (1746144961)
[ 6498.773095] LustreError: 25068:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed.
[ 6498.781241] LustreError: 25068:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 5 previous similar messages
[ 6505.073349] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[ 6505.959460] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[ 6610.970602] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 20:18:05 (1746145085)
[ 6641.017599] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 20:18:35 (1746145115)
[ 6659.161467] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 20:18:53 (1746145133)
[ 6659.712901] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[ 6660.376833] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 20:18:54 (1746145134)
[ 6661.059715] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[ 6661.867869] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 20:18:55 (1746145135)
[ 6671.010333] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[ 6679.013986] Lustre: DEBUG MARKER: Write...
[ 6680.201131] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6720.781455] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 20:19:54 (1746145194)
[ 6736.573392] Lustre: DEBUG MARKER: set to use default quota
[ 6737.226131] Lustre: DEBUG MARKER: set default quota
[ 6737.857849] Lustre: DEBUG MARKER: get default quota
[ 6739.930859] Lustre: DEBUG MARKER: Test not out of quota
[ 6741.915692] Lustre: DEBUG MARKER: Test out of quota
[ 6747.528265] Lustre: DEBUG MARKER: Increase default quota
[ 6759.404162] Lustre: DEBUG MARKER: Set quota to override default quota
[ 6759.426928] LustreError: 24008:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45064 time:1746750033 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 6763.983326] Lustre: DEBUG MARKER: Set to use default quota again
[ 6776.364355] Lustre: DEBUG MARKER: Cleanup
[ 6825.156415] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 20:21:38 (1746145298)
[ 6841.156841] Lustre: DEBUG MARKER: set default quota for qpool1
[ 6841.794320] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[ 6874.280268] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 20:22:28 (1746145348)
[ 6919.565557] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 20:23:13 (1746145393)
[ 7002.064859] Lustre: DEBUG MARKER: Write...
[ 7003.691872] Lustre: DEBUG MARKER: Write out of block quota ...
[ 7112.541811] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 20:26:26 (1746145586)
[ 7142.410409] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 20:26:56 (1746145616)
[ 7145.735078] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 20:26:59 (1746145619)
[ 7146.547050] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate
[ 7147.545200] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 20:27:01 (1746145621)
[ 7148.443109] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate
[ 7149.431295] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 20:27:03 (1746145623)
[ 7160.972717] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 20:27:14 (1746145634)
[ 7161.645307] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[ 7162.452408] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 20:27:16 (1746145636)
[ 7172.755903] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[ 7176.637666] LustreError: 27297:0:(qmt_pool.c:1295:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[ 7176.641317] LustreError: 27297:0:(qmt_pool.c:1295:qmt_pool_recalc()) Skipped 1 previous similar message
[ 7182.667630] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.49@tcp (stopping)
[ 7182.670797] Lustre: Skipped 1 previous similar message
[ 7182.892297] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7182.903585] Lustre: Skipped 1 previous similar message
[ 7186.643653] LustreError: 27297:0:(qmt_pool.c:1295:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[ 7186.740157] LustreError: 27428:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup
[ 7186.742010] LustreError: 27428:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages
[ 7186.835263] Lustre: server umount lustre-MDT0000 complete
[ 7190.565578] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7190.850239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7190.930469] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[ 7190.941936] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[ 7192.485860] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7202.839558] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to  (at 0@lo)
[ 7202.842043] Lustre: Skipped 3 previous similar messages
[ 7221.799432] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 20:28:15 (1746145695)
[ 7241.750169] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 20:28:35 (1746145715)
[ 7260.804939] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 20:28:54 (1746145734)
[ 7276.566910] LustreError: 21073:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-OST0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 7276.573939] LustreError: 21073:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 2 previous similar messages
[ 7291.417323] Lustre: *** cfs_fail_loc=a08, val=0***
[ 7291.419203] Lustre: Skipped 1696 previous similar messages
[ 7291.425125] Lustre: *** cfs_fail_loc=a08, val=0***
[ 7360.624916] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 20:30:34 (1746145834)
[ 7416.474535] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 20:31:30 (1746145890)
[ 7554.088117] LustreError: 28119:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1746750828 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 7729.079605] LustreError: 29218:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1746751003 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 7900.058963] LustreError: 29218:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:1000 enforced:1 hard:2048 soft:2048 granted:65536 time:1746751174 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 7968.530397] Lustre: DEBUG MARKER: == sanity-quota test 87: lfs quota -a should print default quota setting ========================================================== 20:40:42 (1746146442)
[ 7976.234935] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7977.243425] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7977.245103] Lustre: Skipped 115 previous similar messages
[ 7979.247191] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7979.249300] Lustre: Skipped 257 previous similar messages
[ 7983.252682] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7983.254849] Lustre: Skipped 665 previous similar messages
[ 8019.114583] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.49@tcp (stopping)
[ 8019.119848] Lustre: Skipped 2 previous similar messages
[ 8019.197921] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8021.635524] LustreError: 30141:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup
[ 8021.637752] LustreError: 30141:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 8021.727917] Lustre: server umount lustre-MDT0000 complete
[ 8025.554915] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8025.905569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8026.029176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:68 to 0x240000400:97)
[ 8026.031133] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:97)
[ 8027.637389] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8038.860892] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to  (at 0@lo)
[ 8038.873571] Lustre: Skipped 1 previous similar message
[ 8049.125400] Lustre: DEBUG MARKER: == sanity-quota test 88: Writing over quota should not hang ========================================================== 20:42:03 (1746146523)
[ 8049.872078] Lustre: DEBUG MARKER: SKIP: sanity-quota test_88 require client with >4k pages
[ 8050.683770] Lustre: DEBUG MARKER: == sanity-quota test 89: Show default quota with squash_uid ========================================================== 20:42:04 (1746146524)
[ 8093.261962] Lustre: DEBUG MARKER: == sanity-quota test 90a: lfs quota should work without mount point ========================================================== 20:42:47 (1746146567)
[ 8123.443804] Lustre: DEBUG MARKER: == sanity-quota test 90b: lfs quota should work with multiple mount points ========================================================== 20:43:17 (1746146597)
[ 8132.541468] Lustre: DEBUG MARKER: sanity-quota test_90b: @@@@@@ FAIL: quota info from /mnt/lustre2 not ' /mnt/lustre2 root 11731 0 0 - 376 0 0 -