[ 3995.441020] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 3997.614855] Lustre: DEBUG MARKER: Files for user (quota_usr):
[ 3999.253981] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:1482).
[ 3999.991372] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 4001.492537] LustreError: 113369:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 4001.967921] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4002.712896] Lustre: DEBUG MARKER: Fail mds for 0 seconds
[ 4003.593867] Lustre: Failing over lustre-MDT0000
[ 4003.962826] Lustre: server umount lustre-MDT0000 complete
[ 4004.325653] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 4004.333623] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4004.342727] LustreError: Skipped 3 previous similar messages
[ 4007.731086] LustreError: 11-0: MGC192.168.204.115@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107
[ 4007.733852] LustreError: 166-1: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4007.744685] Lustre: Evicted from MGS (at 192.168.204.115@tcp) after server handle changed from 0x993efef251710d21 to 0x993efef2517d0d72
[ 4007.924164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4007.960603] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4010.055674] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4012.580849] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 4013.064076] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted.
[ 4013.092518] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1284 to 0x0:1313
[ 4013.092564] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1108 to 0x0:1153
[ 4014.639805] LustreError: 15893:0:(qsd_reint.c:635:qqi_reint_delayed()) lustre-OST0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 4014.648628] LustreError: 15893:0:(qsd_reint.c:635:qqi_reint_delayed()) Skipped 7 previous similar messages
[ 4015.365251] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4016.082080] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4021.940570] Lustre: DEBUG MARKER: (dd_pid=95775, time=3, timeout=600)
[ 4064.519878] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 4066.857539] Lustre: DEBUG MARKER: Files for user (quota_usr):
[ 4068.802333] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:1482).
[ 4069.612338] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 4070.895151] LustreError: 116334:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 4071.313126] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4072.034271] Lustre: DEBUG MARKER: Fail mds for 0 seconds
[ 4072.925168] Lustre: Failing over lustre-MDT0000
[ 4073.367034] Lustre: server umount lustre-MDT0000 complete
[ 4073.593055] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 4073.600380] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4073.608180] LustreError: Skipped 9 previous similar messages
[ 4077.005129] LustreError: 166-1: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4077.012804] Lustre: Evicted from MGS (at 192.168.204.115@tcp) after server handle changed from 0x993efef2517d0d72 to 0x993efef2517d185b
[ 4077.160693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4077.203610] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4079.058116] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4079.147996] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 4082.177567] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted.
[ 4082.196958] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1315 to 0x0:1345
[ 4082.198202] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1108 to 0x0:1185
[ 4084.742832] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4085.480280] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4092.579804] Lustre: DEBUG MARKER: (dd_pid=98469, time=4, timeout=600)
[ 4160.606911] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 18:32:20 (1778538740)
[ 4174.902243] Lustre: DEBUG MARKER: Files for user (quota_usr):
[ 4176.569666] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:1484).
[ 4178.543858] Lustre: DEBUG MARKER: Files for user (quota_usr):
[ 4180.241388] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:1484).
[ 4214.099729] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 18:33:14 (1778538794)
[ 4223.675943] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 4235.228847] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 4235.956263] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 4237.596593] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 4238.396233] Lustre: DEBUG MARKER: Set quota for 1 times
[ 4241.594400] Lustre: DEBUG MARKER: Set quota for 2 times
[ 4244.835857] Lustre: DEBUG MARKER: Set quota for 3 times
[ 4247.923919] Lustre: DEBUG MARKER: Set quota for 4 times
[ 4251.026177] Lustre: DEBUG MARKER: Set quota for 5 times
[ 4253.978715] Lustre: DEBUG MARKER: Set quota for 6 times
[ 4257.057487] Lustre: DEBUG MARKER: Set quota for 7 times
[ 4260.035860] Lustre: DEBUG MARKER: Set quota for 8 times
[ 4263.006858] Lustre: DEBUG MARKER: Set quota for 9 times
[ 4265.833069] Lustre: DEBUG MARKER: Set quota for 10 times
[ 4305.794806] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 18:34:45 (1778538885)
[ 4309.164976] LustreError: 125037:0:(qsd_reint.c:635:qqi_reint_delayed()) lustre-MDT0001: Delaying reintegration for qtype:1 until pending updates are flushed.
[ 4317.663911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4317.664946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4317.671922] Lustre: Skipped 11 previous similar messages
[ 4317.677535] Lustre: Skipped 1 previous similar message
[ 4317.681969] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 4317.687514] LustreError: Skipped 2 previous similar messages
[ 4321.356814] Lustre: server umount lustre-MDT0000 complete
[ 4322.784519] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4322.793215] LustreError: Skipped 12 previous similar messages
[ 4323.332954] LustreError: 6112:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778538904 with bad export cookie 11042543653061728347
[ 4323.334656] LustreError: 166-1: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4323.347336] LustreError: 6112:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 4331.999147] Lustre: 126046:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778538907/real 1778538907]  req@00000000653ad2a6 x1864928787437056/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778538913 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0'
[ 4332.056423] Lustre: server umount lustre-OST0000 complete
[ 4332.059893] Lustre: Skipped 1 previous similar message
[ 4340.191234] Lustre: 126252:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778538915/real 1778538915]  req@00000000554a7d64 x1864928787437568/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778538921 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0'
[ 4340.297413] Lustre: server umount lustre-OST0001 complete
[ 4347.958838] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local
[ 4352.530068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4354.506571] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4357.599826] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4357.606584] LustreError: Skipped 1 previous similar message
[ 4358.353726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 4360.358030] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4363.507535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 4365.255653] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4367.402371] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1347 to 0x0:1377
[ 4370.288052] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:3 to 0x2c0000400:33
[ 4370.298982] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1188 to 0x0:1217
[ 4370.659495] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4374.852438] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4376.451354] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 4394.066640] Lustre: server umount lustre-MDT0000 complete
[ 4394.464250] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 4396.266932] LustreError: 128937:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778538977 with bad export cookie 11042543653061733541
[ 4396.268723] LustreError: 166-1: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4396.275697] LustreError: 128937:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 4404.704019] Lustre: 131549:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778538979/real 1778538979]  req@000000007c288341 x1864928787470528/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778538985 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0'
[ 4411.871330] Lustre: 131754:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778538987/real 1778538987]  req@00000000a5baa423 x1864928787471040/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778538993 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0'
[ 4418.287290] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local
[ 4421.643056] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4421.650335] LustreError: Skipped 8 previous similar messages
[ 4421.698754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4421.701637] Lustre: Skipped 1 previous similar message
[ 4423.342244] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4427.629143] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4432.151370] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4435.891824] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:3 to 0x2c0000400:65
[ 4436.477776] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4436.899713] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1188 to 0x0:1249
[ 4436.900608] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1347 to 0x0:1409
[ 4439.947043] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4441.222394] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 4449.908451] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 18:37:10 (1778539030)
[ 4450.482270] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 4451.110523] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 18:37:11 (1778539031)
[ 4496.158145] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 18:37:56 (1778539076)
[ 4498.550614] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 18:37:58 (1778539078)
[ 4507.577721] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 18:38:07 (1778539087)
[ 4511.827060] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 18:38:11 (1778539091)
[ 4515.378565] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 18:38:15 (1778539095)
[ 4568.230517] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 4577.534367] Lustre: DEBUG MARKER: sanity-quota test_33: @@@@@@ FAIL: Used space (1482) for user 60000 isn't 0.
[ 4600.427637] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 4610.777094] Lustre: DEBUG MARKER: sanity-quota test_34: @@@@@@ FAIL: Used space (1482) for user 60000 isn't 0.