[ 3995.441020] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 3997.614855] Lustre: DEBUG MARKER: Files for user (quota_usr):
[ 3999.253981] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:1482).
[ 3999.991372] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 4001.492537] LustreError: 113369:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 4001.967921] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4002.712896] Lustre: DEBUG MARKER: Fail mds for 0 seconds
[ 4003.593867] Lustre: Failing over lustre-MDT0000
[ 4003.962826] Lustre: server umount lustre-MDT0000 complete
[ 4004.325653] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 4004.333623] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4004.342727] LustreError: Skipped 3 previous similar messages
[ 4007.731086] LustreError: 11-0: MGC192.168.204.115@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107
[ 4007.733852] LustreError: 166-1: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4007.744685] Lustre: Evicted from MGS (at 192.168.204.115@tcp) after server handle changed from 0x993efef251710d21 to 0x993efef2517d0d72
[ 4007.924164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4007.960603] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4010.055674] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4012.580849] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 4013.064076] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted.
[ 4013.092518] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1284 to 0x0:1313
[ 4013.092564] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1108 to 0x0:1153
[ 4014.639805] LustreError: 15893:0:(qsd_reint.c:635:qqi_reint_delayed()) lustre-OST0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 4014.648628] LustreError: 15893:0:(qsd_reint.c:635:qqi_reint_delayed()) Skipped 7 previous similar messages
[ 4015.365251] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4016.082080] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4021.940570] Lustre: DEBUG MARKER: (dd_pid=95775, time=3, timeout=600)
[ 4064.519878] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 4066.857539] Lustre: DEBUG MARKER: Files for user (quota_usr):
[ 4068.802333] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:1482).
[ 4069.612338] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 4070.895151] LustreError: 116334:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 4071.313126] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4072.034271] Lustre: DEBUG MARKER: Fail mds for 0 seconds
[ 4072.925168] Lustre: Failing over lustre-MDT0000
[ 4073.367034] Lustre: server umount lustre-MDT0000 complete
[ 4073.593055] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 4073.600380] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4073.608180] LustreError: Skipped 9 previous similar messages
[ 4077.005129] LustreError: 166-1: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4077.012804] Lustre: Evicted from MGS (at 192.168.204.115@tcp) after server handle changed from 0x993efef2517d0d72 to 0x993efef2517d185b
[ 4077.160693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4077.203610] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4079.058116] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4079.147996] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 4082.177567] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted.
[ 4082.196958] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1315 to 0x0:1345
[ 4082.198202] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1108 to 0x0:1185
[ 4084.742832] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4085.480280] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4092.579804] Lustre: DEBUG MARKER: (dd_pid=98469, time=4, timeout=600)
[ 4160.606911] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 18:32:20 (1778538740)
[ 4174.902243] Lustre: DEBUG MARKER: Files for user (quota_usr):
[ 4176.569666] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:1484).
[ 4178.543858] Lustre: DEBUG MARKER: Files for user (quota_usr):
[ 4180.241388] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:1484).
[ 4214.099729] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 18:33:14 (1778538794)
[ 4223.675943] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 4235.228847] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 4235.956263] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 4237.596593] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 4238.396233] Lustre: DEBUG MARKER: Set quota for 1 times
[ 4241.594400] Lustre: DEBUG MARKER: Set quota for 2 times
[ 4244.835857] Lustre: DEBUG MARKER: Set quota for 3 times
[ 4247.923919] Lustre: DEBUG MARKER: Set quota for 4 times
[ 4251.026177] Lustre: DEBUG MARKER: Set quota for 5 times
[ 4253.978715] Lustre: DEBUG MARKER: Set quota for 6 times
[ 4257.057487] Lustre: DEBUG MARKER: Set quota for 7 times
[ 4260.035860] Lustre: DEBUG MARKER: Set quota for 8 times
[ 4263.006858] Lustre: DEBUG MARKER: Set quota for 9 times
[ 4265.833069] Lustre: DEBUG MARKER: Set quota for 10 times
[ 4305.794806] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 18:34:45 (1778538885)
[ 4309.164976] LustreError: 125037:0:(qsd_reint.c:635:qqi_reint_delayed()) lustre-MDT0001: Delaying reintegration for qtype:1 until pending updates are flushed.
[ 4317.663911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4317.664946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4317.671922] Lustre: Skipped 11 previous similar messages
[ 4317.677535] Lustre: Skipped 1 previous similar message
[ 4317.681969] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 4317.687514] LustreError: Skipped 2 previous similar messages
[ 4321.356814] Lustre: server umount lustre-MDT0000 complete
[ 4322.784519] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4322.793215] LustreError: Skipped 12 previous similar messages
[ 4323.332954] LustreError: 6112:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778538904 with bad export cookie 11042543653061728347
[ 4323.334656] LustreError: 166-1: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4323.347336] LustreError: 6112:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 4331.999147] Lustre: 126046:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778538907/real 1778538907]  req@00000000653ad2a6 x1864928787437056/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778538913 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0'
[ 4332.056423] Lustre: server umount lustre-OST0000 complete
[ 4332.059893] Lustre: Skipped 1 previous similar message
[ 4340.191234] Lustre: 126252:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778538915/real 1778538915]  req@00000000554a7d64 x1864928787437568/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778538921 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0'
[ 4340.297413] Lustre: server umount lustre-OST0001 complete
[ 4347.958838] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local
[ 4352.530068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4354.506571] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4357.599826] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4357.606584] LustreError: Skipped 1 previous similar message
[ 4358.353726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 4360.358030] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4363.507535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 4365.255653] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4367.402371] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1347 to 0x0:1377
[ 4370.288052] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:3 to 0x2c0000400:33
[ 4370.298982] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1188 to 0x0:1217
[ 4370.659495] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4374.852438] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4376.451354] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 4394.066640] Lustre: server umount lustre-MDT0000 complete
[ 4394.464250] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 4396.266932] LustreError: 128937:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778538977 with bad export cookie 11042543653061733541
[ 4396.268723] LustreError: 166-1: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4396.275697] LustreError: 128937:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 4404.704019] Lustre: 131549:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778538979/real 1778538979]  req@000000007c288341 x1864928787470528/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778538985 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0'
[ 4411.871330] Lustre: 131754:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778538987/real 1778538987]  req@00000000a5baa423 x1864928787471040/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778538993 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0'
[ 4418.287290] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local
[ 4421.643056] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4421.650335] LustreError: Skipped 8 previous similar messages
[ 4421.698754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4421.701637] Lustre: Skipped 1 previous similar message
[ 4423.342244] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4427.629143] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4432.151370] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4435.891824] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:3 to 0x2c0000400:65
[ 4436.477776] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4436.899713] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1188 to 0x0:1249
[ 4436.900608] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1347 to 0x0:1409
[ 4439.947043] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4441.222394] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 4449.908451] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 18:37:10 (1778539030)
[ 4450.482270] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 4451.110523] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 18:37:11 (1778539031)
[ 4496.158145] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 18:37:56 (1778539076)
[ 4498.550614] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 18:37:58 (1778539078)
[ 4507.577721] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 18:38:07 (1778539087)
[ 4511.827060] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 18:38:11 (1778539091)
[ 4515.378565] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 18:38:15 (1778539095)
[ 4568.230517] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 4577.534367] Lustre: DEBUG MARKER: sanity-quota test_33: @@@@@@ FAIL: Used space (1482) for user 60000 isn't 0.
[ 4600.427637] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 4610.777094] Lustre: DEBUG MARKER: sanity-quota test_34: @@@@@@ FAIL: Used space (1482) for user 60000 isn't 0.
[ 4634.742193] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 18:40:14 (1778539214)
[ 4674.491286] Lustre: DEBUG MARKER: Restart...
[ 4677.087840] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 4677.091993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4677.099242] Lustre: Skipped 2 previous similar messages
[ 4682.253301] Lustre: server umount lustre-MDT0000 complete
[ 4682.256488] Lustre: Skipped 3 previous similar messages
[ 4682.722787] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4682.730226] LustreError: Skipped 4 previous similar messages
[ 4683.839204] LustreError: 132898:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778539265 with bad export cookie 11042543653061735802
[ 4683.841586] LustreError: 166-1: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4683.846037] LustreError: 132898:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 4690.911151] Lustre: 144930:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778539266/real 1778539266]  req@00000000f27f871b x1864928787558272/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778539272 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0'
[ 4704.209563] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local
[ 4707.854916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4707.858330] Lustre: Skipped 3 previous similar messages
[ 4709.561558] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4714.468487] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4719.398832] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4723.175773] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1251 to 0x0:1281
[ 4723.179359] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1412 to 0x0:1441
[ 4723.180839] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:3 to 0x2c0000400:97
[ 4723.868050] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4727.651965] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4729.210671] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 4773.058613] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 18:42:33 (1778539353)
[ 4782.491760] Lustre: DEBUG MARKER: sanity-quota test_37: @@@@@@ FAIL: Used space (1482) for user 60000 isn't 0.
[ 4805.950234] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 18:43:06 (1778539386)
[ 4815.276513] Lustre: DEBUG MARKER: sanity-quota test_38: @@@@@@ FAIL: Used space (1482) for user 60000 isn't 0.
[ 4838.667126] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 18:43:38 (1778539418)
[ 4850.866429] Lustre: server umount lustre-MDT0000 complete
[ 4850.868956] Lustre: Skipped 3 previous similar messages
[ 4851.168852] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4851.174499] Lustre: Skipped 11 previous similar messages
[ 4851.177864] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 4852.464344] LustreError: 146279:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778539433 with bad export cookie 11042543653061739610
[ 4852.466542] LustreError: 166-1: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4852.469774] LustreError: 146279:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 4860.383190] Lustre: 154056:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778539435/real 1778539435]  req@0000000024171283 x1864928787605760/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778539441 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0'
[ 4860.393352] Lustre: 154056:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 4874.900908] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local
[ 4878.686334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4878.689992] Lustre: Skipped 3 previous similar messages
[ 4880.557743] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4885.782102] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4890.828735] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4894.059975] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1412 to 0x0:1473
[ 4895.356122] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:3 to 0x2c0000400:129
[ 4895.360241] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1283 to 0x0:1313
[ 4896.020111] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4899.671658] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4901.183708] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 4939.877598] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 18:45:20 (1778539520)
[ 4979.454827] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 18:45:59 (1778539559)
[ 5021.243865] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 18:46:41 (1778539601)
[ 5060.865812] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 18:47:21 (1778539641)
[ 5101.912292] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 18:48:02 (1778539682)
[ 5112.869660] LustreError: 163968:0:(qsd_reint.c:635:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 5112.874542] LustreError: 163968:0:(qsd_reint.c:635:qqi_reint_delayed()) Skipped 7 previous similar messages
[ 5152.606372] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 18:48:52 (1778539732)
[ 5235.588431] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 18:50:15 (1778539815)
[ 5287.343713] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 18:51:07 (1778539867)
[ 5333.085666] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 18:51:53 (1778539913)
[ 5350.434174] Lustre: DEBUG MARKER: rename directory return 255
[ 5384.074439] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 18:52:44 (1778539964)
[ 5412.553963] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 18:53:12 (1778539992)
[ 5444.392601] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 18:53:44 (1778540024)
[ 5506.205811] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 18:54:46 (1778540086)
[ 5535.679482] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 18:55:15 (1778540115)
[ 5572.742305] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 18:55:53 (1778540153)
[ 5573.252240] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[ 5573.779052] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 18:55:54 (1778540154)
[ 5585.211530] Lustre: DEBUG MARKER: Files for group (quota_usr):
[ 5586.407119] Lustre: DEBUG MARKER: Block quota isn't 0 (g:quota_usr:1482).
[ 5587.850603] Lustre: DEBUG MARKER: Files for group (quota_usr):
[ 5589.027967] Lustre: DEBUG MARKER: File quota isn't 0 (g:quota_usr:988).
[ 5589.534374] Lustre: DEBUG MARKER: sanity-quota test_60: @@@@@@ FAIL: create many files failed
[ 5611.265498] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[ 5611.767220] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 18:56:32 (1778540192)
[ 5640.478663] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[ 5642.082466] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non dir/files should succeed ========================================================== 18:57:01 (1778540221)
[ 5687.885731] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[ 5688.638439] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 18:57:48 (1778540268)
[ 5729.087153] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 18:58:29 (1778540309)
[ 5729.890115] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[ 5730.708938] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 18:58:30 (1778540310)
[ 5788.509739] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 18:59:28 (1778540368)
[ 5802.275627] LustreError: 186027:0:(qsd_reint.c:635:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed.
[ 5802.280468] LustreError: 186027:0:(qsd_reint.c:635:qqi_reint_delayed()) Skipped 7 previous similar messages
[ 5811.460249] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[ 5812.130437] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[ 5907.527820] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 19:01:27 (1778540487)
[ 5953.906440] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 19:02:13 (1778540533)
[ 5975.862420] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 19:02:35 (1778540555)
[ 5976.627465] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[ 5977.648808] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 19:02:37 (1778540557)
[ 5978.444697] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[ 5979.335084] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 19:02:39 (1778540559)
[ 5991.613390] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[ 6000.216985] Lustre: DEBUG MARKER: sanity-quota test_72: @@@@@@ FAIL: Used space(1482) for user quota_usr isn't 0.