[ 3729.110810] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 3731.847557] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 3733.234223] LustreError: 127957:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 3733.655533] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3734.458844] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 3735.296576] Lustre: Failing over lustre-MDT0000
[ 3735.577762] Lustre: server umount lustre-MDT0000 complete
[ 3748.942338] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3749.103625] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3749.112059] Lustre: Skipped 1 previous similar message
[ 3749.178387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3749.229095] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3750.175387] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 3750.237530] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 3750.261059] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:884 to 0x280000400:929)
[ 3750.261204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:887 to 0x240000400:929)
[ 3750.987183] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3752.224334] Lustre: 3327:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780327509/real 1780327509]  req@ffff949745e3ce00 x1866804888146048/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1780327525 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 3754.466293] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 3754.812843] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3755.553112] Lustre: 3326:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780327513/real 1780327513]  req@ffff94987618f100 x1866804888146432/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780327529 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3755.603067] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3760.353058] Lustre: 3325:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780327518/real 1780327518]  req@ffff949848ce4000 x1866804888146816/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780327534 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3760.365179] Lustre: 3325:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 3764.920825] Lustre: DEBUG MARKER: (dd_pid=119154, time=6, timeout=600)
[ 3796.776646] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 3799.753162] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 3801.223597] LustreError: 130684:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 3801.675289] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3802.446676] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 3803.311084] Lustre: Failing over lustre-MDT0000
[ 3803.560056] Lustre: server umount lustre-MDT0000 complete
[ 3817.030061] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3817.221753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3817.255749] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3818.955331] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3820.256145] Lustre: 3325:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780327577/real 1780327577]  req@ffff94985146f480 x1866804888176512/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1780327593 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 3820.268228] Lustre: 3325:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 3820.751547] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 3820.817918] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 3820.837084] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:884 to 0x280000400:961)
[ 3820.837084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:931 to 0x240000400:961)
[ 3822.647158] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3823.454498] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3831.654991] Lustre: DEBUG MARKER: (dd_pid=121568, time=5, timeout=600)
[ 3831.776256] Lustre: 3327:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780327589/real 1780327589]  req@ffff9498485bf800 x1866804888177664/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780327605 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3831.791852] Lustre: 3327:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 3875.171736] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 11:27:28 (1780327648)
[ 3887.713830] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 3888.430758] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 3890.705423] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 3891.302269] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 3914.556743] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 11:28:07 (1780327687)
[ 3925.514359] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 3934.710385] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 3935.338235] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 3935.965640] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 3936.672145] Lustre: DEBUG MARKER: Set quota for 1 times
[ 3938.507843] Lustre: DEBUG MARKER: Set quota for 2 times
[ 3940.362315] Lustre: DEBUG MARKER: Set quota for 3 times
[ 3942.161032] Lustre: DEBUG MARKER: Set quota for 4 times
[ 3944.022273] Lustre: DEBUG MARKER: Set quota for 5 times
[ 3945.823877] Lustre: DEBUG MARKER: Set quota for 6 times
[ 3947.647751] Lustre: DEBUG MARKER: Set quota for 7 times
[ 3949.484075] Lustre: DEBUG MARKER: Set quota for 8 times
[ 3951.216406] Lustre: DEBUG MARKER: Set quota for 9 times
[ 3953.013562] Lustre: DEBUG MARKER: Set quota for 10 times
[ 3954.797093] Lustre: DEBUG MARKER: Set quota for 11 times
[ 3956.658370] Lustre: DEBUG MARKER: Set quota for 12 times
[ 3958.615906] Lustre: DEBUG MARKER: Set quota for 13 times
[ 3960.550446] Lustre: DEBUG MARKER: Set quota for 14 times
[ 3962.485421] Lustre: DEBUG MARKER: Set quota for 15 times
[ 3964.388460] Lustre: DEBUG MARKER: Set quota for 16 times
[ 3991.743843] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 11:29:24 (1780327764)
[ 3999.177825] Lustre: server umount lustre-MDT0000 complete
[ 4000.835246] LustreError: 25817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780327774 with bad export cookie 3407745367603198332
[ 4000.840248] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4011.000598] Lustre: server umount lustre-OST0000 complete
[ 4021.529828] Lustre: server umount lustre-OST0001 complete
[ 4026.794819] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local
[ 4030.042146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4031.379907] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4032.494903] Lustre: 140964:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 4034.653549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 4036.971214] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4039.777761] LustreError: 141282:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4039.782845] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:963 to 0x240000400:993)
[ 4039.881484] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 4041.985555] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4045.292550] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:964 to 0x280000400:993)
[ 4045.333722] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4046.638297] Lustre: 142739:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 4055.521614] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4055.530619] Lustre: Skipped 2 previous similar messages
[ 4055.533791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4055.719909] Lustre: server umount lustre-MDT0000 complete
[ 4058.909257] LustreError: 140532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780327832 with bad export cookie 3407745367603208916
[ 4058.914054] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4058.916130] LustreError: 140532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 4058.980281] Lustre: server umount lustre-OST0000 complete
[ 4063.650899] Lustre: server umount lustre-OST0001 complete
[ 4087.180807] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local
[ 4105.337621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4117.245886] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4122.772191] Lustre: 145417:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 4132.512744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 4139.781181] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4149.650764] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:963 to 0x240000400:1025)
[ 4149.653026] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:964 to 0x280000400:1025)
[ 4153.531518] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4162.966427] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4167.948522] Lustre: 147191:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 4177.884875] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 11:32:30 (1780327950)
[ 4179.538931] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 4181.180150] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 11:32:33 (1780327953)
[ 4231.228186] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 11:33:23 (1780328003)
[ 4277.091056] Lustre: DEBUG MARKER: Write...
[ 4279.517876] Lustre: DEBUG MARKER: Write out of block quota ...
[ 4332.222161] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 11:35:04 (1780328104)
[ 4338.203230] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 11:35:10 (1780328110)
[ 4347.175820] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 11:35:19 (1780328119)
[ 4355.765217] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 11:35:27 (1780328127)
[ 4363.214816] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 11:35:35 (1780328135)
[ 4428.715869] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 4572.836162] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 4753.432629] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 11:42:05 (1780328525)
[ 4812.792559] Lustre: DEBUG MARKER: Restart...
[ 4816.847749] Lustre: server umount lustre-MDT0000 complete
[ 4820.366720] LustreError: 147193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780328594 with bad export cookie 3407745367603210449
[ 4820.375789] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4830.761555] Lustre: server umount lustre-OST0000 complete
[ 4844.174729] Lustre: server umount lustre-OST0001 complete
[ 4854.945111] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local
[ 4861.234312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4861.239345] Lustre: Skipped 1 previous similar message
[ 4864.175894] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4866.605923] Lustre: 165961:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 4870.625478] LustreError: 166279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4870.819547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 4876.258376] LustreError: 166280:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4876.273536] LustreError: 166280:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message
[ 4876.307529] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1034 to 0x240000400:1057)
[ 4876.584604] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4881.382039] LustreError: 166280:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4882.737819] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 4887.429912] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4888.056864] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1034 to 0x280000400:1057)
[ 4893.362688] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4896.079467] Lustre: 167751:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 4952.991741] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 11:45:24 (1780328724)
[ 5024.146911] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 11:46:36 (1780328796)
[ 5747.253796] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 11:58:40 (1780329520)
[ 5757.265147] Lustre: server umount lustre-MDT0000 complete
[ 5758.572941] LustreError: 165531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329532 with bad export cookie 3407745367603216945
[ 5758.579710] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5769.213347] Lustre: server umount lustre-OST0000 complete
[ 5780.856015] Lustre: server umount lustre-OST0001 complete
[ 5786.035326] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local
[ 5789.499972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5790.943017] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5791.940815] Lustre: 175812:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5793.671144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5795.618809] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5798.002236] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5799.973895] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5800.043066] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6059 to 0x280000400:6081)
[ 5800.045303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6059 to 0x240000400:6081)
[ 5803.198906] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5804.457060] Lustre: 177586:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5831.361246] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 12:00:04 (1780329604)
[ 5860.842863] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 12:00:33 (1780329633)
[ 5891.585928] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 12:01:04 (1780329664)
[ 5892.106872] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 5892.686571] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 12:01:05 (1780329665)
[ 5893.217366] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 5893.851975] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 12:01:06 (1780329666)
[ 6008.927257] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 12:03:00 (1780329780)
[ 6064.973213] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 12:03:57 (1780329837)
[ 6152.794945] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 12:05:24 (1780329924)
[ 6164.232986] Lustre: *** cfs_fail_loc=a09, val=0***
[ 6164.234822] Lustre: Skipped 1 previous similar message
[ 6166.249953] Lustre: *** cfs_fail_loc=a09, val=0***
[ 6166.254554] Lustre: Skipped 101 previous similar messages
[ 6170.298134] Lustre: *** cfs_fail_loc=a09, val=0***
[ 6170.301866] Lustre: Skipped 201 previous similar messages
[ 6178.308940] Lustre: *** cfs_fail_loc=a09, val=0***
[ 6178.311516] Lustre: Skipped 387 previous similar messages
[ 6194.324417] Lustre: *** cfs_fail_loc=a09, val=0***
[ 6194.329105] Lustre: Skipped 849 previous similar messages
[ 6226.385242] Lustre: *** cfs_fail_loc=a09, val=0***
[ 6226.396379] Lustre: Skipped 1597 previous similar messages
[ 6290.426748] Lustre: *** cfs_fail_loc=a09, val=0***
[ 6290.437818] Lustre: Skipped 2607 previous similar messages
[ 6650.336811] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6650.337509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6650.342252] Lustre: Skipped 2 previous similar messages
[ 6650.344514] Lustre: Skipped 1 previous similar message
[ 6652.862364] Lustre: server umount lustre-MDT0000 complete
[ 6654.311969] LustreError: 175385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780330427 with bad export cookie 3407745367604971460
[ 6654.315975] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6654.453804] Lustre: server umount lustre-OST0000 complete
[ 6656.001274] Lustre: server umount lustre-OST0001 complete
[ 6658.251377] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid
[ 6660.853962] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local
[ 6664.251022]  vdc: vdc1 vdc9
[ 6667.364110]  vde: vde1 vde9
[ 6670.787333]  vdf: vdf1 vdf9
[ 6676.171611] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local
[ 6678.976760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 6679.049660] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 6679.082505] Lustre: lustre-MDT0000: new disk, initializing
[ 6679.171695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6679.189452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 6680.446279] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6682.711955] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 6684.396162] Lustre: lustre-OST0000: new disk, initializing
[ 6684.397652] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 6684.399457] Lustre: Skipped 1 previous similar message
[ 6684.418754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6686.289814] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6686.345836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[ 6686.349742] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[ 6686.387751] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[ 6689.926809] Lustre: lustre-OST0001: new disk, initializing
[ 6689.928154] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 6689.951971] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6691.399731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[ 6691.402976] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[ 6691.432814] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[ 6691.993679] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6696.587640] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6697.847973] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 6718.418899] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 12:14:51 (1780330491)
[ 6720.856753] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 12:14:53 (1780330493)
[ 6751.802200] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 12:15:24 (1780330524)
[ 6794.455241] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 12:16:06 (1780330566)
[ 6821.981547] Lustre: DEBUG MARKER: rename directory return 255
[ 6870.672412] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 12:17:21 (1780330641)
[ 6916.007341] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 12:18:07 (1780330687)
[ 6959.457616] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 12:18:51 (1780330731)
[ 7060.207197] LustreError: 192675:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 7097.518950] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 12:21:09 (1780330869)
[ 7139.703469] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 12:21:51 (1780330911)
[ 7185.026851] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 12:22:37 (1780330957)
[ 7352.906682] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 12:25:25 (1780331125)
[ 7354.096773] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[ 7355.527681] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 12:25:27 (1780331127)
[ 7408.804553] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[ 7409.982665] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 12:26:22 (1780331182)
[ 7438.425310] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[ 7440.104884] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 12:26:52 (1780331212)
[ 7491.139688] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[ 7492.910422] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 12:27:45 (1780331265)
[ 7538.055815] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 12:28:29 (1780331309)
[ 7539.460780] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[ 7540.795748] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 12:28:33 (1780331313)
[ 7565.481534] LustreError: 215904:0:(mgs_handler.c:1140:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[ 7604.680682] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 12:29:37 (1780331377)
[ 7628.108814] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[ 7629.504713] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[ 7788.020376] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 12:32:40 (1780331560)
[ 7833.150526] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 12:33:25 (1780331605)
[ 7859.674776] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 12:33:52 (1780331632)
[ 7861.059986] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[ 7862.535493] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 12:33:54 (1780331634)
[ 7863.906589] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[ 7865.358747] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 12:33:57 (1780331637)
[ 7881.766496] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[ 7898.489213] Lustre: DEBUG MARKER: Write...
[ 7900.491594] Lustre: DEBUG MARKER: Write out of block quota ...
[ 7949.278941] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 12:35:21 (1780331721)
[ 7973.524533] Lustre: DEBUG MARKER: set to use default quota
[ 7974.837315] Lustre: DEBUG MARKER: set default quota
[ 7976.203989] Lustre: DEBUG MARKER: get default quota
[ 7982.084384] Lustre: DEBUG MARKER: Test not out of quota
[ 7986.552129] Lustre: DEBUG MARKER: Test out of quota
[ 7999.371719] Lustre: DEBUG MARKER: Increase default quota
[ 8028.364990] Lustre: DEBUG MARKER: Set quota to override default quota
[ 8028.405656] LustreError: 195234:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1780936602 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 8039.520817] Lustre: DEBUG MARKER: Set to use default quota again
[ 8061.075000] Lustre: DEBUG MARKER: Cleanup
[ 8131.478975] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 12:38:23 (1780331903)
[ 8154.963103] Lustre: DEBUG MARKER: set default quota for qpool1
[ 8156.366608] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[ 8200.936151] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 12:39:33 (1780331973)
[ 8271.526459] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 12:40:44 (1780332044)
[ 8337.130780] Lustre: DEBUG MARKER: Write...
[ 8340.234669] Lustre: DEBUG MARKER: Write out of block quota ...
[ 8453.612787] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 12:43:45 (1780332225)
[ 8497.511838] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 12:44:29 (1780332269)
[ 8504.818849] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 12:44:37 (1780332277)
[ 8506.311924] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate
[ 8507.775764] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 12:44:40 (1780332280)
[ 8509.239593] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate
[ 8510.663790] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 12:44:43 (1780332283)
[ 8527.358413] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 12:44:59 (1780332299)
[ 8528.564533] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[ 8530.177288] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 12:45:02 (1780332302)
[ 8544.616900] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[ 8549.386218] LustreError: 241161:0:(qmt_pool.c:1399:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[ 8553.954754] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8553.972100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 8559.075727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 8559.080358] Lustre: Skipped 3 previous similar messages
[ 8559.426045] LustreError: 241161:0:(qmt_pool.c:1399:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[ 8559.688867] Lustre: server umount lustre-MDT0000 complete
[ 8565.885278] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8566.292772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8566.411813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[ 8566.411854] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[ 8569.762914] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8583.593272] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 8583.607564] Lustre: Skipped 3 previous similar messages
[ 8616.312534] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 12:46:28 (1780332388)
[ 8648.655193] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 12:47:01 (1780332421)
[ 8679.643663] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 12:47:31 (1780332451)
[ 8729.008428] Lustre: *** cfs_fail_loc=a08, val=0***
[ 8729.017916] Lustre: Skipped 2251 previous similar messages
[ 8729.031641] Lustre: *** cfs_fail_loc=a08, val=0***
[ 8844.901863] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 12:50:16 (1780332616)
[ 8927.618823] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 12:51:39 (1780332699)
[ 9004.034031] LustreError: 241695:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1780937577 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 9157.301133] LustreError: 241697:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1780937730 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 9166.107626] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed