[ 5859.837100] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5866.942477] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5870.554813] LustreError: 124985:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5871.509420] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5873.533575] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5875.590261] Lustre: Failing over lustre-MDT0000
[ 5875.966843] Lustre: server umount lustre-MDT0000 complete
[ 5893.791756] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5894.097586] Lustre: 3286:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789454114/real 1789454114]  req@ffff8e1a568b2d80 x1876372541896960/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789454130 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5894.157214] Lustre: 3286:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5894.171026] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5894.421993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5894.492879] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5895.000502] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5895.076716] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5895.146770] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:628 to 0x240000400:673)
[ 5895.147664] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:625 to 0x280000400:641)
[ 5898.278390] Lustre: 3287:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789454119/real 1789454119]  req@ffff8e1a568b1180 x1876372541897344/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789454135 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5898.306792] Lustre: 3287:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5899.378947] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5899.750721] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5903.327303] Lustre: 3287:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789454124/real 1789454124]  req@ffff8e1b7d3c6300 x1876372541897728/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789454140 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5903.352346] Lustre: 3287:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5908.959411] Lustre: 3287:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789454129/real 1789454129]  req@ffff8e1b7d3c7480 x1876372541898240/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789454145 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5909.387583] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5911.294935] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5916.573403] Lustre: DEBUG MARKER: (dd_pid=117693, time=0, timeout=600)
[ 5968.650163] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5975.690499] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5979.324503] LustreError: 127896:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5980.384804] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5982.094504] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5984.267253] Lustre: Failing over lustre-MDT0000
[ 5984.700740] Lustre: server umount lustre-MDT0000 complete
[ 6001.135282] Lustre: 3286:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789454222/real 1789454222]  req@ffff8e1b42f99c00 x1876372541932800/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1789454238 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 6001.173979] Lustre: 3286:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 6002.149250] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6012.934377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6012.991186] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 6013.773252] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 6013.967525] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 6014.032483] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:675 to 0x240000400:705)
[ 6014.035148] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:625 to 0x280000400:673)
[ 6017.633297] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6017.955568] Lustre: 3286:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789454238/real 1789454238]  req@ffff8e1b42f9b800 x1876372541934464/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789454254 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6017.990102] Lustre: 3286:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 6026.159432] LustreError: 3284:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-lwp-OST0001: namespace resource [0x200000006:0x20000:0x0].0x0 (ffff8e1a458bc300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6026.195760] LustreError: 3284:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 5 previous similar messages
[ 6028.022504] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6029.777856] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6043.411672] Lustre: DEBUG MARKER: (dd_pid=120124, time=8, timeout=600)
[ 6107.058794] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 02:39:02 (1789454342)
[ 6130.405446] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6132.497970] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6139.368968] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6141.661929] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6177.205038] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 02:40:13 (1789454413)
[ 6205.942194] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 6222.878331] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 6224.693806] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 6226.728314] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 6229.364267] Lustre: DEBUG MARKER: Set quota for 1 times
[ 6233.178676] Lustre: DEBUG MARKER: Set quota for 2 times
[ 6237.533365] Lustre: DEBUG MARKER: Set quota for 3 times
[ 6241.824491] Lustre: DEBUG MARKER: Set quota for 4 times
[ 6245.890831] Lustre: DEBUG MARKER: Set quota for 5 times
[ 6250.658927] Lustre: DEBUG MARKER: Set quota for 6 times
[ 6255.341249] Lustre: DEBUG MARKER: Set quota for 7 times
[ 6301.686352] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 02:42:17 (1789454537)
[ 6318.052889] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6318.068744] Lustre: Skipped 3 previous similar messages
[ 6318.077572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6318.089481] Lustre: Skipped 1 previous similar message
[ 6319.705403] Lustre: server umount lustre-MDT0000 complete
[ 6324.336566] LustreError: 5849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789454561 with bad export cookie 2177751915343386147
[ 6324.337623] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6324.348841] LustreError: 5849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 6324.442736] Lustre: server umount lustre-OST0000 complete
[ 6329.032176] Lustre: server umount lustre-OST0001 complete
[ 6345.671610] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local
[ 6355.592525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6360.289689] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6363.592292] Lustre: 137477:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6369.674603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6376.054853] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6380.901583] LustreError: 137854:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6380.973759] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:708 to 0x240000400:737)
[ 6384.988744] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6390.285367] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:675 to 0x280000400:705)
[ 6391.964478] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6399.700414] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6403.717415] Lustre: 139406:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6421.759599] Lustre: server umount lustre-MDT0000 complete
[ 6425.739632] LustreError: 137026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789454662 with bad export cookie 2177751915343395163
[ 6425.768522] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6435.439498] Lustre: server umount lustre-OST0000 complete
[ 6450.010489] Lustre: server umount lustre-OST0001 complete
[ 6467.270268] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local
[ 6477.320772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6482.899900] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6486.273944] Lustre: 142062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6493.676838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6501.506834] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6502.845082] LustreError: 142424:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6502.937091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:708 to 0x240000400:769)
[ 6513.123478] LustreError: 142424:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6513.138552] LustreError: 142424:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message
[ 6520.851979] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:675 to 0x280000400:737)
[ 6524.531045] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6533.479823] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6537.655498] Lustre: 143968:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6548.882099] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 02:46:24 (1789454784)
[ 6550.788864] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6552.697228] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 02:46:28 (1789454788)
[ 6611.162385] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 02:47:26 (1789454846)
[ 6676.181543] Lustre: DEBUG MARKER: Write...
[ 6678.619284] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6736.939393] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 02:49:32 (1789454972)
[ 6744.235753] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 02:49:40 (1789454980)
[ 6754.583434] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 02:49:50 (1789454990)
[ 6763.737872] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 02:49:59 (1789454999)
[ 6771.494144] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 02:50:07 (1789455007)
[ 6837.465774] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6992.140807] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 7183.756690] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 02:56:59 (1789455419)
[ 7259.891897] Lustre: DEBUG MARKER: Restart...
[ 7266.283134] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7266.289895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7266.313083] Lustre: Skipped 2 previous similar messages
[ 7267.041693] Lustre: server umount lustre-MDT0000 complete
[ 7271.849517] LustreError: 141616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789455508 with bad export cookie 2177751915343396703
[ 7271.857887] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7271.868798] LustreError: 141616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 7271.979810] Lustre: server umount lustre-OST0000 complete
[ 7277.069734] Lustre: server umount lustre-OST0001 complete
[ 7295.287544] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local
[ 7307.413680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7307.418909] Lustre: Skipped 1 previous similar message
[ 7312.843750] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7315.909832] Lustre: 162396:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7322.583299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7328.473202] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7332.772488] LustreError: 162764:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7332.828113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:779 to 0x240000400:801)
[ 7337.633365] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7343.103257] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:745 to 0x280000400:769)
[ 7344.438857] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7352.956956] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7357.383961] Lustre: 164309:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7438.338284] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 03:01:14 (1789455674)
[ 7521.047628] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 03:02:37 (1789455757)
[ 9566.969949] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 03:36:43 (1789457803)
[ 9581.136905] Lustre: server umount lustre-MDT0000 complete
[ 9585.006510] LustreError: 161950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789457821 with bad export cookie 2177751915343403199
[ 9585.018586] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9594.422741] Lustre: server umount lustre-OST0000 complete
[ 9597.414555] Lustre: 3287:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789457818/real 1789457818]  req@ffff8e1a8718ca80 x1876372545394560/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789457834 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 9597.435075] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9598.255882] Lustre: server umount lustre-OST0001 complete
[ 9611.063456] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local
[ 9620.155079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9624.877114] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9628.326162] Lustre: 172963:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9635.171254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9643.912633] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9644.329907] LustreError: 173332:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9644.402594] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5804 to 0x240000400:5825)
[ 9649.632094] LustreError: 173332:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9654.157650] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9659.401413] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5770 to 0x280000400:5793)
[ 9662.299549] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9669.679929] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9672.923198] Lustre: 174887:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9714.158767] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 03:39:10 (1789457950)
[ 9760.052849] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 03:39:55 (1789457995)
[ 9805.931713] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 03:40:41 (1789458041)
[ 9808.039435] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9810.005917] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 03:40:45 (1789458045)
[ 9811.868255] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9813.591845] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 03:40:49 (1789458049)
[ 9908.789669] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 03:42:24 (1789458144)
[ 9954.964900] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 03:43:11 (1789458191)
[10043.932821] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 03:44:39 (1789458279)
[10054.892342] Lustre: *** cfs_fail_loc=a09, val=0***
[10054.894942] Lustre: Skipped 3 previous similar messages
[10056.899805] Lustre: *** cfs_fail_loc=a09, val=0***
[10056.903482] Lustre: Skipped 99 previous similar messages
[10060.903563] Lustre: *** cfs_fail_loc=a09, val=0***
[10060.906360] Lustre: Skipped 219 previous similar messages
[10068.904741] Lustre: *** cfs_fail_loc=a09, val=0***
[10068.908193] Lustre: Skipped 400 previous similar messages
[10084.950716] Lustre: *** cfs_fail_loc=a09, val=0***
[10084.961376] Lustre: Skipped 834 previous similar messages
[10116.960056] Lustre: *** cfs_fail_loc=a09, val=0***
[10116.963096] Lustre: Skipped 1521 previous similar messages
[10180.968850] Lustre: *** cfs_fail_loc=a09, val=0***
[10180.970444] Lustre: Skipped 2911 previous similar messages
[10593.510463] Lustre: *** cfs_fail_loc=a08, val=0***
[10593.515640] Lustre: Skipped 2014 previous similar messages
[10593.516925] Lustre: *** cfs_fail_loc=a08, val=0***
[10593.528578] Lustre: Skipped 2 previous similar messages
[10829.795847] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10829.805978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10829.809890] Lustre: Skipped 2 previous similar messages
[10834.485402] Lustre: server umount lustre-MDT0000 complete
[10839.337737] LustreError: 172517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789459076 with bad export cookie 2177751915345186288
[10839.361878] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10839.883679] Lustre: server umount lustre-OST0000 complete
[10845.112749] Lustre: server umount lustre-OST0001 complete
[10854.160723] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid
[10863.116499] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local
[10874.595212]  vdc: vdc1 vdc9
[10885.818897]  vde: vde1 vde9
[10896.514400]  vdf: vdf1 vdf9
[10915.079703] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local
[10926.460406] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[10926.763086] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[10926.891368] Lustre: lustre-MDT0000: new disk, initializing
[10927.439270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10927.574240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[10927.579664] Lustre: Skipped 1 previous similar message
[10933.700094] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10939.614851] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[10947.379960] Lustre: lustre-OST0000: new disk, initializing
[10947.390332] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[10947.402841] Lustre: Skipped 1 previous similar message
[10947.562294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10949.244296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[10949.255956] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[10949.260716] Lustre: Skipped 1 previous similar message
[10949.437759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[10955.144563] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10967.346547] Lustre: lustre-OST0001: new disk, initializing
[10967.350259] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[10967.439557] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10973.730253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[10973.742764] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[10973.884492] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[10975.144237] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10986.277295] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10990.430651] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[11030.608591] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 04:01:06 (1789459266)
[11037.587729] Lustre: DEBUG MARKER: == sanity-quota test 49c: lfs quota long options don't consume an extra argument ========================================================== 04:01:13 (1789459273)
[11046.037499] Lustre: DEBUG MARKER: == sanity-quota test 49d: lfs quota -d and --delimiter both work ========================================================== 04:01:21 (1789459281)
[11052.717289] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 04:01:28 (1789459288)
[11103.126636] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 04:02:18 (1789459338)
[11168.902660] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 04:03:24 (1789459404)
[11199.609765] Lustre: DEBUG MARKER: rename directory return 255
[11252.300398] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 04:04:47 (1789459487)
[11289.665933] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 04:05:25 (1789459525)
[11339.923421] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 04:06:15 (1789459575)
[11463.500884] LustreError: 200385:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11504.328989] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 04:08:59 (1789459739)
[11547.467570] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 04:09:43 (1789459783)
[11595.385977] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 04:10:30 (1789459830)
[11792.945630] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 04:13:49 (1789460029)
[11793.936111] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[11795.555195] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 04:13:51 (1789460031)
[11857.076502] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[11859.090446] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 04:14:54 (1789460094)
[11890.873528] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[11892.584836] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 04:15:28 (1789460128)
[11947.612843] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[11949.298908] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 04:16:25 (1789460185)
[12000.772781] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 04:17:16 (1789460236)
[12002.419930] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[12004.285772] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 04:17:20 (1789460240)
[12036.288730] LustreError: 213418:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[12084.833594] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 04:18:40 (1789460320)
[12113.651549] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[12116.123137] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[12327.235336] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 04:22:42 (1789460562)
[12383.510770] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 04:23:39 (1789460619)
[12417.192298] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 04:24:13 (1789460653)
[12418.652796] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[12420.424471] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 04:24:16 (1789460656)
[12421.766655] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[12423.394589] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 04:24:19 (1789460659)
[12439.726421] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[12457.159872] Lustre: DEBUG MARKER: Write...
[12459.719218] Lustre: DEBUG MARKER: Write out of block quota ...
[12517.355618] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 04:25:53 (1789460753)
[12544.500545] Lustre: DEBUG MARKER: set to use default quota
[12546.050726] Lustre: DEBUG MARKER: set default quota
[12547.505894] Lustre: DEBUG MARKER: get default quota
[12552.863916] Lustre: DEBUG MARKER: Test not out of quota
[12556.349928] Lustre: DEBUG MARKER: Test out of quota
[12565.579884] Lustre: DEBUG MARKER: Increase default quota
[12597.152096] Lustre: DEBUG MARKER: Set quota to override default quota
[12597.236327] LustreError: 189754:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1790065634 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12610.859475] Lustre: DEBUG MARKER: Set to use default quota again
[12633.554901] Lustre: DEBUG MARKER: Cleanup
[12702.647831] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 04:28:58 (1789460938)
[12730.441127] Lustre: DEBUG MARKER: set default quota for qpool1
[12732.310412] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[12781.910945] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 04:30:17 (1789461017)
[12865.394449] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 04:31:40 (1789461100)
[12943.141635] Lustre: DEBUG MARKER: Write...
[12946.671811] Lustre: DEBUG MARKER: Write out of block quota ...
[13063.978586] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 04:35:00 (1789461300)
[13107.953248] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 04:35:43 (1789461343)
[13114.433819] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 04:35:50 (1789461350)
[13116.632941] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A fallocate not supported
[13118.361916] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 04:35:54 (1789461354)
[13120.469441] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a fallocate not supported
[13122.022143] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 04:35:58 (1789461358)
[13139.213516] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 04:36:15 (1789461375)
[13140.488555] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[13141.810334] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 04:36:18 (1789461378)
[13156.417680] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[13162.447575] LustreError: 238548:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[13164.886357] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.22@tcp (stopping)
[13165.025824] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[13165.052333] Lustre: Skipped 2 previous similar messages
[13169.990510] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.22@tcp (stopping)
[13170.005803] Lustre: Skipped 2 previous similar messages
[13172.527163] LustreError: 238548:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[13172.800162] Lustre: server umount lustre-MDT0000 complete
[13180.995939] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[13181.443382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[13181.594229] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[13181.614976] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[13186.290093] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[13195.750621] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[13195.758821] Lustre: Skipped 3 previous similar messages
[13220.612701] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 04:37:36 (1789461456)
[13249.068814] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 04:38:05 (1789461485)
[13276.275281] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 04:38:32 (1789461512)
[13319.417532] Lustre: *** cfs_fail_loc=a08, val=0***
[13319.422470] Lustre: Skipped 6 previous similar messages
[13319.432608] Lustre: *** cfs_fail_loc=a08, val=0***
[13319.437285] Lustre: Skipped 10 previous similar messages
[13420.560826] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 04:40:56 (1789461656)
[13497.260464] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 04:42:13 (1789461733)
[13571.058376] LustreError: 247719:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1790066607 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13725.172643] LustreError: 239952:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1790066762 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13733.984235] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed