[ 6135.086708] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 6143.047399] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 6148.161624] LustreError: 124618:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 6149.595619] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6152.080115] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 6155.323289] Lustre: Failing over lustre-MDT0000
[ 6155.972244] Lustre: server umount lustre-MDT0000 complete
[ 6171.424308] Lustre: 3307:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789055109/real 1789055109]  req@ffff8c3b78d75880 x1875953915665024/t0(0) o400->MGC192.168.204.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789055125 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6171.486068] Lustre: 3307:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 6171.516872] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6172.575584] Lustre: 3310:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789055109/real 1789055109]  req@ffff8c3b78d77800 x1875953915665152/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789055125 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6172.600202] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6177.503104] Lustre: 3309:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789055115/real 1789055115]  req@ffff8c3a4c5d6680 x1875953915665664/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789055131 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6177.557830] Lustre: 3309:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 6181.802378] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xd272684c704cc5e4
[ 6181.824520] Lustre: MGC192.168.204.122@tcp: Connection restored to 0@lo (at 0@lo)
[ 6182.525719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6182.664202] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 6183.520783] Lustre: 3308:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789055120/real 1789055120]  req@ffff8c3b80e60700 x1875953915665920/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789055136 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6183.564669] Lustre: 3308:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 6183.823977] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 6183.951229] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 6184.086921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:508 to 0x240000400:545)
[ 6184.092164] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:505 to 0x280000400:545)
[ 6188.440790] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6200.843512] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6203.721930] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6210.429163] Lustre: DEBUG MARKER: (dd_pid=117415, time=0, timeout=600)
[ 6256.754708] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 6263.122829] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 6266.193930] LustreError: 127476:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 6267.108592] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6268.683306] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 6270.505315] Lustre: Failing over lustre-MDT0000
[ 6270.837965] Lustre: server umount lustre-MDT0000 complete
[ 6286.817964] Lustre: 3310:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789055224/real 1789055224]  req@ffff8c3b79201880 x1875953915703936/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1789055240 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 6286.858042] Lustre: 3310:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 6286.873771] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6286.905340] Lustre: Skipped 1 previous similar message
[ 6290.064990] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6290.869409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6290.976253] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 6291.855287] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 6291.876501] Lustre: Skipped 2 previous similar messages
[ 6295.198390] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6298.339506] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 6298.522054] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 6298.622232] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:505 to 0x280000400:577)
[ 6298.627952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:547 to 0x240000400:577)
[ 6303.135150] Lustre: 3309:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789055240/real 1789055240]  req@ffff8c3b4810ce00 x1875953915705600/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789055256 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6303.169859] Lustre: 3309:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 6306.409720] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6308.129324] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6315.044441] Lustre: DEBUG MARKER: (dd_pid=119826, time=1, timeout=600)
[ 6383.652493] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 11:48:55 (1789055335)
[ 6404.854804] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6406.518731] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6412.964709] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6414.474789] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6448.544388] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 11:50:00 (1789055400)
[ 6476.530254] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 6493.263707] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 6495.022696] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 6496.878606] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 6498.908334] Lustre: DEBUG MARKER: Set quota for 1 times
[ 6502.372389] Lustre: DEBUG MARKER: Set quota for 2 times
[ 6505.699090] Lustre: DEBUG MARKER: Set quota for 3 times
[ 6509.374279] Lustre: DEBUG MARKER: Set quota for 4 times
[ 6513.634726] Lustre: DEBUG MARKER: Set quota for 5 times
[ 6517.409871] Lustre: DEBUG MARKER: Set quota for 6 times
[ 6521.455791] Lustre: DEBUG MARKER: Set quota for 7 times
[ 6526.188818] Lustre: DEBUG MARKER: Set quota for 8 times
[ 6564.772721] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 11:51:56 (1789055516)
[ 6578.658772] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6578.678873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6578.694548] Lustre: Skipped 2 previous similar messages
[ 6578.717109] Lustre: Skipped 1 previous similar message
[ 6579.966927] Lustre: server umount lustre-MDT0000 complete
[ 6585.577145] LustreError: 5830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789055539 with bad export cookie 15164297572820766470
[ 6585.585384] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6585.591199] LustreError: 5830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 6585.793391] Lustre: server umount lustre-OST0000 complete
[ 6591.744984] Lustre: server umount lustre-OST0001 complete
[ 6608.112285] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local
[ 6618.424309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6622.962418] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6626.069307] Lustre: 137293:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6632.366262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6639.057521] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6642.600455] LustreError: 137668:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6642.653142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:579 to 0x240000400:609)
[ 6647.961263] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6653.462381] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:580 to 0x280000400:609)
[ 6656.181069] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6664.365141] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6668.430775] Lustre: 139228:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6684.128101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6684.143558] Lustre: Skipped 1 previous similar message
[ 6688.654697] Lustre: server umount lustre-MDT0000 complete
[ 6693.717318] LustreError: 136840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789055647 with bad export cookie 15164297572820776214
[ 6693.725715] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6693.728763] LustreError: 136840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 6693.832121] Lustre: server umount lustre-OST0000 complete
[ 6698.543279] Lustre: server umount lustre-OST0001 complete
[ 6717.095813] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local
[ 6727.363337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6732.134749] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6735.152842] Lustre: 141880:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6747.867985] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6752.162423] LustreError: 142245:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6752.238889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:579 to 0x240000400:641)
[ 6757.516076] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6757.523198] Lustre: Skipped 1 previous similar message
[ 6762.500980] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:580 to 0x280000400:641)
[ 6767.272881] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6777.654491] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6781.811325] Lustre: 143797:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6792.439863] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 11:55:43 (1789055743)
[ 6793.880856] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6795.800131] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 11:55:47 (1789055747)
[ 6853.416774] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 11:56:44 (1789055804)
[ 6906.552689] Lustre: DEBUG MARKER: Write...
[ 6909.149191] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6962.538919] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 11:58:34 (1789055914)
[ 6969.393259] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 11:58:40 (1789055920)
[ 6979.825874] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 11:58:51 (1789055931)
[ 6990.824398] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 11:59:02 (1789055942)
[ 7001.603854] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 11:59:12 (1789055952)
[ 7072.524529] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 7249.223772] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 7461.964792] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 12:06:53 (1789056413)
[ 7526.423150] Lustre: DEBUG MARKER: Restart...
[ 7535.585160] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7535.594744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7535.612463] Lustre: Skipped 3 previous similar messages
[ 7535.633951] Lustre: Skipped 1 previous similar message
[ 7537.579740] Lustre: server umount lustre-MDT0000 complete
[ 7543.041978] LustreError: 144554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789056496 with bad export cookie 15164297572820777747
[ 7543.053507] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7543.067936] LustreError: 144554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 7543.193664] Lustre: server umount lustre-OST0000 complete
[ 7547.813971] Lustre: server umount lustre-OST0001 complete
[ 7565.528934] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local
[ 7577.177773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7582.346484] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7586.085554] Lustre: 162532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7594.571201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7601.640449] LustreError: 162902:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7601.666243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:650 to 0x240000400:673)
[ 7601.921144] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7606.754666] LustreError: 163244:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7611.875297] LustreError: 162902:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7612.415674] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7617.546358] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:650 to 0x280000400:673)
[ 7620.399714] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7630.360056] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7634.745449] Lustre: 164462:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7715.761404] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 12:11:06 (1789056666)
[ 7797.148255] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 12:12:28 (1789056748)
[ 9517.406576] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 12:41:09 (1789058469)
[ 9536.541160] Lustre: server umount lustre-MDT0000 complete
[ 9542.458334] LustreError: 162085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789058496 with bad export cookie 15164297572820784243
[ 9542.489793] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9552.997271] Lustre: server umount lustre-OST0000 complete
[ 9553.890173] Lustre: 3309:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789058491/real 1789058491]  req@ffff8c3a7fba4700 x1875953919059200/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789058507 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 9553.966233] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9558.179327] Lustre: server umount lustre-OST0001 complete
[ 9575.030537] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local
[ 9587.462448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9593.373474] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9598.282873] Lustre: 172715:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9607.663366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9611.680982] LustreError: 173083:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9611.746756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5675 to 0x240000400:5697)
[ 9615.888801] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9616.875594] LustreError: 173083:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9621.987727] LustreError: 173082:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9625.532625] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9630.726381] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5675 to 0x280000400:5697)
[ 9633.921472] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9642.380424] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9646.545960] Lustre: 174630:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9687.986854] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 12:43:59 (1789058639)
[ 9741.327430] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 12:44:52 (1789058692)
[ 9792.718641] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 12:45:43 (1789058743)
[ 9795.722816] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9798.906275] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 12:45:49 (1789058749)
[ 9801.733665] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9804.655891] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 12:45:55 (1789058755)
[ 9902.078717] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 12:47:33 (1789058853)
[ 9959.705596] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 12:48:31 (1789058911)
[10055.671925] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 12:50:06 (1789059006)
[10068.127798] Lustre: *** cfs_fail_loc=a09, val=0***
[10068.136441] Lustre: Skipped 2 previous similar messages
[10070.153994] Lustre: *** cfs_fail_loc=a09, val=0***
[10070.158144] Lustre: Skipped 67 previous similar messages
[10074.211449] Lustre: *** cfs_fail_loc=a09, val=0***
[10074.218078] Lustre: Skipped 155 previous similar messages
[10082.251967] Lustre: *** cfs_fail_loc=a09, val=0***
[10082.263088] Lustre: Skipped 271 previous similar messages
[10098.266358] Lustre: *** cfs_fail_loc=a09, val=0***
[10098.271503] Lustre: Skipped 563 previous similar messages
[10130.278715] Lustre: *** cfs_fail_loc=a09, val=0***
[10130.303612] Lustre: Skipped 1247 previous similar messages
[10194.285447] Lustre: *** cfs_fail_loc=a09, val=0***
[10194.287592] Lustre: Skipped 2703 previous similar messages
[10600.683540] Lustre: *** cfs_fail_loc=a08, val=0***
[10600.692085] Lustre: Skipped 2993 previous similar messages
[10600.692858] Lustre: *** cfs_fail_loc=a08, val=0***
[10600.704898] Lustre: Skipped 4 previous similar messages
[10891.515142] Lustre: *** cfs_fail_loc=a08, val=0***
[10891.533269] Lustre: Skipped 9 previous similar messages
[10891.559585] Lustre: *** cfs_fail_loc=a08, val=0***
[10891.566693] Lustre: Skipped 12 previous similar messages
[10896.620019] Lustre: *** cfs_fail_loc=a08, val=0***
[10896.629227] Lustre: Skipped 43 previous similar messages
[10901.779114] Lustre: *** cfs_fail_loc=a08, val=0***
[10901.794487] Lustre: Skipped 199 previous similar messages
[10906.879237] Lustre: *** cfs_fail_loc=a08, val=0***
[10906.891328] Lustre: Skipped 250 previous similar messages
[10932.412047] LustreError: 185859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789059886 with bad export cookie 15164297572822562005
[10932.433643] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10932.470705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10936.291747] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10936.291979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10936.312795] Lustre: Skipped 1 previous similar message
[10939.387762] Lustre: server umount lustre-MDT0000 complete
[10944.795659] LustreError: 185944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789059898 with bad export cookie 15164297572822561326
[10944.807730] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10945.038337] Lustre: server umount lustre-OST0000 complete
[10949.638387] Lustre: server umount lustre-OST0001 complete
[10957.072225] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid
[10966.784329] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local
[10977.154926]  vdc: vdc1 vdc9
[10988.431391]  vde: vde1 vde9
[11000.185224]  vdf: vdf1 vdf9
[11021.409672] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local
[11033.741756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[11034.095752] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[11034.213564] Lustre: lustre-MDT0000: new disk, initializing
[11034.745593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[11034.782509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[11040.607899] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11046.293281] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[11054.167608] Lustre: lustre-OST0000: new disk, initializing
[11054.172439] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[11054.176535] Lustre: Skipped 1 previous similar message
[11054.319165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[11055.846504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[11055.853574] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[11056.130738] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[11062.456975] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11074.054779] Lustre: lustre-OST0001: new disk, initializing
[11074.058767] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[11074.222797] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[11075.953375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[11075.970871] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[11076.269901] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[11081.458212] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11091.329613] Lustre: DEBUG MARKER: Using TIMEOUT=20
[11096.400253] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[11137.474595] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 13:08:09 (1789060089)
[11145.959448] Lustre: DEBUG MARKER: == sanity-quota test 49c: lfs quota long options don't consume an extra argument ========================================================== 13:08:17 (1789060097)
[11154.061395] Lustre: DEBUG MARKER: == sanity-quota test 49d: lfs quota -d and --delimiter both work ========================================================== 13:08:25 (1789060105)
[11162.510056] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 13:08:33 (1789060113)
[11219.813412] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 13:09:31 (1789060171)
[11289.100489] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 13:10:40 (1789060240)
[11319.888777] Lustre: DEBUG MARKER: rename directory return 255
[11370.777791] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 13:12:02 (1789060322)
[11406.857537] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 13:12:38 (1789060358)
[11453.087158] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 13:13:24 (1789060404)
[11586.289450] LustreError: 200128:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11631.503672] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 13:16:23 (1789060583)
[11670.276492] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 13:17:02 (1789060622)
[11716.877041] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 13:17:48 (1789060668)
[11910.942882] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 13:21:02 (1789060862)
[11913.375539] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[11914.880199] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 13:21:06 (1789060866)
[11978.081440] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[11979.679872] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 13:22:11 (1789060931)
[12010.856845] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[12012.571958] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 13:22:44 (1789060964)
[12064.840988] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[12067.234034] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 13:23:38 (1789061018)
[12117.509312] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 13:24:29 (1789061069)
[12118.955987] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[12120.978949] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 13:24:32 (1789061072)
[12148.405493] LustreError: 213401:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[12194.642759] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 13:25:45 (1789061145)
[12226.068545] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[12228.061990] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[12426.892981] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 13:29:38 (1789061378)
[12481.766428] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 13:30:33 (1789061433)
[12517.201593] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 13:31:08 (1789061468)
[12519.195627] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[12521.071797] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 13:31:12 (1789061472)
[12522.690205] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[12524.619607] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 13:31:16 (1789061476)
[12541.697811] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[12559.773865] Lustre: DEBUG MARKER: Write...
[12562.827493] Lustre: DEBUG MARKER: Write out of block quota ...
[12621.516758] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 13:32:52 (1789061572)
[12651.006361] Lustre: DEBUG MARKER: set to use default quota
[12652.871217] Lustre: DEBUG MARKER: set default quota
[12654.690451] Lustre: DEBUG MARKER: get default quota
[12660.867987] Lustre: DEBUG MARKER: Test not out of quota
[12665.436832] Lustre: DEBUG MARKER: Test out of quota
[12679.282754] Lustre: DEBUG MARKER: Increase default quota
[12715.748900] Lustre: DEBUG MARKER: Set quota to override default quota
[12715.818804] LustreError: 189467:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1789666469 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12731.930042] Lustre: DEBUG MARKER: Set to use default quota again
[12754.900740] Lustre: DEBUG MARKER: Cleanup
[12832.862508] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 13:36:24 (1789061784)
[12858.650216] Lustre: DEBUG MARKER: set default quota for qpool1
[12860.844982] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[12919.351855] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 13:37:50 (1789061870)
[13009.965324] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 13:39:21 (1789061961)
[13096.284263] Lustre: DEBUG MARKER: Write...
[13102.316320] Lustre: DEBUG MARKER: Write out of block quota ...
[13234.586894] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 13:43:06 (1789062186)
[13285.665582] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 13:43:57 (1789062237)
[13293.561450] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 13:44:05 (1789062245)
[13296.183854] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A fallocate not supported
[13298.232955] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 13:44:09 (1789062249)
[13300.913533] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a fallocate not supported
[13302.937579] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 13:44:14 (1789062254)
[13323.507375] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 13:44:35 (1789062275)
[13325.307667] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[13326.932442] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 13:44:38 (1789062278)
[13345.323755] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[13352.826455] LustreError: 238687:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[13357.970962] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.22@tcp (stopping)
[13359.584697] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[13359.593171] Lustre: Skipped 1 previous similar message
[13359.597342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[13359.603256] Lustre: Skipped 1 previous similar message
[13362.839132] LustreError: 238687:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[13363.065138] Lustre: server umount lustre-MDT0000 complete
[13373.906435] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[13374.794773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[13375.063705] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[13375.070522] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[13379.843470] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[13379.849883] Lustre: Skipped 1 previous similar message
[13380.567500] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[13424.672563] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 13:46:16 (1789062376)
[13458.000495] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 13:46:49 (1789062409)
[13492.808711] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 13:47:24 (1789062444)
[13546.005259] Lustre: *** cfs_fail_loc=a08, val=0***
[13546.009171] Lustre: Skipped 1135 previous similar messages
[13546.034189] Lustre: *** cfs_fail_loc=a08, val=0***
[13546.037570] Lustre: Skipped 650 previous similar messages
[13679.152926] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 13:50:30 (1789062630)
[13766.899168] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 13:51:58 (1789062718)
[13846.656107] LustreError: 239278:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1789667600 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[14013.641949] LustreError: 239277:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1789667767 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[14024.525576] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed