[ 6101.234230] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 6108.976516] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 6114.195907] LustreError: 121085:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 6115.587684] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6118.021616] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 6120.750863] Lustre: Failing over lustre-MDT0000
[ 6121.412496] Lustre: server umount lustre-MDT0000 complete
[ 6137.315673] Lustre: 3322:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787581675/real 1787581675]  req@ffff94618b065180 x1874408892592512/t0(0) o400->MGC192.168.201.129@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787581691 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6137.316737] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6137.340888] Lustre: 3322:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 6137.340979] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6137.394163] Lustre: Skipped 1 previous similar message
[ 6143.462204] Lustre: 3323:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787581681/real 1787581681]  req@ffff9461897b5180 x1874408892592896/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787581697 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6143.515533] Lustre: 3323:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 6147.569579] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbf4fb7546bafd075
[ 6147.572279] Lustre: MGC192.168.201.129@tcp: Connection restored to 0@lo (at 0@lo)
[ 6148.511813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6148.719806] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 6149.153411] Lustre: 3320:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787581686/real 1787581686]  req@ffff94618b066300 x1874408892593280/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787581702 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6149.222387] Lustre: 3320:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 6151.930844] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 6152.212969] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 6152.273549] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:568 to 0x280000400:609)
[ 6152.274538] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:571 to 0x240000400:609)
[ 6155.183915] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6162.663441] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 6166.336850] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6167.974772] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6178.927538] Lustre: DEBUG MARKER: (dd_pid=114552, time=5, timeout=600)
[ 6231.580971] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 6238.244141] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 6242.241474] LustreError: 123959:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 6243.035571] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6245.431600] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 6248.150730] Lustre: Failing over lustre-MDT0000
[ 6248.549664] Lustre: server umount lustre-MDT0000 complete
[ 6265.187310] Lustre: 3323:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787581803/real 1787581803]  req@ffff94626c454000 x1874408892634880/t0(0) o400->MGC192.168.201.129@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787581819 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6265.218146] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6265.233127] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6275.839995] LustreError: 124615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6276.276212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6276.366616] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 6277.024632] Lustre: 3320:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787581814/real 1787581814]  req@ffff946150ddf100 x1874408892635776/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787581830 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6277.055454] Lustre: 3320:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 6277.074739] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 6277.338349] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 6277.411068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:611 to 0x240000400:641)
[ 6277.411808] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:568 to 0x280000400:641)
[ 6281.286823] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6290.387561] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6290.469082] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 6290.474784] Lustre: Skipped 1 previous similar message
[ 6292.075580] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6311.678963] Lustre: DEBUG MARKER: (dd_pid=116961, time=12, timeout=600)
[ 6384.630607] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 10:32:16 (1787581936)
[ 6408.311174] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6410.886557] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6419.514534] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6422.287634] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6465.460339] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 10:33:36 (1787582016)
[ 6498.845788] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 6516.416448] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 6518.361723] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 6519.902493] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 6522.307598] Lustre: DEBUG MARKER: Set quota for 1 times
[ 6526.689401] Lustre: DEBUG MARKER: Set quota for 2 times
[ 6531.218658] Lustre: DEBUG MARKER: Set quota for 3 times
[ 6535.176266] Lustre: DEBUG MARKER: Set quota for 4 times
[ 6540.008915] Lustre: DEBUG MARKER: Set quota for 5 times
[ 6544.131934] Lustre: DEBUG MARKER: Set quota for 6 times
[ 6548.405537] Lustre: DEBUG MARKER: Set quota for 7 times
[ 6589.438253] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 10:35:41 (1787582141)
[ 6602.722785] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6602.723638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6602.750402] Lustre: Skipped 2 previous similar messages
[ 6602.765561] Lustre: Skipped 1 previous similar message
[ 6603.249880] Lustre: server umount lustre-MDT0000 complete
[ 6607.631836] LustreError: 26127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582161 with bad export cookie 13785438557616200066
[ 6607.645268] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6607.727798] Lustre: server umount lustre-OST0000 complete
[ 6613.690320] Lustre: server umount lustre-OST0001 complete
[ 6631.124104] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local
[ 6640.969468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6645.276698] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6648.108571] Lustre: 133722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6654.610026] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6662.030261] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6665.826994] LustreError: 134630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6665.847280] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:644 to 0x240000400:673)
[ 6671.339528] LustreError: 134630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6671.662697] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6677.007646] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:643 to 0x280000400:673)
[ 6678.637565] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6687.152538] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6690.934586] Lustre: 135652:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6708.434364] Lustre: server umount lustre-MDT0000 complete
[ 6714.151447] LustreError: 135901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582268 with bad export cookie 13785438557616206828
[ 6714.163081] LustreError: 135901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 6714.173561] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6724.752818] Lustre: server umount lustre-OST0000 complete
[ 6728.672149] Lustre: 3322:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787582266/real 1787582266]  req@ffff946243e39880 x1874408892803072/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787582282 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6728.711948] Lustre: 3322:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 6728.732850] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6730.037440] Lustre: server umount lustre-OST0001 complete
[ 6746.879771] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local
[ 6758.177326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6762.578468] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6765.841724] Lustre: 138311:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6778.524381] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6782.562242] LustreError: 138680:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6782.617469] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:644 to 0x240000400:705)
[ 6786.700497] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6786.704197] Lustre: Skipped 1 previous similar message
[ 6791.668978] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:643 to 0x280000400:705)
[ 6793.038292] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6801.248431] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6805.198615] Lustre: 140222:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6816.516606] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 10:39:28 (1787582368)
[ 6818.592574] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6820.550479] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 10:39:32 (1787582372)
[ 6875.724605] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 10:40:27 (1787582427)
[ 6927.295938] Lustre: DEBUG MARKER: Write...
[ 6930.488621] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6990.520833] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 10:42:22 (1787582542)
[ 6999.082344] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 10:42:30 (1787582550)
[ 7012.373595] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 10:42:44 (1787582564)
[ 7022.508035] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 10:42:54 (1787582574)
[ 7032.261757] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 10:43:03 (1787582583)
[ 7103.501571] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 7281.896246] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 7485.338738] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 10:50:37 (1787583037)
[ 7550.115814] Lustre: DEBUG MARKER: Restart...
[ 7559.650353] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7559.664708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7561.406959] Lustre: server umount lustre-MDT0000 complete
[ 7565.422670] LustreError: 137864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583119 with bad export cookie 13785438557616208361
[ 7565.430846] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7565.545068] Lustre: server umount lustre-OST0000 complete
[ 7570.307635] Lustre: server umount lustre-OST0001 complete
[ 7585.481917] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local
[ 7596.216794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7601.031712] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7604.056563] Lustre: 158959:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7610.276657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7617.515721] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7620.564393] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:715 to 0x240000400:737)
[ 7621.541779] LustreError: 159328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7626.724465] LustreError: 159326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7630.059036] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7633.088520] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:713 to 0x280000400:737)
[ 7637.823215] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7647.974595] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7652.621265] Lustre: 160878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7731.433503] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 10:54:43 (1787583283)
[ 7813.714279] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 10:56:05 (1787583365)
[ 9711.623596] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 11:27:43 (1787585263)
[ 9731.092215] Lustre: server umount lustre-MDT0000 complete
[ 9737.145256] LustreError: 158514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585291 with bad export cookie 13785438557616214857
[ 9737.158616] LustreError: 158514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 9737.165825] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9747.701178] Lustre: server umount lustre-OST0000 complete
[ 9748.965046] Lustre: 3322:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787585286/real 1787585286]  req@ffff946254229c00 x1874408896070272/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787585302 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 9748.987371] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9749.008639] Lustre: Skipped 1 previous similar message
[ 9753.665211] Lustre: server umount lustre-OST0001 complete
[ 9774.185820] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local
[ 9787.909909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9793.524132] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9797.513897] Lustre: 169219:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9805.373216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9812.342561] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9812.518047] LustreError: 169580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9812.565691] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5739 to 0x240000400:5761)
[ 9817.572463] LustreError: 169953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9821.363161] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9826.327070] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5739 to 0x280000400:5761)
[ 9828.819726] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9837.041088] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9840.964864] Lustre: 171135:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9880.017536] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 11:30:31 (1787585431)
[ 9927.764647] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 11:31:19 (1787585479)
[ 9983.158928] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 11:32:14 (1787585534)
[ 9985.324466] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9987.253180] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 11:32:18 (1787585538)
[ 9989.538878] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9991.468905] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 11:32:23 (1787585543)
[10090.247711] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 11:34:02 (1787585642)
[10147.665227] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 11:34:59 (1787585699)
[10247.163316] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 11:36:39 (1787585799)
[10259.279567] Lustre: *** cfs_fail_loc=a09, val=0***
[10259.286665] Lustre: Skipped 1 previous similar message
[10261.279752] Lustre: *** cfs_fail_loc=a09, val=0***
[10261.282322] Lustre: Skipped 92 previous similar messages
[10265.296227] Lustre: *** cfs_fail_loc=a09, val=0***
[10265.299576] Lustre: Skipped 226 previous similar messages
[10273.365437] Lustre: *** cfs_fail_loc=a09, val=0***
[10273.368559] Lustre: Skipped 431 previous similar messages
[10289.366713] Lustre: *** cfs_fail_loc=a09, val=0***
[10289.369795] Lustre: Skipped 797 previous similar messages
[10321.369423] Lustre: *** cfs_fail_loc=a09, val=0***
[10321.375644] Lustre: Skipped 996 previous similar messages
[10385.399806] Lustre: *** cfs_fail_loc=a09, val=0***
[10385.405656] Lustre: Skipped 2368 previous similar messages
[10785.583556] Lustre: *** cfs_fail_loc=a08, val=0***
[10785.598021] Lustre: Skipped 3085 previous similar messages
[10785.620152] Lustre: *** cfs_fail_loc=a08, val=0***
[10785.622107] Lustre: Skipped 1 previous similar message
[11082.535810] Lustre: *** cfs_fail_loc=a08, val=0***
[11082.541402] Lustre: Skipped 13 previous similar messages
[11082.550943] Lustre: *** cfs_fail_loc=a08, val=0***
[11082.558140] Lustre: Skipped 10 previous similar messages
[11085.670514] Lustre: *** cfs_fail_loc=a08, val=0***
[11085.679548] Lustre: Skipped 91 previous similar messages
[11092.880724] Lustre: *** cfs_fail_loc=a08, val=0***
[11092.889142] Lustre: Skipped 112 previous similar messages
[11103.030869] Lustre: *** cfs_fail_loc=a08, val=0***
[11103.040028] Lustre: Skipped 576 previous similar messages
[11131.886285] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[11131.909740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[11131.912279] Lustre: Skipped 1 previous similar message
[11133.862239] Lustre: server umount lustre-MDT0000 complete
[11138.798243] LustreError: 168770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586692 with bad export cookie 13785438557617994460
[11138.819283] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[11139.063446] Lustre: server umount lustre-OST0000 complete
[11144.733898] Lustre: server umount lustre-OST0001 complete
[11154.041925] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid
[11165.702787] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local
[11178.409790]  vdc: vdc1 vdc9
[11190.026533]  vde: vde1 vde9
[11204.206396]  vdf: vdf1 vdf9
[11225.482246] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local
[11237.330734] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[11237.622240] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[11237.752959] Lustre: lustre-MDT0000: new disk, initializing
[11238.183843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[11238.237727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[11243.559281] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11248.798671] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[11256.669422] Lustre: lustre-OST0000: new disk, initializing
[11256.672754] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[11256.680722] Lustre: Skipped 1 previous similar message
[11256.781659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[11258.113923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[11258.123208] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[11258.280444] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[11264.854077] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11277.049132] Lustre: lustre-OST0001: new disk, initializing
[11277.053119] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[11277.198401] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[11279.281920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[11279.298337] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[11279.473770] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[11284.637034] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11295.639799] Lustre: DEBUG MARKER: Using TIMEOUT=20
[11299.420709] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[11341.465925] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 11:54:52 (1787586892)
[11350.090939] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 11:55:01 (1787586901)
[11405.122523] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 11:55:56 (1787586956)
[11476.511103] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 11:57:08 (1787587028)
[11512.669507] Lustre: DEBUG MARKER: rename directory return 255
[11566.161604] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 11:58:37 (1787587117)
[11602.965958] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 11:59:14 (1787587154)
[11653.774356] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 12:00:04 (1787587204)
[11767.477727] LustreError: 186130:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11810.462856] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 12:02:41 (1787587361)
[11854.624235] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 12:03:26 (1787587406)
[11907.539743] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 12:04:18 (1787587458)
[12120.081394] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 12:07:51 (1787587671)
[12121.632618] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[12123.163672] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 12:07:55 (1787587675)
[12188.620583] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[12190.439272] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 12:09:02 (1787587742)
[12226.161796] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[12228.196298] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 12:09:39 (1787587779)
[12284.933223] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[12287.665878] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 12:10:38 (1787587838)
[12342.174502] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 12:11:32 (1787587892)
[12344.452454] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[12346.604913] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 12:11:38 (1787587898)
[12378.347942] LustreError: 209347:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[12426.129305] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 12:12:57 (1787587977)
[12454.850683] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[12456.718717] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[12650.923991] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 12:16:42 (1787588202)
[12711.057116] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 12:17:41 (1787588261)
[12750.398658] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 12:18:21 (1787588301)
[12752.532526] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[12754.806154] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 12:18:26 (1787588306)
[12756.281857] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[12757.718740] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 12:18:29 (1787588309)
[12776.198562] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[12794.194323] Lustre: DEBUG MARKER: Write...
[12797.029693] Lustre: DEBUG MARKER: Write out of block quota ...
[12860.150822] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 12:20:11 (1787588411)
[12891.307699] Lustre: DEBUG MARKER: set to use default quota
[12893.485613] Lustre: DEBUG MARKER: set default quota
[12895.486069] Lustre: DEBUG MARKER: get default quota
[12902.314237] Lustre: DEBUG MARKER: Test not out of quota
[12906.995258] Lustre: DEBUG MARKER: Test out of quota
[12919.806664] Lustre: DEBUG MARKER: Increase default quota
[12953.102884] Lustre: DEBUG MARKER: Set quota to override default quota
[12953.176432] LustreError: 186128:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1788193307 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12966.610987] Lustre: DEBUG MARKER: Set to use default quota again
[12990.722402] Lustre: DEBUG MARKER: Cleanup
[13065.369652] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 12:23:37 (1787588617)
[13090.633673] Lustre: DEBUG MARKER: set default quota for qpool1
[13092.434630] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[13146.434542] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 12:24:58 (1787588698)
[13237.260070] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 12:26:28 (1787588788)
[13316.615332] Lustre: DEBUG MARKER: Write...
[13319.612954] Lustre: DEBUG MARKER: Write out of block quota ...
[13437.412746] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 12:29:48 (1787588988)
[13489.578742] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 12:30:41 (1787589041)
[13497.728633] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 12:30:49 (1787589049)
[13499.607620] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate
[13501.353376] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 12:30:53 (1787589053)
[13503.196586] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate
[13505.186783] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 12:30:56 (1787589056)
[13528.199611] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 12:31:20 (1787589080)
[13529.972557] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[13531.873650] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 12:31:23 (1787589083)
[13551.170437] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[13558.868807] LustreError: 234636:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[13562.137106] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.29@tcp (stopping)
[13562.848956] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[13562.851818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[13562.879915] Lustre: Skipped 2 previous similar messages
[13562.896102] Lustre: Skipped 1 previous similar message
[13567.224665] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.29@tcp (stopping)
[13568.928462] LustreError: 234636:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[13569.281440] Lustre: server umount lustre-MDT0000 complete
[13579.467742] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[13580.148928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[13580.439758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[13580.441880] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[13584.736745] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[13593.574132] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[13593.600806] Lustre: Skipped 1 previous similar message
[13624.942063] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 12:32:56 (1787589176)
[13662.603762] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 12:33:34 (1787589214)
[13701.019751] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 12:34:12 (1787589252)
[13756.802378] Lustre: *** cfs_fail_loc=a08, val=0***
[13756.808306] Lustre: Skipped 1388 previous similar messages
[13756.840461] Lustre: *** cfs_fail_loc=a08, val=0***
[13756.845643] Lustre: Skipped 605 previous similar messages
[13887.438384] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 12:37:19 (1787589439)
[13977.081783] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 12:38:48 (1787589528)
[14060.270611] LustreError: 235220:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1788194414 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[14229.290576] LustreError: 236314:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1788194583 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[14238.053803] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed