[ 6131.259323] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 6137.627687] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 6141.305965] LustreError: 120686:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 6142.057091] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6143.720537] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 6146.221824] Lustre: Failing over lustre-MDT0000
[ 6146.547308] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6146.568116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6146.570967] Lustre: Skipped 1 previous similar message
[ 6146.642659] Lustre: server umount lustre-MDT0000 complete
[ 6164.959228] Lustre: 3309:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789137381/real 1789137381]  req@ffff9fde20780780 x1876040196846592/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1789137397 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 6164.976068] Lustre: 3309:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 6164.998127] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6165.793119] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6166.372827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6166.426648] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 6167.263624] Lustre: 3308:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789137384/real 1789137384]  req@ffff9fdd470b1a40 x1876040196846976/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789137400 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6167.332548] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 6171.010692] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6172.383347] Lustre: 3309:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789137389/real 1789137389]  req@ffff9fde037cc000 x1876040196847232/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789137405 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6175.341985] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 6175.507437] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 6175.572353] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:610 to 0x280000400:641)
[ 6175.576468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:614 to 0x240000400:641)
[ 6181.333359] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6182.771488] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6193.367675] Lustre: DEBUG MARKER: (dd_pid=114205, time=5, timeout=600)
[ 6237.515470] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 6244.071872] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 6247.723511] LustreError: 123544:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 6248.764023] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6250.776479] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 6252.978875] Lustre: Failing over lustre-MDT0000
[ 6253.468592] Lustre: server umount lustre-MDT0000 complete
[ 6269.923817] Lustre: 3309:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789137486/real 1789137486]  req@ffff9fde03799680 x1876040196886144/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1789137502 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 6269.938250] Lustre: 3309:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 6269.952276] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6273.121199] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6273.131034] Lustre: MGS: Not available for connect from 0@lo (not set up)
[ 6274.071638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6274.230698] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 6275.434806] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 6275.652133] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 6275.689701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:643 to 0x240000400:673)
[ 6275.693525] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:610 to 0x280000400:673)
[ 6279.241921] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6280.098397] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 6280.117876] Lustre: Skipped 1 previous similar message
[ 6281.211083] LustreError: 3306:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-lwp-OST0001: namespace resource [0x200000006:0x20000:0x0].0x0 (ffff9fdd442c9f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6281.227063] LustreError: 3306:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 5 previous similar messages
[ 6288.759294] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6290.915983] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6301.520791] Lustre: DEBUG MARKER: (dd_pid=116603, time=4, timeout=600)
[ 6363.322357] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 10:39:53 (1789137593)
[ 6391.915397] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6394.244445] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6402.196803] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6404.310996] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6440.672728] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 10:41:10 (1789137670)
[ 6468.771883] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 6487.562540] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 6489.503333] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 6491.191275] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 6493.625977] Lustre: DEBUG MARKER: Set quota for 1 times
[ 6498.046459] Lustre: DEBUG MARKER: Set quota for 2 times
[ 6502.852271] Lustre: DEBUG MARKER: Set quota for 3 times
[ 6507.994032] Lustre: DEBUG MARKER: Set quota for 4 times
[ 6512.357780] Lustre: DEBUG MARKER: Set quota for 5 times
[ 6517.003324] Lustre: DEBUG MARKER: Set quota for 6 times
[ 6521.529487] Lustre: DEBUG MARKER: Set quota for 7 times
[ 6559.352473] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 10:43:10 (1789137790)
[ 6573.959198] Lustre: server umount lustre-MDT0000 complete
[ 6578.266028] LustreError: 15975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137811 with bad export cookie 29673868322049792
[ 6578.277724] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6589.039741] Lustre: server umount lustre-OST0000 complete
[ 6603.926486] Lustre: server umount lustre-OST0001 complete
[ 6619.596554] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local
[ 6632.363107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6637.838547] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6641.029872] Lustre: 133168:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6649.758331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6656.812315] LustreError: 133854:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6656.836832] LustreError: 133854:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message
[ 6656.840892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:675 to 0x240000400:705)
[ 6657.842988] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6662.121609] LustreError: 133854:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6667.265956] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6672.396262] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:676 to 0x280000400:705)
[ 6674.172141] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6683.481424] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6688.659628] Lustre: 135089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6709.917937] Lustre: server umount lustre-MDT0000 complete
[ 6715.301301] LustreError: 132711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137948 with bad export cookie 29673868322056575
[ 6715.318584] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6725.855097] Lustre: server umount lustre-OST0000 complete
[ 6729.633302] Lustre: 3310:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789137946/real 1789137946]  req@ffff9fde13c292c0 x1876040197052544/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789137962 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6729.658761] Lustre: 3310:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
[ 6729.670093] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6729.688439] Lustre: Skipped 1 previous similar message
[ 6731.158870] Lustre: server umount lustre-OST0001 complete
[ 6748.280086] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local
[ 6760.770176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6767.241776] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6770.995592] Lustre: 137741:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6779.127394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6786.145223] LustreError: 138107:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6786.195695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:675 to 0x240000400:737)
[ 6786.824240] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6791.654971] LustreError: 138107:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6802.434444] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:676 to 0x280000400:737)
[ 6804.487995] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6814.022603] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6818.222852] Lustre: 139653:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6828.892459] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 10:47:39 (1789138059)
[ 6830.941957] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6832.683129] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 10:47:43 (1789138063)
[ 6893.414695] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 10:48:43 (1789138123)
[ 6945.968940] Lustre: DEBUG MARKER: Write...
[ 6948.439190] Lustre: DEBUG MARKER: Write out of block quota ...
[ 7012.776692] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 10:50:43 (1789138243)
[ 7020.338932] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 10:50:50 (1789138250)
[ 7031.534538] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 10:51:01 (1789138261)
[ 7041.286442] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 10:51:11 (1789138271)
[ 7051.020434] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 10:51:21 (1789138281)
[ 7118.921910] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 7279.441307] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 7463.180131] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 10:58:13 (1789138693)
[ 7529.473236] Lustre: DEBUG MARKER: Restart...
[ 7537.632545] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7537.635494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7537.658514] Lustre: Skipped 1 previous similar message
[ 7537.676782] Lustre: Skipped 1 previous similar message
[ 7540.751734] Lustre: server umount lustre-MDT0000 complete
[ 7546.404097] LustreError: 142081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138779 with bad export cookie 29673868322058108
[ 7546.405143] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7546.419125] LustreError: 142081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 7546.675401] Lustre: server umount lustre-OST0000 complete
[ 7551.236349] Lustre: server umount lustre-OST0001 complete
[ 7566.248345] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local
[ 7577.330648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7577.334593] Lustre: Skipped 1 previous similar message
[ 7582.063527] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7585.011751] Lustre: 158051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7590.935089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7597.365344] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7602.147282] LustreError: 158416:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7602.173025] LustreError: 158416:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message
[ 7602.221428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:747 to 0x240000400:769)
[ 7606.197082] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7611.376169] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:745 to 0x280000400:769)
[ 7612.883435] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7621.135646] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7624.517913] Lustre: 159967:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7692.155185] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 11:02:01 (1789138921)
[ 7777.911564] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 11:03:28 (1789139008)
[ 9470.554662] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 11:31:40 (1789140700)
[ 9486.307480] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9486.309709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9486.332436] Lustre: Skipped 1 previous similar message
[ 9486.356773] Lustre: Skipped 1 previous similar message
[ 9488.467454] Lustre: server umount lustre-MDT0000 complete
[ 9492.894056] LustreError: 157603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140725 with bad export cookie 29673868322064597
[ 9492.901910] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9492.910133] LustreError: 157603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 9493.051791] Lustre: server umount lustre-OST0000 complete
[ 9499.200517] Lustre: server umount lustre-OST0001 complete
[ 9515.419558] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local
[ 9529.144930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9535.593693] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9539.959286] Lustre: 168147:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9549.592737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9553.636251] LustreError: 168519:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9553.680760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5771 to 0x240000400:5793)
[ 9558.284231] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9559.012719] LustreError: 168519:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9564.128884] LustreError: 168519:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9567.779639] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9572.879124] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5771 to 0x280000400:5793)
[ 9576.126194] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9584.427762] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9588.051139] Lustre: 170069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9628.754274] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 11:34:19 (1789140859)
[ 9682.588879] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 11:35:12 (1789140912)
[ 9731.410268] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 11:36:01 (1789140961)
[ 9733.709559] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9735.577390] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 11:36:06 (1789140966)
[ 9737.758988] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9740.579893] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 11:36:10 (1789140970)
[ 9844.994527] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 11:37:54 (1789141074)
[ 9900.623917] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 11:38:50 (1789141130)
[ 9994.289826] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 11:40:24 (1789141224)
[10006.579831] Lustre: *** cfs_fail_loc=a09, val=0***
[10006.581768] Lustre: Skipped 2 previous similar messages
[10008.602103] Lustre: *** cfs_fail_loc=a09, val=0***
[10008.604866] Lustre: Skipped 127 previous similar messages
[10012.610836] Lustre: *** cfs_fail_loc=a09, val=0***
[10012.619710] Lustre: Skipped 253 previous similar messages
[10020.630311] Lustre: *** cfs_fail_loc=a09, val=0***
[10020.633227] Lustre: Skipped 365 previous similar messages
[10036.676062] Lustre: *** cfs_fail_loc=a09, val=0***
[10036.679942] Lustre: Skipped 603 previous similar messages
[10068.725649] Lustre: *** cfs_fail_loc=a09, val=0***
[10068.734822] Lustre: Skipped 1239 previous similar messages
[10132.745899] Lustre: *** cfs_fail_loc=a09, val=0***
[10132.752575] Lustre: Skipped 3295 previous similar messages
[10535.930154] Lustre: *** cfs_fail_loc=a08, val=0***
[10535.932081] Lustre: Skipped 2111 previous similar messages
[10535.957122] Lustre: *** cfs_fail_loc=a08, val=0***
[10535.967218] Lustre: Skipped 1 previous similar message
[10806.323956] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[10806.324215] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10806.337200] LustreError: Skipped 1 previous similar message
[10806.378834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10806.394356] LustreError: 173364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789142039 with bad export cookie 29673868323842492
[10806.420533] LustreError: 173364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 14 previous similar messages
[10811.887325] Lustre: server umount lustre-MDT0000 complete
[10817.491530] LustreError: 167697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789142050 with bad export cookie 29673868323841813
[10817.535204] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10818.187828] Lustre: server umount lustre-OST0000 complete
[10824.585213] Lustre: server umount lustre-OST0001 complete
[10833.188557] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid
[10841.938375] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local
[10852.682247]  vdc: vdc1 vdc9
[10852.715584]  vdc: vdc1 vdc9
[10852.755981]  vdc: vdc1 vdc9
[10864.095634]  vde: vde1 vde9
[10876.196974]  vdf: vdf1 vdf9
[10893.567903] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local
[10903.390361] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[10903.611988] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[10903.688310] Lustre: lustre-MDT0000: new disk, initializing
[10904.231437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10904.366830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[10909.536599] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10914.723838] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[10921.372473] Lustre: lustre-OST0000: new disk, initializing
[10921.375560] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[10921.378792] Lustre: Skipped 1 previous similar message
[10921.503954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10923.104312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[10923.126760] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[10923.317216] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[10929.046529] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10939.712201] Lustre: lustre-OST0001: new disk, initializing
[10939.716867] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[10939.804963] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10941.781211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[10941.794893] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[10941.953433] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[10946.760294] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10956.097386] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10960.108816] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[10999.513938] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 11:57:09 (1789142229)
[11006.206721] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 11:57:16 (1789142236)
[11056.421719] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 11:58:06 (1789142286)
[11123.970778] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 11:59:14 (1789142354)
[11156.208502] Lustre: DEBUG MARKER: rename directory return 255
[11204.142228] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 12:00:34 (1789142434)
[11242.189919] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 12:01:11 (1789142471)
[11291.660733] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 12:02:01 (1789142521)
[11415.640689] LustreError: 184785:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11456.035468] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 12:04:46 (1789142686)
[11493.937294] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 12:05:24 (1789142724)
[11539.441108] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 12:06:09 (1789142769)
[11722.460495] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 12:09:12 (1789142952)
[11724.125914] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[11726.057084] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 12:09:16 (1789142956)
[11787.556573] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[11789.126376] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 12:10:19 (1789143019)
[11822.121421] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[11824.373901] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 12:10:54 (1789143054)
[11878.307082] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[11880.566406] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 12:11:50 (1789143110)
[11928.019566] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 12:12:38 (1789143158)
[11929.593948] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[11931.399234] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 12:12:41 (1789143161)
[11959.934603] LustreError: 207788:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[12004.302392] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 12:13:54 (1789143234)
[12035.927225] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[12039.267331] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[12215.328268] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 12:17:25 (1789143445)
[12268.528484] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 12:18:19 (1789143499)
[12305.011529] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 12:18:55 (1789143535)
[12306.797374] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[12309.323409] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 12:18:59 (1789143539)
[12311.132038] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[12313.172862] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 12:19:03 (1789143543)
[12333.353744] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[12352.683642] Lustre: DEBUG MARKER: Write...
[12355.579758] Lustre: DEBUG MARKER: Write out of block quota ...
[12410.156548] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 12:20:40 (1789143640)
[12439.153718] Lustre: DEBUG MARKER: set to use default quota
[12441.107438] Lustre: DEBUG MARKER: set default quota
[12442.800787] Lustre: DEBUG MARKER: get default quota
[12450.318442] Lustre: DEBUG MARKER: Test not out of quota
[12455.650504] Lustre: DEBUG MARKER: Test out of quota
[12467.102632] Lustre: DEBUG MARKER: Increase default quota
[12504.640127] Lustre: DEBUG MARKER: Set quota to override default quota
[12504.742394] LustreError: 191928:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1789748537 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12519.825448] Lustre: DEBUG MARKER: Set to use default quota again
[12541.068496] Lustre: DEBUG MARKER: Cleanup
[12616.481644] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 12:24:06 (1789143846)
[12642.522908] Lustre: DEBUG MARKER: set default quota for qpool1
[12644.212344] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[12696.086259] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 12:25:26 (1789143926)
[12782.424575] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 12:26:52 (1789144012)
[12870.276879] Lustre: DEBUG MARKER: Write...
[12873.575373] Lustre: DEBUG MARKER: Write out of block quota ...
[13004.531939] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 12:30:34 (1789144234)
[13055.410988] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 12:31:25 (1789144285)
[13062.638831] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 12:31:33 (1789144293)
[13063.989235] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate
[13065.564921] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 12:31:36 (1789144296)
[13067.064324] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate
[13068.767823] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 12:31:39 (1789144299)
[13094.929350] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 12:32:03 (1789144323)
[13097.403668] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[13099.974646] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 12:32:09 (1789144329)
[13118.707348] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[13126.339728] LustreError: 232601:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[13126.350043] LustreError: 232601:0:(qmt_pool.c:1407:qmt_pool_recalc()) Skipped 5 previous similar messages
[13129.185910] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[13129.213632] Lustre: Skipped 1 previous similar message
[13129.222294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[13129.224677] Lustre: Skipped 1 previous similar message
[13132.932269] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.16@tcp (stopping)
[13134.305190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[13136.415269] LustreError: 232601:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[13136.864061] Lustre: server umount lustre-MDT0000 complete
[13146.200470] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[13147.188445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[13147.402297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[13147.405476] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[13153.085100] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[13159.397526] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[13159.404098] Lustre: Skipped 1 previous similar message
[13190.634764] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 12:33:41 (1789144421)
[13227.693783] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 12:34:17 (1789144457)
[13268.969231] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 12:34:58 (1789144498)
[13330.382586] Lustre: *** cfs_fail_loc=a08, val=0***
[13330.399358] Lustre: Skipped 14 previous similar messages
[13330.416774] Lustre: *** cfs_fail_loc=a08, val=0***
[13330.420445] Lustre: Skipped 15 previous similar messages
[13469.175601] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 12:38:19 (1789144699)
[13559.392946] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 12:39:49 (1789144789)
[13631.056941] LustreError: 233190:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1789749663 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13800.161759] LustreError: 233604:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1789749832 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13810.614082] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed