[ 5668.936442] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5674.279097] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5677.739214] LustreError: 125663:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5678.671141] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5680.442408] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5682.021808] Lustre: Failing over lustre-MDT0000
[ 5682.391954] Lustre: server umount lustre-MDT0000 complete
[ 5699.936086] Lustre: 3307:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788844433/real 1788844433]  req@ffff95498e8b6a00 x1875733489394048/t0(0) o400->MGC192.168.202.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788844449 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5699.951493] Lustre: 3307:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5699.956248] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5700.048262] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5705.184455] Lustre: 3307:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788844438/real 1788844438]  req@ffff954ac1362300 x1875733489394560/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788844454 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5705.218649] Lustre: 3307:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5710.307057] LustreError: 3303:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff95498e8b5880 x1875733489396096/t0(0) o250->MGC192.168.202.119@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5710.313605] Lustre: 3306:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788844443/real 1788844443]  req@ffff954ab8a42680 x1875733489395072/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788844459 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5710.898186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5710.978078] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5713.188456] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5713.306161] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5713.380897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:629 to 0x240000400:673)
[ 5713.384779] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:625 to 0x280000400:641)
[ 5715.425258] Lustre: 3304:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788844448/real 1788844448]  req@ffff954ab80fca80 x1875733489395328/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788844464 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5715.445685] Lustre: 3304:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5716.395626] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5725.168672] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5725.707614] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5726.902230] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5731.805129] Lustre: DEBUG MARKER: (dd_pid=117984, time=0, timeout=600)
[ 5779.356297] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5785.094955] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5788.123266] LustreError: 128614:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5788.836737] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5790.289894] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5792.037431] Lustre: Failing over lustre-MDT0000
[ 5792.464322] Lustre: server umount lustre-MDT0000 complete
[ 5808.096044] Lustre: 3307:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788844541/real 1788844541]  req@ffff954ac0931f80 x1875733489430656/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1788844557 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5808.127538] Lustre: 3307:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5808.145577] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5808.156598] Lustre: Skipped 1 previous similar message
[ 5810.999944] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5811.314091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5811.363088] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5811.831708] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5812.114651] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5812.176876] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:675 to 0x240000400:705)
[ 5812.177775] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:625 to 0x280000400:673)
[ 5816.646970] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5818.419469] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 5818.422017] Lustre: Skipped 1 previous similar message
[ 5819.492777] LustreError: 3303:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-lwp-OST0001: namespace resource [0x200000006:0x20000:0x0].0x0 (ffff9549c8ff6600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5819.515365] LustreError: 3303:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 5 previous similar messages
[ 5824.607581] Lustre: 3306:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788844557/real 1788844557]  req@ffff95498f1ed500 x1875733489432192/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788844573 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5824.629741] Lustre: 3306:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 5826.782358] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5828.854823] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5835.348522] Lustre: DEBUG MARKER: (dd_pid=120466, time=1, timeout=600)
[ 5902.294693] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 01:17:30 (1788844650)
[ 5928.174456] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5930.075760] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5937.305132] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5939.497454] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5975.040391] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 01:18:43 (1788844723)
[ 6000.940151] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 6017.859579] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 6019.332783] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 6021.354931] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 6022.909173] Lustre: DEBUG MARKER: Set quota for 1 times
[ 6025.803897] Lustre: DEBUG MARKER: Set quota for 2 times
[ 6030.030536] Lustre: DEBUG MARKER: Set quota for 3 times
[ 6034.202860] Lustre: DEBUG MARKER: Set quota for 4 times
[ 6039.287670] Lustre: DEBUG MARKER: Set quota for 5 times
[ 6043.367719] Lustre: DEBUG MARKER: Set quota for 6 times
[ 6046.980718] Lustre: DEBUG MARKER: Set quota for 7 times
[ 6050.961949] Lustre: DEBUG MARKER: Set quota for 8 times
[ 6088.201503] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 01:20:36 (1788844836)
[ 6101.472253] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6101.477595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6101.491731] Lustre: Skipped 2 previous similar messages
[ 6101.498595] Lustre: Skipped 1 previous similar message
[ 6103.095730] Lustre: server umount lustre-MDT0000 complete
[ 6107.175444] LustreError: 14309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844856 with bad export cookie 17733005252457748042
[ 6107.183982] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6107.187268] LustreError: 14309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 6107.286104] Lustre: server umount lustre-OST0000 complete
[ 6111.672459] Lustre: server umount lustre-OST0001 complete
[ 6127.084482] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local
[ 6136.007726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6140.828905] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6144.799314] Lustre: 138279:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6153.888776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6160.828986] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6160.929795] LustreError: 138654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6161.011368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:707 to 0x240000400:737)
[ 6165.995196] LustreError: 139031:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6169.255717] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6174.786931] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:676 to 0x280000400:705)
[ 6177.048319] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6187.158379] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6191.337208] Lustre: 140201:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6213.665796] Lustre: server umount lustre-MDT0000 complete
[ 6218.795483] LustreError: 137827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844968 with bad export cookie 17733005252457757779
[ 6218.815924] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6229.129492] Lustre: server umount lustre-OST0000 complete
[ 6230.943155] Lustre: 3305:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788844964/real 1788844964]  req@ffff954ab9511c00 x1875733489597312/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788844980 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6235.004357] Lustre: server umount lustre-OST0001 complete
[ 6253.437633] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local
[ 6266.279607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6270.820296] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6273.506694] Lustre: 142863:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6286.367879] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6291.048509] LustreError: 143236:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6291.073588] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:707 to 0x240000400:769)
[ 6295.960438] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6295.972799] Lustre: Skipped 1 previous similar message
[ 6301.173696] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:676 to 0x280000400:737)
[ 6303.397584] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6310.837121] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6314.375720] Lustre: 144773:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6324.920591] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 01:24:33 (1788845073)
[ 6327.113424] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6329.689190] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 01:24:37 (1788845077)
[ 6390.164306] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 01:25:38 (1788845138)
[ 6439.138154] Lustre: DEBUG MARKER: Write...
[ 6441.450921] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6496.893303] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 01:27:25 (1788845245)
[ 6506.342611] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 01:27:33 (1788845253)
[ 6519.548249] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 01:27:48 (1788845268)
[ 6529.527810] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 01:27:57 (1788845277)
[ 6537.080385] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 01:28:05 (1788845285)
[ 6601.006258] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6765.894635] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 6956.940460] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 01:35:05 (1788845705)
[ 7023.361977] Lustre: DEBUG MARKER: Restart...
[ 7029.727739] LustreError: 3306:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff9549c602ce00 x1875733489828736/t0(0) o601->lustre-MDT0000-lwp-MDT0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 7029.728552] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 7029.760961] LustreError: 3306:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-MDT0000 qtype:usr lqe: ffff954a910f2900 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2661 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 7029.761590] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7029.773941] LustreError: Skipped 2 previous similar messages
[ 7029.797644] Lustre: Skipped 1 previous similar message
[ 7029.810446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7029.831572] Lustre: Skipped 1 previous similar message
[ 7031.267714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7034.163994] Lustre: server umount lustre-MDT0000 complete
[ 7038.547961] LustreError: 142418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845787 with bad export cookie 17733005252457759305
[ 7038.555469] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7038.558954] LustreError: 142418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 7038.703295] Lustre: server umount lustre-OST0000 complete
[ 7042.965276] Lustre: server umount lustre-OST0001 complete
[ 7059.183235] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local
[ 7069.110903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7073.104609] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7075.524268] Lustre: 163381:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7081.652763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7088.200753] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7093.858987] LustreError: 163745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7093.877376] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:779 to 0x240000400:801)
[ 7096.258613] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7101.453591] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:745 to 0x280000400:769)
[ 7102.978548] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7111.596170] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7116.299417] Lustre: 165281:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7183.305113] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 01:38:51 (1788845931)
[ 7258.276637] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 01:40:06 (1788846006)
[ 8819.954749] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 02:06:07 (1788847567)
[ 8838.136825] Lustre: server umount lustre-MDT0000 complete
[ 8842.892456] LustreError: 162934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847592 with bad export cookie 17733005252457765808
[ 8842.906591] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8853.604972] Lustre: server umount lustre-OST0000 complete
[ 8867.926333] Lustre: server umount lustre-OST0001 complete
[ 8884.131405] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local
[ 8897.124625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8903.075470] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8907.059971] Lustre: 173428:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 8914.167972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 8921.746117] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8922.337698] LustreError: 173793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8922.377424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5803 to 0x240000400:5825)
[ 8927.714191] LustreError: 173793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8931.794690] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 8936.986773] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5771 to 0x280000400:5793)
[ 8940.383161] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8947.909094] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8951.570537] Lustre: 175336:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 8988.404056] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 02:08:56 (1788847736)
[ 9034.656304] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 02:09:42 (1788847782)
[ 9088.419460] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 02:10:36 (1788847836)
[ 9090.002745] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9091.947078] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 02:10:40 (1788847840)
[ 9093.394716] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9095.428548] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 02:10:43 (1788847843)
[ 9177.772361] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 02:12:06 (1788847926)
[ 9226.102211] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 02:12:54 (1788847974)
[ 9311.188309] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 02:14:19 (1788848059)
[ 9322.879130] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9322.881639] Lustre: Skipped 2 previous similar messages
[ 9324.958642] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9324.960514] Lustre: Skipped 111 previous similar messages
[ 9328.965918] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9328.971566] Lustre: Skipped 247 previous similar messages
[ 9336.968142] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9336.970512] Lustre: Skipped 513 previous similar messages
[ 9352.973787] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9352.977987] Lustre: Skipped 546 previous similar messages
[ 9384.980669] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9384.983777] Lustre: Skipped 1378 previous similar messages
[ 9448.987579] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9448.991980] Lustre: Skipped 3367 previous similar messages
[ 9717.545160] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9717.551904] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9717.552616] Lustre: Skipped 1836 previous similar messages
[ 9717.566541] Lustre: Skipped 4 previous similar messages
[ 9858.887831] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9858.893623] Lustre: Skipped 9 previous similar messages
[10017.251406] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10017.253177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10017.281064] Lustre: Skipped 1 previous similar message
[10017.308117] Lustre: Skipped 2 previous similar messages
[10021.421099] Lustre: server umount lustre-MDT0000 complete
[10025.750793] LustreError: 178825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848775 with bad export cookie 17733005252459541127
[10025.772575] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10026.074165] Lustre: server umount lustre-OST0000 complete
[10031.521407] Lustre: server umount lustre-OST0001 complete
[10039.848429] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid
[10049.112827] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local
[10059.282190]  vdc: vdc1 vdc9
[10070.602343]  vde: vde1 vde9
[10082.050857]  vdf: vdf1 vdf9
[10100.546621] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local
[10112.134659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[10112.431857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[10112.528415] Lustre: lustre-MDT0000: new disk, initializing
[10113.048659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10113.180949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[10118.786903] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10124.336876] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[10131.380669] Lustre: lustre-OST0000: new disk, initializing
[10131.384754] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[10131.390667] Lustre: Skipped 1 previous similar message
[10131.487687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10133.150669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[10133.161680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[10133.309837] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[10138.148773] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10147.903967] Lustre: lustre-OST0001: new disk, initializing
[10147.907720] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[10147.995792] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10149.509229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[10149.516517] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[10149.717870] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[10154.652898] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10162.637732] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10165.725334] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[10200.506687] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 02:29:08 (1788848948)
[10206.591937] Lustre: DEBUG MARKER: == sanity-quota test 49c: lfs quota long options don't consume an extra argument ========================================================== 02:29:15 (1788848955)
[10213.503065] Lustre: DEBUG MARKER: == sanity-quota test 49d: lfs quota -d and --delimiter both work ========================================================== 02:29:21 (1788848961)
[10219.720520] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 02:29:28 (1788848968)
[10261.888916] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 02:30:10 (1788849010)
[10318.183159] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 02:31:06 (1788849066)
[10344.884660] Lustre: DEBUG MARKER: rename directory return 255
[10390.912402] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 02:32:19 (1788849139)
[10422.729786] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 02:32:50 (1788849170)
[10463.789748] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 02:33:32 (1788849212)
[10600.435593] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 02:35:49 (1788849349)
[10631.376850] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 02:36:20 (1788849380)
[10677.249649] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 02:37:05 (1788849425)
[10829.111563] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 02:39:37 (1788849577)
[10830.312470] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[10831.774724] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 02:39:40 (1788849580)
[10886.104333] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[10887.738328] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 02:40:36 (1788849636)
[10916.459877] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[10917.708408] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 02:41:06 (1788849666)
[10967.240463] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[10968.807589] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 02:41:57 (1788849717)
[11014.365317] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 02:42:42 (1788849762)
[11015.668514] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[11017.214713] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 02:42:45 (1788849765)
[11041.932602] LustreError: 213941:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[11079.870289] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 02:43:48 (1788849828)
[11103.303460] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[11104.962677] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[11243.797500] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 02:46:32 (1788849992)
[11291.367518] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 02:47:20 (1788850040)
[11321.294761] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 02:47:49 (1788850069)
[11322.518949] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[11323.778938] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 02:47:52 (1788850072)
[11325.084788] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[11326.790903] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 02:47:55 (1788850075)
[11341.923232] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[11355.913755] Lustre: DEBUG MARKER: Write...
[11358.102938] Lustre: DEBUG MARKER: Write out of block quota ...
[11409.550570] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 02:49:18 (1788850158)
[11434.444184] Lustre: DEBUG MARKER: set to use default quota
[11435.706696] Lustre: DEBUG MARKER: set default quota
[11437.114285] Lustre: DEBUG MARKER: get default quota
[11442.176756] Lustre: DEBUG MARKER: Test not out of quota
[11445.152133] Lustre: DEBUG MARKER: Test out of quota
[11455.307531] Lustre: DEBUG MARKER: Increase default quota
[11480.660895] Lustre: DEBUG MARKER: Set quota to override default quota
[11480.749063] LustreError: 216206:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1789455030 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11487.601791] Lustre: DEBUG MARKER: Set to use default quota again
[11503.621948] Lustre: DEBUG MARKER: Cleanup
[11570.080655] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 02:51:58 (1788850318)
[11591.841422] Lustre: DEBUG MARKER: set default quota for qpool1
[11593.001678] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[11637.729049] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 02:53:06 (1788850386)
[11702.517965] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 02:54:11 (1788850451)
[11758.518784] Lustre: DEBUG MARKER: Write...
[11760.719370] Lustre: DEBUG MARKER: Write out of block quota ...
[11852.371536] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 02:56:41 (1788850601)
[11892.762766] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 02:57:21 (1788850641)
[11897.763409] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 02:57:26 (1788850646)
[11899.472118] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A fallocate not supported
[11900.727230] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 02:57:29 (1788850649)
[11902.661505] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a fallocate not supported
[11903.940549] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 02:57:32 (1788850652)
[11928.138355] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 02:57:55 (1788850675)
[11930.783520] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[11933.331365] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 02:58:00 (1788850680)
[11956.714687] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[11965.660808] LustreError: 239529:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[11970.224485] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.19@tcp (stopping)
[11972.064408] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[11972.067114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[11972.095059] Lustre: Skipped 1 previous similar message
[11972.115943] Lustre: Skipped 1 previous similar message
[11975.323085] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.19@tcp (stopping)
[11975.679145] LustreError: 239529:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[11976.161753] Lustre: server umount lustre-MDT0000 complete
[11986.527246] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[11987.155930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[11987.394389] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[11987.423364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[11992.232924] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[11992.242118] Lustre: Skipped 1 previous similar message
[11992.844298] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[12028.970405] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 02:59:37 (1788850777)
[12062.537357] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 03:00:10 (1788850810)
[12094.627377] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 03:00:42 (1788850842)
[12148.984760] Lustre: *** cfs_fail_loc=a08, val=0***
[12148.993521] Lustre: Skipped 26 previous similar messages
[12149.010792] Lustre: *** cfs_fail_loc=a08, val=0***
[12149.013809] Lustre: Skipped 15 previous similar messages
[12270.765061] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 03:03:39 (1788851019)
[12353.754994] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 03:05:02 (1788851102)
[12427.912050] LustreError: 245525:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1789455977 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12573.188477] LustreError: 240115:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1789456122 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12573.290929] LustreError: 240153:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:1 until pending updates are flushed.
[12581.878482] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed