[ 5689.346699] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5695.153941] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5698.387623] LustreError: 120920:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5699.506677] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5701.402146] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5703.376238] Lustre: Failing over lustre-MDT0000
[ 5703.681058] Lustre: server umount lustre-MDT0000 complete
[ 5721.056691] Lustre: 3301:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787120871/real 1787120871]  req@ffff8b01895d0a80 x1873926183657856/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787120887 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5721.065854] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5721.102210] Lustre: 3301:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5721.102273] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5722.079160] Lustre: 3299:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787120872/real 1787120872]  req@ffff8b0142dc6300 x1873926183657984/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1787120888 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5722.116871] Lustre: 3299:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5725.471174] Lustre: 3302:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787120876/real 1787120876]  req@ffff8b0274877b80 x1873926183658368/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787120892 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5730.271171] Lustre: 3300:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787120881/real 1787120881]  req@ffff8b0258436680 x1873926183658624/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787120897 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5730.336765] Lustre: 3300:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5730.854771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5730.998613] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5733.590092] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5733.805651] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5733.880490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:581 to 0x240000400:609)
[ 5733.880904] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:577 to 0x280000400:609)
[ 5736.302510] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5746.147317] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5746.964400] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5748.645053] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5764.746700] Lustre: DEBUG MARKER: (dd_pid=114475, time=10, timeout=600)
[ 5813.483962] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5819.876595] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5823.039250] LustreError: 123867:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5824.051407] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5825.929594] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5828.101861] Lustre: Failing over lustre-MDT0000
[ 5828.539635] Lustre: server umount lustre-MDT0000 complete
[ 5844.447215] Lustre: 3302:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787120995/real 1787120995]  req@ffff8b027b47e680 x1873926183698688/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1787121011 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5844.463251] Lustre: 3302:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5844.471393] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5844.489406] Lustre: Skipped 1 previous similar message
[ 5845.521825] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5845.996801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5846.047949] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5846.260976] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5846.430686] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5846.477169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:611 to 0x240000400:641)
[ 5846.478905] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:577 to 0x280000400:641)
[ 5850.018465] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5850.022779] Lustre: Skipped 1 previous similar message
[ 5850.223240] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5860.214458] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5860.559498] Lustre: 3301:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787121011/real 1787121011]  req@ffff8b018473f100 x1873926183700096/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787121027 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5860.585867] Lustre: 3301:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 5862.066176] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5876.250224] Lustre: DEBUG MARKER: (dd_pid=116967, time=7, timeout=600)
[ 5951.569345] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 02:31:56 (1787121116)
[ 5973.664412] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5975.407578] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5981.970500] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5983.640118] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6021.485621] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 02:33:05 (1787121185)
[ 6049.270716] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 6065.437139] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 6067.293572] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 6069.136491] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 6071.610234] Lustre: DEBUG MARKER: Set quota for 1 times
[ 6075.814556] Lustre: DEBUG MARKER: Set quota for 2 times
[ 6079.362550] Lustre: DEBUG MARKER: Set quota for 3 times
[ 6082.819733] Lustre: DEBUG MARKER: Set quota for 4 times
[ 6086.777344] Lustre: DEBUG MARKER: Set quota for 5 times
[ 6090.055591] Lustre: DEBUG MARKER: Set quota for 6 times
[ 6093.756279] Lustre: DEBUG MARKER: Set quota for 7 times
[ 6097.538503] Lustre: DEBUG MARKER: Set quota for 8 times
[ 6156.495171] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 02:35:20 (1787121320)
[ 6171.278813] Lustre: server umount lustre-MDT0000 complete
[ 6176.282130] LustreError: 5771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121343 with bad export cookie 5130498964340221027
[ 6176.292991] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6186.532424] Lustre: server umount lustre-OST0000 complete
[ 6188.833084] Lustre: 3299:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787121339/real 1787121339]  req@ffff8b0143251180 x1873926183826432/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787121355 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6188.861836] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6188.877185] Lustre: Skipped 1 previous similar message
[ 6190.343054] Lustre: server umount lustre-OST0001 complete
[ 6203.103750] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local
[ 6212.268128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6216.584591] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6219.616706] Lustre: 133649:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6225.401535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6231.406949] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6236.647145] LustreError: 133979:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6236.679368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:644 to 0x240000400:673)
[ 6236.688264] LustreError: 133979:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message
[ 6239.098259] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6244.352940] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:643 to 0x280000400:673)
[ 6246.585499] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6254.463532] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6259.151310] Lustre: 135481:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6280.162681] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6280.190281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6282.073299] Lustre: server umount lustre-MDT0000 complete
[ 6286.687665] LustreError: 133199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121453 with bad export cookie 5130498964340228384
[ 6286.707462] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6286.826522] Lustre: server umount lustre-OST0000 complete
[ 6291.463446] Lustre: server umount lustre-OST0001 complete
[ 6309.082636] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local
[ 6318.154307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6322.529453] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6325.306688] Lustre: 138086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6336.493919] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6342.773868] LustreError: 139165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6342.798957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:644 to 0x240000400:705)
[ 6343.384809] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6343.388828] Lustre: Skipped 1 previous similar message
[ 6348.808091] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:643 to 0x280000400:705)
[ 6349.834364] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6357.287056] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6360.579678] Lustre: 139894:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6370.909277] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 02:38:55 (1787121535)
[ 6372.623971] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6374.598152] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 02:38:58 (1787121538)
[ 6434.176258] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 02:39:58 (1787121598)
[ 6483.904642] Lustre: DEBUG MARKER: Write...
[ 6486.658297] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6541.347210] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 02:41:45 (1787121705)
[ 6547.427753] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 02:41:51 (1787121711)
[ 6558.254910] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 02:42:02 (1787121722)
[ 6567.131230] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 02:42:11 (1787121731)
[ 6576.106602] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 02:42:20 (1787121740)
[ 6640.336351] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6790.467431] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 6983.394283] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 02:49:08 (1787122148)
[ 7041.068509] Lustre: DEBUG MARKER: Restart...
[ 7045.089824] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7045.100438] Lustre: Skipped 1 previous similar message
[ 7045.103729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7045.106684] Lustre: Skipped 1 previous similar message
[ 7045.287650] Lustre: server umount lustre-MDT0000 complete
[ 7049.094189] LustreError: 140642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122215 with bad export cookie 5130498964340229917
[ 7049.101935] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7049.106681] LustreError: 140642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 7051.279399] Lustre: server umount lustre-OST0000 complete
[ 7055.212394] Lustre: server umount lustre-OST0001 complete
[ 7068.687028] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local
[ 7077.109761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7081.422397] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7084.259238] Lustre: 159089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7089.236408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7095.065180] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7101.410299] LustreError: 159394:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7101.456313] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:714 to 0x240000400:737)
[ 7101.924436] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7107.087826] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:714 to 0x280000400:737)
[ 7107.760653] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7115.206738] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7118.697941] Lustre: 160893:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7187.075434] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 02:52:31 (1787122351)
[ 7260.760103] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 02:53:45 (1787122425)
[ 9109.565117] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 03:24:33 (1787124273)
[ 9127.392779] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9127.416113] Lustre: Skipped 1 previous similar message
[ 9127.439544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9127.444841] Lustre: Skipped 1 previous similar message
[ 9128.445842] Lustre: server umount lustre-MDT0000 complete
[ 9132.774604] LustreError: 158645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124299 with bad export cookie 5130498964340236413
[ 9132.784105] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9132.791980] LustreError: 158645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 9132.897985] Lustre: server umount lustre-OST0000 complete
[ 9136.829455] Lustre: server umount lustre-OST0001 complete
[ 9151.450986] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local
[ 9161.192049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9166.359403] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9170.049562] Lustre: 169451:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9178.645624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9185.761231] LustreError: 169773:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9185.805081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5739 to 0x240000400:5761)
[ 9187.190045] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9190.885551] LustreError: 170081:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9196.000761] LustreError: 169773:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9197.219309] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9202.177324] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5739 to 0x280000400:5761)
[ 9205.448144] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9213.721620] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9216.929637] Lustre: 171283:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9256.613166] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 03:27:00 (1787124420)
[ 9299.728605] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 03:27:43 (1787124463)
[ 9347.504674] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 03:28:32 (1787124512)
[ 9349.022691] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9351.050238] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 03:28:35 (1787124515)
[ 9352.640622] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9354.608825] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 03:28:38 (1787124518)
[ 9448.845588] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 03:30:13 (1787124613)
[ 9496.739206] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 03:31:01 (1787124661)
[ 9594.003633] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 03:32:38 (1787124758)
[ 9605.237291] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9605.239320] Lustre: Skipped 1 previous similar message
[ 9607.287596] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9607.289767] Lustre: Skipped 99 previous similar messages
[ 9611.334409] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9611.340082] Lustre: Skipped 183 previous similar messages
[ 9619.384436] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9619.388881] Lustre: Skipped 349 previous similar messages
[ 9635.388411] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9635.394493] Lustre: Skipped 707 previous similar messages
[ 9667.391099] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9667.396137] Lustre: Skipped 1427 previous similar messages
[ 9731.442316] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9731.445465] Lustre: Skipped 2941 previous similar messages
[10137.581243] Lustre: *** cfs_fail_loc=a08, val=0***
[10137.588088] Lustre: Skipped 2288 previous similar messages
[10137.609196] Lustre: *** cfs_fail_loc=a08, val=0***
[10379.747462] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10379.772223] Lustre: Skipped 1 previous similar message
[10379.780814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10379.793479] Lustre: Skipped 1 previous similar message
[10383.771174] Lustre: server umount lustre-MDT0000 complete
[10388.550862] LustreError: 169004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787125555 with bad export cookie 5130498964342017122
[10388.567360] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10389.099578] Lustre: server umount lustre-OST0000 complete
[10395.256322] Lustre: server umount lustre-OST0001 complete
[10403.512212] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid
[10412.016757] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local
[10421.593293]  vdc: vdc1 vdc9
[10421.617082]  vdc: vdc1 vdc9
[10421.650473]  vdc: vdc1 vdc9
[10432.325363]  vde: vde1 vde9
[10443.196161]  vdf: vdf1 vdf9
[10458.458792] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local
[10466.886887] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[10467.143649] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[10467.213765] Lustre: lustre-MDT0000: new disk, initializing
[10467.482260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10467.532823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[10471.757815] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10476.676931] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[10481.965544] Lustre: lustre-OST0000: new disk, initializing
[10481.970631] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[10481.974896] Lustre: Skipped 1 previous similar message
[10482.095159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10483.319027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[10483.328865] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[10483.414975] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[10488.122493] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10497.360451] Lustre: lustre-OST0001: new disk, initializing
[10497.363755] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[10497.495231] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10499.202700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[10499.209619] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[10499.359533] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[10503.917884] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10513.436241] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10517.914946] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[10559.328452] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 03:48:43 (1787125723)
[10566.197790] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 03:48:50 (1787125730)
[10618.970986] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 03:49:43 (1787125783)
[10679.949941] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 03:50:44 (1787125844)
[10713.105198] Lustre: DEBUG MARKER: rename directory return 255
[10765.530725] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 03:52:09 (1787125929)
[10797.954867] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 03:52:42 (1787125962)
[10843.818489] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 03:53:28 (1787126008)
[10953.537488] LustreError: 186280:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[10995.233787] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 03:55:58 (1787126158)
[11035.831254] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 03:56:39 (1787126199)
[11083.003518] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 03:57:27 (1787126247)
[11282.874992] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 04:00:47 (1787126447)
[11284.305303] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[11285.947477] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 04:00:50 (1787126450)
[11350.996406] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[11352.560836] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 04:01:57 (1787126517)
[11384.722418] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[11386.394874] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 04:02:30 (1787126550)
[11442.184801] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[11443.660650] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 04:03:28 (1787126608)
[11493.445977] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 04:04:17 (1787126657)
[11495.123780] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[11497.514375] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 04:04:21 (1787126661)
[11526.452491] LustreError: 209488:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[11572.075286] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 04:05:36 (1787126736)
[11599.693281] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[11601.636564] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[11811.324793] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 04:09:35 (1787126975)
[11864.965167] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 04:10:29 (1787127029)
[11900.595303] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 04:11:04 (1787127064)
[11902.113718] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[11903.910911] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 04:11:08 (1787127068)
[11905.548070] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[11907.410227] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 04:11:11 (1787127071)
[11927.723362] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[11944.848485] Lustre: DEBUG MARKER: Write...
[11947.732906] Lustre: DEBUG MARKER: Write out of block quota ...
[12007.504962] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 04:12:51 (1787127171)
[12039.388378] Lustre: DEBUG MARKER: set to use default quota
[12041.309735] Lustre: DEBUG MARKER: set default quota
[12042.922179] Lustre: DEBUG MARKER: get default quota
[12049.857454] Lustre: DEBUG MARKER: Test not out of quota
[12055.152964] Lustre: DEBUG MARKER: Test out of quota
[12068.744842] Lustre: DEBUG MARKER: Increase default quota
[12099.380692] Lustre: DEBUG MARKER: Set quota to override default quota
[12099.462794] LustreError: 186279:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1787732066 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12114.149914] Lustre: DEBUG MARKER: Set to use default quota again
[12138.698752] Lustre: DEBUG MARKER: Cleanup
[12212.147511] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 04:16:16 (1787127376)
[12236.704401] Lustre: DEBUG MARKER: set default quota for qpool1
[12238.232835] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[12293.509494] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 04:17:37 (1787127457)
[12382.338558] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 04:19:06 (1787127546)
[12459.879832] Lustre: DEBUG MARKER: Write...
[12463.883525] Lustre: DEBUG MARKER: Write out of block quota ...
[12591.501773] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 04:22:35 (1787127755)
[12642.171672] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 04:23:26 (1787127806)
[12650.905410] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 04:23:35 (1787127815)
[12652.619776] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate
[12654.628670] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 04:23:38 (1787127818)
[12656.364406] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate
[12658.617641] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 04:23:42 (1787127822)
[12680.914907] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 04:24:05 (1787127845)
[12682.901681] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[12685.043288] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 04:24:09 (1787127849)
[12701.898705] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[12708.487936] LustreError: 234673:0:(qmt_pool.c:1413:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[12710.884899] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[12710.901595] Lustre: Skipped 1 previous similar message
[12710.913742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[12710.917905] Lustre: Skipped 1 previous similar message
[12714.735745] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.42@tcp (stopping)
[12714.975262] LustreError: 3299:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff8b027d866a00 x1873926192902528/t0(0) o601->lustre-MDT0000-lwp-MDT0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[12714.999463] LustreError: 3302:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-MDT0000 qtype:prj lqe: ffff8b018013cd80 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 374 qunit:0 qtune:0 edquot:0 default:no revoke:0
[12715.028633] LustreError: 3302:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 2 previous similar messages
[12716.007248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[12716.016518] Lustre: Skipped 1 previous similar message
[12718.592059] LustreError: 234673:0:(qmt_pool.c:1413:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[12718.986375] Lustre: server umount lustre-MDT0000 complete
[12726.809329] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[12727.262187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[12727.361605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[12727.362938] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[12731.768585] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[12738.533156] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[12738.551846] Lustre: Skipped 1 previous similar message
[12765.913276] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 04:25:30 (1787127930)
[12802.376768] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 04:26:06 (1787127966)
[12833.982479] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 04:26:38 (1787127998)
[12889.317656] Lustre: *** cfs_fail_loc=a08, val=0***
[12889.322817] Lustre: Skipped 13 previous similar messages
[12889.328160] Lustre: *** cfs_fail_loc=a08, val=0***
[12889.332657] Lustre: Skipped 15 previous similar messages
[13014.595281] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 04:29:39 (1787128179)
[13097.852160] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 04:31:02 (1787128262)
[13180.054683] LustreError: 235213:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1787733146 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13189.148710] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed