[ 5681.283808] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5688.144484] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5691.823212] LustreError: 120564:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5692.909948] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5694.995382] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5697.383966] Lustre: Failing over lustre-MDT0000
[ 5697.758069] Lustre: server umount lustre-MDT0000 complete
[ 5715.938527] Lustre: 3302:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786080291/real 1786080291]  req@ffff9f087d1ea300 x1872835077362688/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1786080307 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5715.943493] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5715.978353] Lustre: 3302:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5715.978413] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5720.099714] Lustre: 3302:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786080296/real 1786080296]  req@ffff9f087c92f800 x1872835077363200/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786080312 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5720.165205] Lustre: 3302:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5725.152468] LustreError: 3298:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff9f0848c28700 x1872835077365504/t0(0) o250->MGC192.168.202.134@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5725.265389] Lustre: 3301:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786080301/real 1786080301]  req@ffff9f0848c2b800 x1872835077363584/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786080317 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5725.598368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5725.753542] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5730.582790] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5730.720941] Lustre: 3300:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786080306/real 1786080306]  req@ffff9f074f33ed80 x1872835077363968/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786080322 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5730.764082] Lustre: 3300:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5739.400100] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5739.686147] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5739.803805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:575 to 0x240000400:609)
[ 5739.807297] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:571 to 0x280000400:609)
[ 5741.038373] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5747.270910] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5749.055363] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5763.929198] Lustre: DEBUG MARKER: (dd_pid=114523, time=6, timeout=600)
[ 5817.384269] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5824.181178] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5827.384432] LustreError: 123391:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5828.449952] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5830.070585] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5832.098624] Lustre: Failing over lustre-MDT0000
[ 5832.580207] Lustre: server umount lustre-MDT0000 complete
[ 5848.551099] Lustre: 3300:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786080424/real 1786080424]  req@ffff9f086c3e4700 x1872835077406336/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1786080440 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5848.578681] Lustre: 3300:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5848.588740] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5848.610059] Lustre: Skipped 1 previous similar message
[ 5849.632872] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5859.844635] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x4b185927a70de945
[ 5859.856827] Lustre: MGC192.168.202.134@tcp: Connection restored to 0@lo (at 0@lo)
[ 5859.859208] Lustre: Skipped 1 previous similar message
[ 5860.392840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5860.484533] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5862.281500] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5862.444951] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5862.503438] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:611 to 0x240000400:641)
[ 5862.504730] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:571 to 0x280000400:641)
[ 5865.247832] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5876.016977] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5877.753627] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5896.427150] Lustre: DEBUG MARKER: (dd_pid=116941, time=11, timeout=600)
[ 5962.786672] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 01:29:12 (1786080552)
[ 5988.556943] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5990.790891] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5997.120758] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5999.077303] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6035.205460] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 01:30:25 (1786080625)
[ 6063.579521] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 6079.748484] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 6081.264665] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 6082.902745] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 6084.676397] Lustre: DEBUG MARKER: Set quota for 1 times
[ 6088.515648] Lustre: DEBUG MARKER: Set quota for 2 times
[ 6093.008331] Lustre: DEBUG MARKER: Set quota for 3 times
[ 6097.310152] Lustre: DEBUG MARKER: Set quota for 4 times
[ 6101.636233] Lustre: DEBUG MARKER: Set quota for 5 times
[ 6106.461934] Lustre: DEBUG MARKER: Set quota for 6 times
[ 6110.709093] Lustre: DEBUG MARKER: Set quota for 7 times
[ 6180.590349] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 01:32:50 (1786080770)
[ 6195.738270] Lustre: server umount lustre-MDT0000 complete
[ 6200.066623] LustreError: 5759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786080792 with bad export cookie 5411172979126954309
[ 6200.079808] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6210.613045] Lustre: server umount lustre-OST0000 complete
[ 6213.536109] Lustre: 3302:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786080789/real 1786080789]  req@ffff9f0785ff4380 x1872835077537664/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786080805 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6213.584949] Lustre: 3302:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
[ 6213.590602] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6213.606221] Lustre: Skipped 1 previous similar message
[ 6215.444730] Lustre: server umount lustre-OST0001 complete
[ 6232.442501] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local
[ 6244.280396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6249.385991] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6252.170650] Lustre: 133077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6258.441725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6266.079942] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6268.642105] LustreError: 133397:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6268.663358] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:643 to 0x240000400:673)
[ 6268.668263] LustreError: 133397:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message
[ 6274.020446] Lustre: lustre-OST0001: Not available for connect from 0@lo (not set up)
[ 6274.446521] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6279.692649] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:644 to 0x280000400:673)
[ 6283.470426] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6290.718349] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6294.445471] Lustre: 134904:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6310.388852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6310.404273] Lustre: Skipped 1 previous similar message
[ 6312.265878] Lustre: server umount lustre-MDT0000 complete
[ 6317.164391] LustreError: 132624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786080909 with bad export cookie 5411172979126961099
[ 6317.167807] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6317.177025] LustreError: 132624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 6317.295713] Lustre: server umount lustre-OST0000 complete
[ 6321.982283] Lustre: server umount lustre-OST0001 complete
[ 6337.581643] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local
[ 6347.394666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6352.495784] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6355.272763] Lustre: 137504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6366.984513] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6372.962587] LustreError: 137827:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6372.993788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:643 to 0x240000400:705)
[ 6374.816464] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6374.819583] Lustre: Skipped 1 previous similar message
[ 6380.065432] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:644 to 0x280000400:705)
[ 6382.342428] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6390.375953] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6393.901866] Lustre: 139319:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6403.664881] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 01:36:33 (1786080993)
[ 6405.545274] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6407.430548] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 01:36:37 (1786080997)
[ 6462.583382] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 01:37:32 (1786081052)
[ 6514.854936] Lustre: DEBUG MARKER: Write...
[ 6517.419738] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6572.160370] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 01:39:22 (1786081162)
[ 6579.068313] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 01:39:29 (1786081169)
[ 6589.761824] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 01:39:39 (1786081179)
[ 6599.273125] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 01:39:49 (1786081189)
[ 6608.014477] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 01:39:58 (1786081198)
[ 6672.082765] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6824.595988] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 7013.314656] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 01:46:43 (1786081603)
[ 7073.371919] Lustre: DEBUG MARKER: Restart...
[ 7079.395573] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7079.400087] LustreError: 141817:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7079.423740] Lustre: Skipped 3 previous similar messages
[ 7079.473809] LustreError: 141817:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message
[ 7080.253882] Lustre: server umount lustre-MDT0000 complete
[ 7084.551626] LustreError: 141797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786081676 with bad export cookie 5411172979126962632
[ 7084.564352] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7084.662601] Lustre: server umount lustre-OST0000 complete
[ 7088.694670] Lustre: server umount lustre-OST0001 complete
[ 7103.134228] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local
[ 7111.748895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7116.464801] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7119.258280] Lustre: 157984:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7124.732300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7132.180535] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7136.942318] LustreError: 158297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7137.034659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:714 to 0x240000400:737)
[ 7139.565644] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7144.965635] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:714 to 0x280000400:737)
[ 7145.771458] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7153.422799] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7157.112655] Lustre: 159795:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7224.493926] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 01:50:14 (1786081814)
[ 7299.867981] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 01:51:30 (1786081890)
[ 8942.768753] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 02:18:53 (1786083533)
[ 8957.263989] Lustre: server umount lustre-MDT0000 complete
[ 8961.676520] LustreError: 157539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786083553 with bad export cookie 5411172979126969128
[ 8961.687375] LustreError: 157539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 8961.702750] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8971.416347] Lustre: server umount lustre-OST0000 complete
[ 8973.609986] Lustre: 3300:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786083549/real 1786083549]  req@ffff9f078553d880 x1872835080740864/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786083565 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 8973.640723] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8975.694936] Lustre: server umount lustre-OST0001 complete
[ 8990.151120] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local
[ 9000.407607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9005.797159] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9009.047981] Lustre: 168301:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9014.881293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9021.252949] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9025.059843] LustreError: 168615:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9025.183983] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5739 to 0x240000400:5761)
[ 9030.113166] LustreError: 168615:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9030.935127] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9036.273347] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5739 to 0x280000400:5761)
[ 9039.096306] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9046.787197] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9050.578517] Lustre: 170118:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9091.461695] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 02:21:21 (1786083681)
[ 9140.667288] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 02:22:10 (1786083730)
[ 9193.422294] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 02:23:03 (1786083783)
[ 9195.110289] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9197.211742] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 02:23:07 (1786083787)
[ 9198.931357] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9200.854547] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 02:23:11 (1786083791)
[ 9299.491921] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 02:24:49 (1786083889)
[ 9353.557839] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 02:25:43 (1786083943)
[ 9451.653658] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 02:27:21 (1786084041)
[ 9464.397398] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9464.402353] Lustre: Skipped 3 previous similar messages
[ 9466.439977] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9466.441681] Lustre: Skipped 71 previous similar messages
[ 9470.444977] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9470.452756] Lustre: Skipped 167 previous similar messages
[ 9478.484828] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9478.486630] Lustre: Skipped 335 previous similar messages
[ 9494.523319] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9494.527435] Lustre: Skipped 801 previous similar messages
[ 9526.527468] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9526.530410] Lustre: Skipped 1501 previous similar messages
[ 9590.541754] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9590.551193] Lustre: Skipped 2611 previous similar messages
[ 9997.160900] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9997.174362] Lustre: Skipped 2508 previous similar messages
[ 9997.183430] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9997.188516] Lustre: Skipped 2 previous similar messages
[10283.502091] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10283.512442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10283.530530] Lustre: Skipped 1 previous similar message
[10283.550529] Lustre: Skipped 1 previous similar message
[10288.613523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10288.627565] Lustre: Skipped 1 previous similar message
[10288.876874] Lustre: server umount lustre-MDT0000 complete
[10293.197896] LustreError: 167854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786084885 with bad export cookie 5411172979128745000
[10293.214440] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10293.730578] Lustre: server umount lustre-OST0000 complete
[10298.565415] Lustre: server umount lustre-OST0001 complete
[10305.757406] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid
[10315.407703] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local
[10327.785921]  vdc: vdc1 vdc9
[10339.830805]  vde: vde1 vde9
[10339.867278]  vde: vde1 vde9
[10339.906273]  vde: vde1 vde9
[10349.802450]  vdf: vdf1 vdf9
[10367.642980] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local
[10377.497638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[10377.810431] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[10377.865616] Lustre: lustre-MDT0000: new disk, initializing
[10378.241733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10378.377601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[10383.807668] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10389.513410] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[10395.654259] Lustre: lustre-OST0000: new disk, initializing
[10395.657495] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[10395.664496] Lustre: Skipped 1 previous similar message
[10395.768273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10397.271508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[10397.281447] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[10397.475692] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[10402.800328] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10412.792777] Lustre: lustre-OST0001: new disk, initializing
[10412.795722] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[10412.896919] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10414.186776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[10414.194753] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[10414.378288] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[10419.255871] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10428.737434] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10432.389184] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[10469.434291] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 02:44:19 (1786085059)
[10475.747215] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 02:44:26 (1786085066)
[10527.426233] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 02:45:17 (1786085117)
[10591.250425] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 02:46:21 (1786085181)
[10620.549140] Lustre: DEBUG MARKER: rename directory return 255
[10665.112194] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 02:47:35 (1786085255)
[10700.878080] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 02:48:10 (1786085290)
[10747.451477] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 02:48:57 (1786085337)
[10865.110703] LustreError: 195168:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[10905.050983] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 02:51:35 (1786085495)
[10944.136221] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 02:52:14 (1786085534)
[10988.477604] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 02:52:58 (1786085578)
[11179.650595] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 02:56:09 (1786085769)
[11181.315492] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[11183.058614] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 02:56:13 (1786085773)
[11249.457615] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[11251.422494] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 02:57:21 (1786085841)
[11287.536697] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[11289.614064] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 02:57:59 (1786085879)
[11345.786401] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[11347.443662] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 02:58:57 (1786085937)
[11399.495775] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 02:59:50 (1786085990)
[11401.759534] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[11403.924599] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 02:59:53 (1786085993)
[11435.493492] LustreError: 208074:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[11483.388114] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 03:01:13 (1786086073)
[11511.804906] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[11513.884383] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[11707.396570] LustreError: 3299:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[11710.408473] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 03:05:00 (1786086300)
[11765.354718] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 03:05:55 (1786086355)
[11802.828246] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 03:06:32 (1786086392)
[11804.778905] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[11806.827822] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 03:06:36 (1786086396)
[11808.678681] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[11810.483832] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 03:06:40 (1786086400)
[11827.122248] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[11845.785932] Lustre: DEBUG MARKER: Write...
[11848.468565] Lustre: DEBUG MARKER: Write out of block quota ...
[11910.736742] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 03:08:20 (1786086500)
[11942.674733] Lustre: DEBUG MARKER: set to use default quota
[11945.460375] Lustre: DEBUG MARKER: set default quota
[11948.193984] Lustre: DEBUG MARKER: get default quota
[11953.847749] Lustre: DEBUG MARKER: Test not out of quota
[11960.312663] Lustre: DEBUG MARKER: Test out of quota
[11975.644702] Lustre: DEBUG MARKER: Increase default quota
[12009.164718] Lustre: DEBUG MARKER: Set quota to override default quota
[12009.229199] LustreError: 187994:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1786691401 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12028.324845] Lustre: DEBUG MARKER: Set to use default quota again
[12053.142532] Lustre: DEBUG MARKER: Cleanup
[12134.432506] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 03:12:04 (1786086724)
[12161.464609] Lustre: DEBUG MARKER: set default quota for qpool1
[12163.099866] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[12214.553973] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 03:13:24 (1786086804)
[12306.691348] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 03:14:56 (1786086896)
[12390.343790] Lustre: DEBUG MARKER: Write...
[12393.765493] Lustre: DEBUG MARKER: Write out of block quota ...
[12525.120751] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 03:18:35 (1786087115)
[12571.860880] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 03:19:22 (1786087162)
[12580.247952] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 03:19:30 (1786087170)
[12582.118399] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate
[12584.288569] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 03:19:34 (1786087174)
[12586.308381] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate
[12588.282802] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 03:19:38 (1786087178)
[12613.405580] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 03:20:02 (1786087202)
[12615.984274] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[12618.702398] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 03:20:08 (1786087208)
[12637.612514] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[12644.259169] LustreError: 233177:0:(qmt_pool.c:1413:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[12644.273228] LustreError: 233177:0:(qmt_pool.c:1413:qmt_pool_recalc()) Skipped 4 previous similar messages
[12649.365387] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.34@tcp (stopping)
[12651.491154] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[12651.506387] Lustre: Skipped 1 previous similar message
[12651.508225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[12651.530137] Lustre: Skipped 1 previous similar message
[12654.296516] LustreError: 233177:0:(qmt_pool.c:1413:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[12654.470921] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.34@tcp (stopping)
[12654.809454] Lustre: server umount lustre-MDT0000 complete
[12663.278677] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[12664.025577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[12664.131902] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[12664.138609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[12669.122497] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[12671.465827] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[12671.479663] Lustre: Skipped 2 previous similar messages
[12712.017929] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 03:21:42 (1786087302)
[12747.350499] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 03:22:17 (1786087337)
[12780.659810] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 03:22:51 (1786087371)
[12834.483760] Lustre: *** cfs_fail_loc=a08, val=0***
[12834.491711] Lustre: Skipped 11 previous similar messages
[12834.502358] Lustre: *** cfs_fail_loc=a08, val=0***
[12834.504628] Lustre: Skipped 9 previous similar messages
[12951.043435] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 03:25:40 (1786087540)
[12968.345624] LustreError: 3302:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-OST0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[13041.632335] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 03:27:11 (1786087631)
[13115.558321] LustreError: 233704:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1786692507 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13267.055275] LustreError: 235337:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1786692659 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13267.139943] LustreError: 233740:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:1 until pending updates are flushed.
[13276.057316] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed