[ 4625.176423] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 4631.633341] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 4642.546565] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4645.171942] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 4647.765223] Lustre: Failing over lustre-MDT0000
[ 4648.169162] Lustre: server umount lustre-MDT0000 complete
[ 4664.799205] Lustre: 3420:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785082439/real 1785082439]  req@ffff8ad5cacf4000 x1871789790966272/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785082455 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4664.801708] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4664.841410] Lustre: 3420:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4664.863295] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4666.335573] Lustre: 14689:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785082441/real 1785082441]  req@ffff8ad6c4c23100 x1871789790966528/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1785082457 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'ll_ost_io00_003.0' uid:0 gid:0 projid:4294967295
[ 4666.376681] Lustre: 14689:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 4668.962083] Lustre: 3421:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785082444/real 1785082444]  req@ffff8ad5c60fa680 x1871789790966784/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785082460 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4672.579617] LDISKFS-fs (dm-0): 5 truncates cleaned up
[ 4672.582877] LDISKFS-fs (dm-0): recovery complete
[ 4672.618755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4674.143177] Lustre: 3421:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785082449/real 1785082449]  req@ffff8ad5c60fb480 x1871789790967296/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785082465 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4674.184317] Lustre: 3421:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 4675.039903] LustreError: 3419:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff8ad6c4c20e00 x1871789790968576/t0(0) o250->MGC192.168.201.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4675.613050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4675.696825] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4675.942893] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 4676.083728] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 4676.163391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:682 to 0x240000400:705)
[ 4676.165115] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:615 to 0x280000400:641)
[ 4681.150642] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4689.891987] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 4692.078595] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4693.670639] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4699.555275] Lustre: DEBUG MARKER: (dd_pid=120159, time=0, timeout=600)
[ 4732.663239] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 4739.548836] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 4748.121259] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4749.968407] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 4751.652109] Lustre: Failing over lustre-MDT0000
[ 4752.075640] Lustre: server umount lustre-MDT0000 complete
[ 4772.833576] Lustre: 3421:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785082547/real 1785082547]  req@ffff8ad6f4c2fb80 x1871789791003264/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785082563 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4772.838056] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4772.858178] Lustre: 3421:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 4772.878029] Lustre: Skipped 2 previous similar messages
[ 4772.888552] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4775.933158] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 4775.934974] LDISKFS-fs (dm-0): recovery complete
[ 4775.942533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4783.548567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4783.587497] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4788.896755] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4794.726554] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 4794.812871] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 4794.877716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:707 to 0x240000400:737)
[ 4794.879995] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:615 to 0x280000400:673)
[ 4797.930976] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 4797.936441] Lustre: Skipped 1 previous similar message
[ 4801.015678] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4802.630662] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4808.543096] Lustre: DEBUG MARKER: (dd_pid=122567, time=0, timeout=600)
[ 4851.493979] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 12:17:21 (1785082641)
[ 4899.601954] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 12:18:09 (1785082689)
[ 4926.686354] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 4938.222321] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 4940.431463] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 4942.360077] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 4944.653412] Lustre: DEBUG MARKER: Set quota for 1 times
[ 4948.393215] Lustre: DEBUG MARKER: Set quota for 2 times
[ 4951.609856] Lustre: DEBUG MARKER: Set quota for 3 times
[ 4955.311834] Lustre: DEBUG MARKER: Set quota for 4 times
[ 4958.861402] Lustre: DEBUG MARKER: Set quota for 5 times
[ 4962.243198] Lustre: DEBUG MARKER: Set quota for 6 times
[ 4965.790850] Lustre: DEBUG MARKER: Set quota for 7 times
[ 4969.140874] Lustre: DEBUG MARKER: Set quota for 8 times
[ 4972.457363] Lustre: DEBUG MARKER: Set quota for 9 times
[ 5016.453844] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 12:20:06 (1785082806)
[ 5032.427481] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5032.434101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5032.454032] Lustre: Skipped 1 previous similar message
[ 5037.193862] Lustre: server umount lustre-MDT0000 complete
[ 5042.717358] LustreError: 8806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785082833 with bad export cookie 6681375943465135306
[ 5042.722241] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5042.739015] LustreError: 8806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 5042.897439] Lustre: server umount lustre-OST0000 complete
[ 5047.591283] Lustre: server umount lustre-OST0001 complete
[ 5066.233643] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local
[ 5077.933586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5078.704822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5084.066570] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5087.174539] Lustre: 139888:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5094.385555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5094.844342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5101.833669] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5103.055354] LustreError: 140242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5103.082887] LustreError: 140242:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages
[ 5103.089923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:740 to 0x240000400:769)
[ 5108.203188] LustreError: 140242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5111.357374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5111.624351] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5116.914259] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:675 to 0x280000400:705)
[ 5119.419129] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5128.422813] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5132.890409] Lustre: 141758:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5157.856149] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5157.879982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5157.888548] Lustre: Skipped 2 previous similar messages
[ 5160.065741] Lustre: server umount lustre-MDT0000 complete
[ 5164.943338] LustreError: 139440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785082956 with bad export cookie 6681375943465142397
[ 5164.956575] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5165.129605] Lustre: server umount lustre-OST0000 complete
[ 5169.124894] Lustre: server umount lustre-OST0001 complete
[ 5186.950388] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local
[ 5199.614963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5200.082859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5204.669661] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5208.486289] Lustre: 144469:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5215.756898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5224.296819] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5225.249374] LustreError: 144822:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5225.262287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:740 to 0x240000400:801)
[ 5230.569129] LustreError: 144822:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5233.303835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5233.579972] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5233.589077] Lustre: Skipped 1 previous similar message
[ 5238.762144] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:675 to 0x280000400:737)
[ 5241.120919] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5249.801494] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5253.923708] Lustre: 146335:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5265.415135] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 12:24:15 (1785083055)
[ 5267.379846] Lustre: DEBUG MARKER: OST0_SIZE: 3605532 required: 6144
[ 5269.132382] Lustre: DEBUG MARKER: run for 4MB test file
[ 5280.859660] Lustre: DEBUG MARKER: User quota (limit: 4 MB)
[ 5287.536775] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 5289.106467] Lustre: DEBUG MARKER: Write half of file
[ 5291.413399] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5292.983151] Lustre: DEBUG MARKER: Step1: done
[ 5294.579356] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 5296.392775] Lustre: DEBUG MARKER: Step2: done
[ 5321.505526] Lustre: DEBUG MARKER: OST0_SIZE: 3605532 required: 61440
[ 5323.277023] Lustre: DEBUG MARKER: run for 40MB test file
[ 5334.292541] Lustre: DEBUG MARKER: User quota (limit: 40 MB)
[ 5340.033291] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 5342.102654] Lustre: DEBUG MARKER: Write half of file
[ 5345.319636] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5348.363865] Lustre: DEBUG MARKER: Step1: done
[ 5349.876666] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 5351.671692] Lustre: DEBUG MARKER: Step2: done
[ 5399.671378] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 12:26:29 (1785083189)
[ 5443.671768] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 12:27:13 (1785083233)
[ 5481.655601] Lustre: DEBUG MARKER: Write...
[ 5484.038228] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5531.628885] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 12:28:41 (1785083321)
[ 5538.620347] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 12:28:48 (1785083328)
[ 5549.389224] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 12:28:59 (1785083339)
[ 5557.966564] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 12:29:08 (1785083348)
[ 5565.878568] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 12:29:16 (1785083356)
[ 5625.529471] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 5755.568956] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 5908.839964] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 12:34:58 (1785083698)
[ 5960.972863] Lustre: DEBUG MARKER: Restart...
[ 5965.802428] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5965.808868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5965.822315] Lustre: Skipped 2 previous similar messages
[ 5965.846191] Lustre: Skipped 1 previous similar message
[ 5970.912736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5970.918759] Lustre: Skipped 1 previous similar message
[ 5971.457714] Lustre: server umount lustre-MDT0000 complete
[ 5975.459875] LustreError: 144025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785083766 with bad export cookie 6681375943465143930
[ 5975.475260] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5975.476328] LustreError: 144025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 5975.629882] Lustre: server umount lustre-OST0000 complete
[ 5979.661184] Lustre: server umount lustre-OST0001 complete
[ 5996.608445] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local
[ 6008.696142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6009.357843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6015.083814] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6017.862602] Lustre: 169135:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6025.195310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6025.499708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6032.855589] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6034.743786] LustreError: 169486:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6034.766140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:813 to 0x240000400:833)
[ 6040.033037] LustreError: 169486:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6042.901913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6043.103873] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6048.248157] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:745 to 0x280000400:769)
[ 6050.116546] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6058.750074] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6062.993599] Lustre: 171001:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6128.783970] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 12:38:38 (1785083918)
[ 6183.504822] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 12:39:33 (1785083973)
[ 7863.197654] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 13:07:33 (1785085653)
[ 7875.480500] Lustre: server umount lustre-MDT0000 complete
[ 7879.770050] LustreError: 168691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785085670 with bad export cookie 6681375943465152036
[ 7879.779540] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7889.638777] Lustre: server umount lustre-OST0000 complete
[ 7892.447872] Lustre: 3422:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785085667/real 1785085667]  req@ffff8ad6fe352680 x1871789794370560/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785085683 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7892.488264] Lustre: 3422:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 7892.491430] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7893.942661] LustreError: 3422:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0001 qtype:usr lqe: ffff8ad6c3742000 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 1460 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 7894.422660] Lustre: server umount lustre-OST0001 complete
[ 7911.331259] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local
[ 7924.303817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7924.753471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7930.032930] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7933.011182] Lustre: 178609:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7940.934527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 7941.607182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7948.948965] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7949.798314] LustreError: 178961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7949.841879] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5835 to 0x240000400:5857)
[ 7954.920287] LustreError: 179362:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7958.942609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 7959.157687] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7964.656994] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5771 to 0x280000400:5793)
[ 7966.472835] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7973.928561] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7978.237703] Lustre: 180475:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 8008.574623] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 13:09:58 (1785085798)
[ 8043.101101] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 13:10:32 (1785085832)
[ 8079.025067] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 13:11:08 (1785085868)
[ 8080.892307] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 8082.838677] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 13:11:12 (1785085872)
[ 8084.813870] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 8086.802671] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 13:11:16 (1785085876)
[ 8172.797841] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 13:12:42 (1785085962)
[ 8210.756830] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 13:13:20 (1785086000)
[ 8290.076272] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 13:14:39 (1785086079)
[ 8297.031483] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8297.033892] Lustre: Skipped 2 previous similar messages
[ 8299.088326] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8299.095382] Lustre: Skipped 97 previous similar messages
[ 8303.103144] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8303.104536] Lustre: Skipped 187 previous similar messages
[ 8311.136451] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8311.140050] Lustre: Skipped 505 previous similar messages
[ 8327.169390] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8327.178447] Lustre: Skipped 843 previous similar messages
[ 8359.216753] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8359.219932] Lustre: Skipped 1647 previous similar messages
[ 8423.286845] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8423.288220] Lustre: Skipped 2531 previous similar messages
[ 8848.352472] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8848.370381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 8848.375729] Lustre: Skipped 1 previous similar message
[ 8851.051329] Lustre: server umount lustre-MDT0000 complete
[ 8855.432970] LustreError: 183819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785086646 with bad export cookie 6681375943466928734
[ 8855.461905] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8855.987953] Lustre: server umount lustre-OST0000 complete
[ 8860.170570] Lustre: server umount lustre-OST0001 complete
[ 8867.253458] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid
[ 8876.705397] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local
[ 8918.164226] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local
[ 8927.918661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8928.212561] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 8928.241852] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 8928.353710] Lustre: lustre-MDT0000: new disk, initializing
[ 8928.411948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8928.426960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 8932.616988] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8937.479481] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 8944.072790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 8944.440325] Lustre: lustre-OST0000: new disk, initializing
[ 8944.447863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 8944.462168] Lustre: Skipped 1 previous similar message
[ 8944.469829] Lustre: 195374:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0]
[ 8944.568934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 8945.807239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[ 8945.813773] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[ 8945.850830] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[ 8950.920067] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8962.032124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 8962.161448] Lustre: lustre-OST0001: new disk, initializing
[ 8962.164726] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 8962.169389] Lustre: 196236:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0]
[ 8962.211788] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 8963.604918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[ 8963.616690] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[ 8963.647647] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[ 8968.430572] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8978.625430] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8983.445238] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 9012.626899] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 13:26:42 (1785086802)
[ 9021.823160] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 13:26:51 (1785086811)
[ 9055.579489] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 13:27:25 (1785086845)
[ 9106.361346] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 13:28:16 (1785086896)
[ 9129.594596] Lustre: DEBUG MARKER: rename directory return 255
[ 9175.022606] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 13:29:24 (1785086964)
[ 9200.362655] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 13:29:50 (1785086990)
[ 9233.026989] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 13:30:23 (1785087023)
[ 9376.194856] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 13:32:46 (1785087166)
[ 9404.945181] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 13:33:15 (1785087195)
[ 9441.598826] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 13:33:51 (1785087231)
[ 9562.245262] LustreError: 194528:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:1001 enforced:1 hard:204800 soft:0 granted:204800 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 9589.105740] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 13:36:18 (1785087378)
[ 9598.433688] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9598.444704] Lustre: Skipped 1 previous similar message
[ 9598.450134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9598.464733] Lustre: Skipped 1 previous similar message
[ 9600.309315] Lustre: server umount lustre-MDT0000 complete
[ 9604.840841] LustreError: 195373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785087396 with bad export cookie 6681375943467295548
[ 9604.842959] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9604.849723] LustreError: 195373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 9605.030262] Lustre: server umount lustre-OST0000 complete
[ 9608.937226] Lustre: server umount lustre-OST0001 complete
[ 9633.849863] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local
[ 9646.585845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9647.212363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9647.221356] LustreError: lustre-MDT0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 9652.142094] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9656.359758] Lustre: 213435:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9664.827026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 9665.328818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9665.343791] LustreError: lustre-OST0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 9672.178916] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9672.502795] LustreError: 213789:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9672.533396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:20 to 0x240000400:65)
[ 9677.792777] LustreError: 213789:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9681.741382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 9681.927431] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9681.934710] LustreError: lustre-OST0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 9687.028303] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:20 to 0x280000400:65)
[ 9688.633338] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9697.099579] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9700.518723] Lustre: 215304:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9706.816840] LustreError: 213004:0:(osd_handler.c:3426:osd_quota_transfer()) lustre-MDT0000: quota transfer failed. Is project enforcement enabled on the ldiskfs filesystem? rc = -95
[ 9710.805532] Lustre: server umount lustre-MDT0000 complete
[ 9714.569503] LustreError: 212991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785087505 with bad export cookie 6681375943467343680
[ 9714.593222] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9723.950799] Lustre: server umount lustre-OST0000 complete
[ 9737.946249] Lustre: server umount lustre-OST0001 complete
[ 9755.208433] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local
[ 9766.310569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9766.952363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9771.656123] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9774.393352] Lustre: 217811:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9781.331863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 9781.778927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9788.238933] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9792.032906] LustreError: 218164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9792.073896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:20 to 0x240000400:97)
[ 9797.090505] LustreError: 218699:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9797.589640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 9797.810403] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9803.239372] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97)
[ 9805.458729] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9813.760537] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9818.560712] Lustre: 219681:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9844.915772] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 13:40:34 (1785087634)
[ 9890.950520] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[ 9892.554080] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 13:41:22 (1785087682)
[ 9913.793969] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[ 9915.406534] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 13:41:45 (1785087705)
[ 9952.118747] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[ 9953.946701] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 13:42:24 (1785087744)
[ 9988.952053] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 13:42:58 (1785087778)
[10003.168259] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[10010.827502] Lustre: 226187:0:(qsd_reint.c:249:qsd_reint_index()) lustre-OST0000: index version for fid [0x200000005:0x1004:0x0] is 0, but index isn't empty (1)
[10015.355572] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[10022.063734] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[10025.523634] Lustre: DEBUG MARKER: Write...
[10042.228903] LustreError: 227430:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[10047.230154] Lustre: DEBUG MARKER: Write...
[10058.401189] Lustre: DEBUG MARKER: Write...
[10123.460906] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 13:45:13 (1785087913)
[10148.378521] LustreError: 231025:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[10190.329929] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 13:46:20 (1785087980)
[10211.935585] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[10213.566650] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[10289.219063] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 13:47:59 (1785088079)
[10327.442881] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 13:48:37 (1785088117)
[10360.990273] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 13:49:10 (1785088150)
[10376.511561] Lustre: DEBUG MARKER: User quota (block hardlimit:100 MB)
[10397.741702] Lustre: DEBUG MARKER: Write...
[10400.614786] Lustre: DEBUG MARKER: Write out of block quota ...
[10496.241368] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 13:51:26 (1785088286)
[10509.840274] Lustre: DEBUG MARKER: User quota (block hardlimit:1000 MB)
[10592.254815] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 13:53:02 (1785088382)
[10605.003732] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[10623.096242] Lustre: DEBUG MARKER: Write...
[10625.190402] Lustre: DEBUG MARKER: Write out of block quota ...
[10625.609788] LustreError: 217383:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:10240 soft:0 granted:10240 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[10680.337579] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 13:54:30 (1785088470)
[10705.972147] Lustre: DEBUG MARKER: set to use default quota
[10707.959838] Lustre: DEBUG MARKER: set default quota
[10709.877567] Lustre: DEBUG MARKER: get default quota
[10716.115597] Lustre: DEBUG MARKER: Test not out of quota
[10720.082558] Lustre: DEBUG MARKER: Test out of quota
[10731.331324] Lustre: DEBUG MARKER: Increase default quota
[10757.944984] Lustre: DEBUG MARKER: Set quota to override default quota
[10758.033540] LustreError: 218186:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45056 time:1785693349 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[10772.402343] Lustre: DEBUG MARKER: Set to use default quota again
[10793.887953] Lustre: DEBUG MARKER: Cleanup
[10850.031543] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 13:57:19 (1785088639)
[10871.903152] Lustre: DEBUG MARKER: set default quota for qpool1
[10873.959964] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[10925.902047] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 13:58:35 (1785088715)
[11007.279280] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 13:59:57 (1785088797)
[11071.822169] Lustre: DEBUG MARKER: Write...
[11075.400072] Lustre: DEBUG MARKER: Write out of block quota ...
[11190.566508] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 14:03:00 (1785088980)
[11225.671770] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 14:03:35 (1785089015)
[11233.192695] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 14:03:43 (1785089023)
[11273.506559] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 14:04:23 (1785089063)
[11320.492405] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 14:05:10 (1785089110)
[11342.563864] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 14:05:32 (1785089132)
[11372.674578] Lustre: *** cfs_fail_loc=a06, val=0***
[11372.677943] Lustre: Skipped 2183 previous similar messages
[11373.189678] LustreError: 217399:0:(qmt_lock.c:479:qmt_lvbo_update()) $$$ failed to release quota space on glimpse 0!=1024 : rc = -11
                 qmt:lustre-QMT0000 pool:dt-0x0 id:60000 enforced:1 hard:102400 soft:0 granted:24576 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no
[11378.775354] Lustre: Failing over lustre-OST0001
[11378.913638] Lustre: server umount lustre-OST0001 complete
[11379.181174] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107
[11379.194954] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[11379.214863] Lustre: Skipped 1 previous similar message
[11379.228566] LustreError: 218164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[11380.116866] LustreError: 218699:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[11383.264743] LustreError: 218163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[11388.403213] LustreError: 218165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[11388.425290] LustreError: 218165:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message
[11392.138275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[11392.501996] Lustre: lustre-OST0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[11392.562267] Lustre: lustre-OST0001: in recovery but waiting for the first client to connect
[11393.928949] Lustre: lustre-OST0001: Will be in recovery for at least 1:00, or until 2 clients reconnect
[11394.199876] Lustre: lustre-OST0001: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted.
[11394.206238] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo)
[11394.222134] Lustre: Skipped 1 previous similar message
[11400.665093] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11452.147633] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 14:07:21 (1785089241)
[11467.027715] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[11474.323914] LustreError: 266748:0:(qmt_pool.c:1413:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[11474.330092] LustreError: 266748:0:(qmt_pool.c:1413:qmt_pool_recalc()) Skipped 5 previous similar messages
[11477.368891] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.9@tcp (stopping)
[11477.985450] LustreError: 3422:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff8ad5cee40000 x1871789800419840/t0(0) o601->lustre-MDT0000-lwp-MDT0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[11478.004910] LustreError: 3422:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-MDT0000 qtype:usr lqe: ffff8ad5c3a8ac00 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 312 qunit:0 qtune:0 edquot:0 default:no revoke:0
[11478.038214] LustreError: 3422:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 2 previous similar messages
[11479.520983] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[11479.553889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[11482.470170] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.9@tcp (stopping)
[11482.478193] Lustre: Skipped 1 previous similar message
[11484.351536] LustreError: 266748:0:(qmt_pool.c:1413:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[11484.640534] LustreError: 218186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[11484.656029] LustreError: 218186:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages
[11484.761571] Lustre: server umount lustre-MDT0000 complete
[11495.101913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[11495.325792] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[11495.650780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[11495.684958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:112 to 0x240000400:129)
[11495.691117] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:113 to 0x280000400:129)
[11500.783878] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11509.229177] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[11539.528596] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 14:08:49 (1785089329)
[11567.854334] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 14:09:17 (1785089357)
[11594.034924] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 14:09:43 (1785089383)
[11641.542945] Lustre: *** cfs_fail_loc=a08, val=0***
[11641.545726] Lustre: Skipped 8 previous similar messages
[11641.558367] Lustre: *** cfs_fail_loc=a08, val=0***
[11641.565634] Lustre: Skipped 1 previous similar message
[11735.203423] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 14:12:04 (1785089524)
[11763.205958] LustreError: 267310:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:3072 soft:0 granted:3072 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11814.862864] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 14:13:25 (1785089605)
[11874.371284] LustreError: 267307:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1785694465 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12025.560026] LustreError: 267308:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1785694616 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12168.398877] LustreError: 267309:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:1000 enforced:1 hard:2048 soft:2048 granted:65536 time:1785694759 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12177.982067] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed