[ 4896.648967] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 4902.917931] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 4912.505409] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4914.490537] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 4916.985262] Lustre: Failing over lustre-MDT0000
[ 4917.168666] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.57@tcp (stopping)
[ 4917.299089] LustreError: 125213:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup
[ 4917.304614] LustreError: 125213:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages
[ 4917.407173] Lustre: server umount lustre-MDT0000 complete
[ 4917.732642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4917.742337] Lustre: Skipped 4 previous similar messages
[ 4934.112914] Lustre: 7025:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767320016/real 1767320016]  req@ffff925923f5ca80 x1853164350227072/t0(0) o400->MGC192.168.203.157@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767320032 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4934.143125] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4948.660929] LDISKFS-fs (dm-0): 7 truncates cleaned up
[ 4948.671178] LDISKFS-fs (dm-0): recovery complete
[ 4948.684626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4959.716057] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa4cc7d2270be06cf
[ 4959.740466] Lustre: MGC192.168.203.157@tcp: Connection restored to 0@lo (at 0@lo)
[ 4959.748914] Lustre: Skipped 1 previous similar message
[ 4960.077913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4960.122253] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4960.733766] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 4965.451094] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4965.469642] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted.
[ 4965.509157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:660 to 0x280000400:705)
[ 4965.509271] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:593 to 0x2c0000400:609)
[ 4973.243884] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4974.970230] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4980.653404] Lustre: DEBUG MARKER: (dd_pid=107941, time=0, timeout=600)
[ 5008.231275] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5013.858276] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5022.955940] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5024.982278] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5027.365048] Lustre: Failing over lustre-MDT0000
[ 5027.579632] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.57@tcp (stopping)
[ 5027.581541] LustreError: 128220:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup
[ 5027.595768] LustreError: 128220:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 5027.876480] Lustre: server umount lustre-MDT0000 complete
[ 5031.907986] LustreError: 15477:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5031.932173] LustreError: 15477:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 64 previous similar messages
[ 5048.287555] Lustre: 7027:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767320130/real 1767320130]  req@ffff92580fd45180 x1853164350294784/t0(0) o400->MGC192.168.203.157@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767320146 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5048.315348] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5058.670041] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 5058.677415] LDISKFS-fs (dm-0): recovery complete
[ 5058.693753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5073.922534] LustreError: 7023:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff925808147800 x1853164350307584/t0(0) o250->MGC192.168.203.157@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5074.279704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5074.349795] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5075.320314] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 5079.523617] LustreError: 7023:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-lwp-MDT0001: namespace resource [0x200000006:0x10000:0x0].0x0 (ffff925806c17900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5079.548101] LustreError: 7023:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 5 previous similar messages
[ 5079.627425] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 5079.676132] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:707 to 0x280000400:737)
[ 5079.685315] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:593 to 0x2c0000400:641)
[ 5079.906620] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5088.492965] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5090.936506] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5096.301705] Lustre: DEBUG MARKER: (dd_pid=110190, time=0, timeout=600)
[ 5131.829855] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 21:17:08 (1767320228)
[ 5181.510869] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 21:17:58 (1767320278)
[ 5204.949223] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 5217.034959] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 5219.329900] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 5221.881305] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 5224.790373] Lustre: DEBUG MARKER: Set quota for 1 times
[ 5228.104891] Lustre: DEBUG MARKER: Set quota for 2 times
[ 5231.957725] Lustre: DEBUG MARKER: Set quota for 3 times
[ 5235.479977] Lustre: DEBUG MARKER: Set quota for 4 times
[ 5239.040589] Lustre: DEBUG MARKER: Set quota for 5 times
[ 5242.475957] Lustre: DEBUG MARKER: Set quota for 6 times
[ 5245.968075] Lustre: DEBUG MARKER: Set quota for 7 times
[ 5249.127055] Lustre: DEBUG MARKER: Set quota for 8 times
[ 5252.597476] Lustre: DEBUG MARKER: Set quota for 9 times
[ 5297.370918] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 21:19:54 (1767320394)
[ 5318.114413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5318.131311] Lustre: Skipped 3 previous similar messages
[ 5321.724041] LustreError: 135046:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup
[ 5321.741477] LustreError: 135046:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 5322.234987] Lustre: server umount lustre-MDT0000 complete
[ 5323.691051] Lustre: server umount lustre-MDT0001 complete
[ 5329.011288] LustreError: 106051:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767320427 with bad export cookie 11875003904343019764
[ 5329.023562] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5329.031174] LustreError: 106051:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 5329.404133] Lustre: server umount lustre-OST0000 complete
[ 5329.957163] Lustre: server umount lustre-OST0001 complete
[ 5354.598235] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local
[ 5374.686938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5374.873367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 5375.344975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5380.144755] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5380.563993] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5387.156942] Lustre: 139036:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5387.783523] Lustre: 139077:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5401.246371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 5401.589987] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 5401.877945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5401.884767] Lustre: Skipped 1 previous similar message
[ 5410.899402] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5411.100776] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5420.667233] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:740 to 0x280000400:769)
[ 5420.823198] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:643 to 0x2c0000400:673)
[ 5420.823287] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:97)
[ 5431.386534] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5435.653575] Lustre: 141432:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5461.992684] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5462.005176] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 5462.016112] Lustre: Skipped 10 previous similar messages
[ 5462.031884] Lustre: Skipped 3 previous similar messages
[ 5467.111245] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 5467.122211] Lustre: Skipped 3 previous similar messages
[ 5467.273516] LustreError: 142378:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup
[ 5467.279987] LustreError: 142378:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages
[ 5467.443985] Lustre: server umount lustre-MDT0000 complete
[ 5468.394272] Lustre: server umount lustre-MDT0001 complete
[ 5473.428451] LustreError: 137936:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767320572 with bad export cookie 11875003904343026820
[ 5473.455156] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5473.455881] LustreError: 137936:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 5473.601824] Lustre: server umount lustre-OST0001 complete
[ 5474.037846] Lustre: server umount lustre-OST0000 complete
[ 5497.755444] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local
[ 5516.234460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 5516.339824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5516.926044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 5516.931559] Lustre: Skipped 1 previous similar message
[ 5521.928068] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5522.571577] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5528.537510] Lustre: 146382:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5541.785256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 5541.813226] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 5542.303117] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5542.313342] Lustre: Skipped 1 previous similar message
[ 5543.793417] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:740 to 0x280000400:801)
[ 5543.817475] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:643 to 0x2c0000400:705)
[ 5549.343412] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5549.720860] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5562.182567] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:129)
[ 5568.017544] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5572.018581] Lustre: 148768:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5572.024934] Lustre: 148768:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message
[ 5588.248416] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 21:24:44 (1767320684)
[ 5589.946698] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 6144
[ 5591.385249] Lustre: DEBUG MARKER: run for 4MB test file
[ 5602.326342] Lustre: DEBUG MARKER: User quota (limit: 4 MB)
[ 5607.457717] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 5609.411396] Lustre: DEBUG MARKER: Write half of file
[ 5611.477820] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5613.608616] Lustre: DEBUG MARKER: Step1: done
[ 5615.399605] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 5617.087660] Lustre: DEBUG MARKER: Step2: done
[ 5638.640286] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 61440
[ 5640.503953] Lustre: DEBUG MARKER: run for 40MB test file
[ 5651.772158] Lustre: DEBUG MARKER: User quota (limit: 40 MB)
[ 5656.451222] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 5658.046803] Lustre: DEBUG MARKER: Write half of file
[ 5661.209614] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5663.662496] Lustre: DEBUG MARKER: Step1: done
[ 5664.947884] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 5666.525769] Lustre: DEBUG MARKER: Step2: done
[ 5704.139989] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 21:26:41 (1767320801)
[ 5740.617531] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 21:27:17 (1767320837)
[ 5777.860253] Lustre: DEBUG MARKER: Write...
[ 5780.694373] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5827.095603] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 21:28:44 (1767320924)
[ 5832.806605] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 21:28:49 (1767320929)
[ 5841.172447] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 21:28:57 (1767320937)
[ 5849.377210] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 21:29:06 (1767320946)
[ 5856.868463] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 21:29:13 (1767320953)
[ 5903.579051] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6000.646348] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 6107.956440] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 21:33:24 (1767321204)
[ 6143.306860] Lustre: DEBUG MARKER: Restart...
[ 6151.141933] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6151.146343] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 6151.161578] Lustre: Skipped 3 previous similar messages
[ 6151.176574] Lustre: Skipped 3 previous similar messages
[ 6155.719200] LustreError: 164915:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 6155.734745] LustreError: 164915:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages
[ 6155.988965] Lustre: server umount lustre-MDT0001 complete
[ 6156.264580] LustreError: 145300:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6156.264958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6156.286804] LustreError: 145300:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 53 previous similar messages
[ 6156.305813] Lustre: Skipped 1 previous similar message
[ 6157.109374] Lustre: server umount lustre-MDT0000 complete
[ 6161.223187] LustreError: 145293:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767321260 with bad export cookie 11875003904343029102
[ 6161.226772] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6161.241310] LustreError: 145293:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 6161.298948] LustreError: 165397:0:(obd_class.h:479:obd_check_dev()) Device 24 not setup
[ 6161.305340] LustreError: 165397:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages
[ 6161.374098] Lustre: server umount lustre-OST0001 complete
[ 6161.687962] Lustre: server umount lustre-OST0000 complete
[ 6182.713031] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local
[ 6199.070268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 6199.389920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6199.531348] LustreError: 7023:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff9259403cca80 x1853164350988800/t0(0) o250->MGC192.168.203.157@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6200.070240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 6200.079189] Lustre: Skipped 1 previous similar message
[ 6204.600900] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6204.641364] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6211.097305] Lustre: 168941:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6220.764586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 6221.015915] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 6221.218790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6221.224743] Lustre: Skipped 1 previous similar message
[ 6224.624087] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:161)
[ 6225.671828] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:713 to 0x2c0000400:737)
[ 6225.675532] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:813 to 0x280000400:833)
[ 6227.961288] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6228.217937] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6236.797504] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6240.126819] Lustre: 171279:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6240.140688] Lustre: 171279:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message
[ 6289.240862] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 21:36:26 (1767321386)
[ 6332.066321] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 21:37:08 (1767321428)
[ 7802.911617] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 22:01:40 (1767322900)
[ 7813.088781] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7813.089305] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 7813.104977] Lustre: Skipped 4 previous similar messages
[ 7813.111915] Lustre: Skipped 3 previous similar messages
[ 7814.976416] LustreError: 175788:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup
[ 7814.981688] LustreError: 175788:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages
[ 7815.096365] Lustre: server umount lustre-MDT0000 complete
[ 7815.408244] Lustre: server umount lustre-MDT0001 complete
[ 7818.316209] LustreError: 167808:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767322917 with bad export cookie 11875003904343038146
[ 7818.321350] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7818.324368] LustreError: 167808:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 7818.400142] LustreError: 176272:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup
[ 7818.407149] LustreError: 176272:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages
[ 7818.523342] Lustre: server umount lustre-OST0001 complete
[ 7818.685168] Lustre: server umount lustre-OST0000 complete
[ 7834.745989] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local
[ 7847.620817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 7847.693655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7847.978634] LustreError: 178709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7847.990706] LustreError: 178709:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages
[ 7848.077967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7848.081906] Lustre: Skipped 1 previous similar message
[ 7851.545281] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7851.747870] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7855.636224] Lustre: 179774:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7864.238162] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 7864.249995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 7864.495151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7864.499277] Lustre: Skipped 1 previous similar message
[ 7869.169259] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7869.191381] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7872.879863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5835 to 0x280000400:5857)
[ 7872.894180] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:5739 to 0x2c0000400:5761)
[ 7873.899547] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:193)
[ 7880.296781] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7882.233298] Lustre: 182142:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7882.238888] Lustre: 182142:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message
[ 7903.873510] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 22:03:21 (1767323001)
[ 7919.881257] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 22:03:37 (1767323017)
[ 7935.481666] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 22:03:52 (1767323032)
[ 7951.538727] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 22:04:08 (1767323048)
[ 7977.242848] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 22:04:34 (1767323074)
[ 8033.880752] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 22:05:30 (1767323130)
[ 8056.715688] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 22:05:53 (1767323153)
[ 8109.393596] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 22:06:46 (1767323206)
[ 8113.194864] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8113.199954] Lustre: Skipped 2 previous similar messages
[ 8115.216658] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8115.219773] Lustre: Skipped 115 previous similar messages
[ 8119.217972] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8119.221046] Lustre: Skipped 191 previous similar messages
[ 8127.227943] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8127.230016] Lustre: Skipped 499 previous similar messages
[ 8143.229896] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8143.232870] Lustre: Skipped 924 previous similar messages
[ 8175.278746] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8175.280121] Lustre: Skipped 2022 previous similar messages
[ 8239.316882] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8239.318711] Lustre: Skipped 3647 previous similar messages
[ 8493.542182] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8493.554566] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 8493.554583] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107
[ 8498.655889] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 8498.658371] Lustre: Skipped 5 previous similar messages
[ 8499.662593] LustreError: 192699:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup
[ 8499.665417] LustreError: 192699:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages
[ 8499.744456] Lustre: server umount lustre-MDT0001 complete
[ 8500.499757] LustreError: 192701:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup
[ 8500.502677] LustreError: 192701:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 8500.641696] Lustre: server umount lustre-MDT0000 complete
[ 8507.972588] LustreError: 179776:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323606 with bad export cookie 11875003904344813500
[ 8507.990906] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8508.342190] LustreError: 193700:0:(obd_class.h:479:obd_check_dev()) Device 24 not setup
[ 8508.344347] LustreError: 193700:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 8508.385204] Lustre: server umount lustre-OST0000 complete
[ 8508.454676] Lustre: server umount lustre-OST0001 complete
[ 8519.507930] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid
[ 8524.549995] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local
[ 8534.699967] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 8534.919036] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 8536.399782] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 8537.899206] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 8546.162863] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local
[ 8557.153437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 8557.208146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 8557.308423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 8557.376260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8559.467793] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001
[ 8559.495091] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
[ 8559.498171] Lustre: Skipped 1 previous similar message
[ 8559.549398] Lustre: lustre-MDT0001: new disk, initializing
[ 8559.551532] Lustre: Skipped 1 previous similar message
[ 8559.586453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 8559.592909] Lustre: Skipped 1 previous similar message
[ 8559.603305] Lustre: 200140:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115
[ 8559.612647] LustreError: 200140:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115
[ 8559.656606] Lustre: 200123:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17
[ 8559.827803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 8559.835924] Lustre: Skipped 1 previous similar message
[ 8560.883523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt
[ 8560.889876] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt]
[ 8560.892518] Lustre: Skipped 1 previous similar message
[ 8562.424766] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8562.723518] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8568.062696] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 8568.233316] Lustre: 201329:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 8576.347507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 8576.399866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 8576.490831] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 8576.531145] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 8576.542389] Lustre: lustre-OST0000: new disk, initializing
[ 8576.544770] Lustre: Skipped 1 previous similar message
[ 8576.551027] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 8576.556910] Lustre: Skipped 2 previous similar messages
[ 8576.583102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 8576.585585] Lustre: Skipped 1 previous similar message
[ 8578.034922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost
[ 8578.046744] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost]
[ 8578.057018] Lustre: Skipped 1 previous similar message
[ 8578.072601] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000400
[ 8578.072940] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400
[ 8580.054112] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8580.064381] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8586.637234] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8588.604775] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 8605.116646] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 22:15:02 (1767323702)
[ 8608.918963] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 22:15:06 (1767323706)
[ 8629.238446] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 22:15:26 (1767323726)
[ 8653.885688] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 22:15:51 (1767323751)
[ 8662.443306] Lustre: DEBUG MARKER: rename directory return 255
[ 8685.422522] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 22:16:23 (1767323783)
[ 8694.621428] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 22:16:32 (1767323792)
[ 8711.131943] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 22:16:48 (1767323808)
[ 8745.981452] LustreError: 200103:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 8761.077375] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 22:17:38 (1767323858)
[ 8772.167465] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 22:17:49 (1767323869)
[ 8788.737403] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 22:18:06 (1767323886)
[ 8849.088300] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 22:19:06 (1767323946)
[ 8854.496309] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8854.497075] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 8854.501987] Lustre: Skipped 7 previous similar messages
[ 8854.504193] Lustre: Skipped 2 previous similar messages
[ 8856.901467] LustreError: 215585:0:(obd_class.h:479:obd_check_dev()) Device 26 not setup
[ 8856.903746] LustreError: 215585:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages
[ 8856.942347] Lustre: server umount lustre-MDT0000 complete
[ 8857.033642] Lustre: server umount lustre-MDT0001 complete
[ 8858.604085] LustreError: 200088:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323957 with bad export cookie 11875003904345185585
[ 8858.606983] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8858.609776] LustreError: 200088:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 8858.685971] Lustre: server umount lustre-OST0000 complete
[ 8858.753212] Lustre: server umount lustre-OST0001 complete
[ 8862.713031] Lustre: DEBUG MARKER: sanity-quota test_59: @@@@@@ FAIL: tune2fs /dev/vdc failed
[ 8866.600420] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 22:19:24 (1767323964)
[ 8868.437231] Lustre: DEBUG MARKER: sanity-quota test_60: @@@@@@ FAIL: setup quota failed with 1