[ 6036.572284] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 6040.985475] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 6043.383138] LustreError: 124832:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 6044.217557] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6045.537469] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 6046.918639] Lustre: Failing over lustre-MDT0000
[ 6047.163652] Lustre: server umount lustre-MDT0000 complete
[ 6064.101678] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6064.326611] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6064.524525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6064.571684] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 6066.399136] Lustre: 3307:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788335697/real 1788335697]  req@ffff9a4cc7b29500 x1875199630080000/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788335713 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6066.644822] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 6066.696182] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 6066.729241] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:584 to 0x280000400:609)
[ 6066.734139] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:587 to 0x240000400:609)
[ 6067.174384] Lustre: 3306:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788335697/real 1788335697]  req@ffff9a4cc7b28a80 x1875199630079872/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788335713 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6067.630156] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6069.731193] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 6071.776091] Lustre: 3306:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788335702/real 1788335702]  req@ffff9a4cf58f3480 x1875199630080384/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788335718 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6074.493546] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6075.463928] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6077.023181] Lustre: 3309:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788335707/real 1788335707]  req@ffff9a4cf58f0700 x1875199630080640/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788335723 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6077.043159] Lustre: 3309:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 6079.252502] Lustre: DEBUG MARKER: (dd_pid=117274, time=0, timeout=600)
[ 6122.931272] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 6127.871865] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 6130.481779] LustreError: 127887:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 6131.252608] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6132.612858] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 6134.233984] Lustre: Failing over lustre-MDT0000
[ 6134.537147] Lustre: server umount lustre-MDT0000 complete
[ 6150.624086] Lustre: 3309:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788335781/real 1788335781]  req@ffff9a4cff553800 x1875199630112768/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1788335797 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 6150.646961] Lustre: 3309:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 6150.656555] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6150.670421] Lustre: Skipped 1 previous similar message
[ 6151.066506] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6151.498842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6151.598611] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 6153.691489] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 6153.870286] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 6153.901908] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:584 to 0x280000400:641)
[ 6153.910552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:611 to 0x240000400:641)
[ 6155.146834] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6155.427824] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 6155.431029] Lustre: Skipped 1 previous similar message
[ 6162.742942] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6162.912436] Lustre: 3306:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788335793/real 1788335793]  req@ffff9a4cd8a2dc00 x1875199630113920/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788335809 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6162.932777] Lustre: 3306:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 6163.903232] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6176.531184] Lustre: DEBUG MARKER: (dd_pid=119834, time=8, timeout=600)
[ 6228.880452] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 03:57:54 (1788335874)
[ 6247.316323] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6248.503451] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6253.235299] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6254.220442] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6287.614449] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 03:58:53 (1788335933)
[ 6306.145417] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 6318.824844] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 6319.924709] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 6320.995429] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 6322.295892] Lustre: DEBUG MARKER: Set quota for 1 times
[ 6324.976986] Lustre: DEBUG MARKER: Set quota for 2 times
[ 6327.941153] Lustre: DEBUG MARKER: Set quota for 3 times
[ 6330.602739] Lustre: DEBUG MARKER: Set quota for 4 times
[ 6333.376812] Lustre: DEBUG MARKER: Set quota for 5 times
[ 6336.331673] Lustre: DEBUG MARKER: Set quota for 6 times
[ 6339.143321] Lustre: DEBUG MARKER: Set quota for 7 times
[ 6341.987908] Lustre: DEBUG MARKER: Set quota for 8 times
[ 6344.802613] Lustre: DEBUG MARKER: Set quota for 9 times
[ 6347.667759] Lustre: DEBUG MARKER: Set quota for 10 times
[ 6350.605976] Lustre: DEBUG MARKER: Set quota for 11 times
[ 6384.064837] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 04:00:29 (1788336029)
[ 6393.351504] Lustre: server umount lustre-MDT0000 complete
[ 6396.100262] LustreError: 14038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788336042 with bad export cookie 10317984416633779945
[ 6396.108060] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6405.702323] Lustre: server umount lustre-OST0000 complete
[ 6419.115744] Lustre: server umount lustre-OST0001 complete
[ 6429.503813] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local
[ 6436.386927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6439.398439] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6441.381482] Lustre: 138147:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6445.875086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6446.881946] LustreError: 138521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6446.917210] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:643 to 0x240000400:673)
[ 6450.520871] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6452.193650] LustreError: 138521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6456.409777] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6460.334864] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6461.431334] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:644 to 0x280000400:673)
[ 6465.404792] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6467.504685] Lustre: 140066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6481.889127] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6481.894795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6481.899796] Lustre: Skipped 2 previous similar messages
[ 6481.910907] Lustre: Skipped 2 previous similar messages
[ 6482.980272] Lustre: server umount lustre-MDT0000 complete
[ 6485.478569] LustreError: 137696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788336132 with bad export cookie 10317984416633793322
[ 6485.487978] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6485.544512] Lustre: server umount lustre-OST0000 complete
[ 6488.063334] Lustre: server umount lustre-OST0001 complete
[ 6496.865702] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local
[ 6503.305112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6506.436102] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6508.599344] Lustre: 142808:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6512.610137] LustreError: 143176:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6517.133300] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6518.258850] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:643 to 0x240000400:705)
[ 6522.429344] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6522.432343] Lustre: Skipped 1 previous similar message
[ 6526.182273] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6527.475186] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:644 to 0x280000400:705)
[ 6531.156790] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6533.457720] Lustre: 144723:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6540.469976] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 04:03:06 (1788336186)
[ 6541.579674] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6542.744960] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 04:03:08 (1788336188)
[ 6586.506266] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 04:03:52 (1788336232)
[ 6622.998138] Lustre: DEBUG MARKER: Write...
[ 6624.641247] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6667.937807] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 04:05:13 (1788336313)
[ 6671.811195] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 04:05:17 (1788336317)
[ 6678.597085] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 04:05:24 (1788336324)
[ 6684.395702] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 04:05:30 (1788336330)
[ 6689.153680] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 04:05:35 (1788336335)
[ 6740.663527] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6841.858378] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 6991.224637] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 04:10:37 (1788336637)
[ 7039.971687] Lustre: DEBUG MARKER: Restart...
[ 7044.577565] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7044.578409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7044.585857] Lustre: Skipped 1 previous similar message
[ 7044.593106] Lustre: Skipped 1 previous similar message
[ 7044.688454] Lustre: server umount lustre-MDT0000 complete
[ 7047.200482] LustreError: 142362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788336694 with bad export cookie 10317984416633794855
[ 7047.207674] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7047.216250] LustreError: 142362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 7047.300622] Lustre: server umount lustre-OST0000 complete
[ 7049.964179] Lustre: server umount lustre-OST0001 complete
[ 7057.816333] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local
[ 7063.728495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7066.501251] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7068.356715] Lustre: 163974:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7073.952874] LustreError: 164334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7073.971121] LustreError: 164334:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages
[ 7073.981206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:715 to 0x240000400:737)
[ 7077.339391] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7086.920279] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7088.110873] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:713 to 0x280000400:737)
[ 7092.424892] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7094.929266] Lustre: 165882:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7140.358190] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 04:13:06 (1788336786)
[ 7198.720986] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 04:14:04 (1788336844)
[ 8363.740816] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 04:33:29 (1788338009)
[ 8373.217444] Lustre: server umount lustre-MDT0000 complete
[ 8375.313680] LustreError: 163529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788338022 with bad export cookie 10317984416633801274
[ 8375.322494] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8385.566212] Lustre: server umount lustre-OST0000 complete
[ 8387.420865] LustreError: 3309:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0001 qtype:usr lqe: ffff9a4cef930480 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2383 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 8397.684527] Lustre: server umount lustre-OST0001 complete
[ 8404.277451] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local
[ 8409.030299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8409.034083] Lustre: Skipped 2 previous similar messages
[ 8410.825935] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8412.178614] Lustre: 174371:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 8417.423846] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8419.107662] LustreError: 174746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8419.122530] LustreError: 174746:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message
[ 8419.133217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5740 to 0x240000400:5761)
[ 8421.191110] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 8421.195679] Lustre: Skipped 1 previous similar message
[ 8424.028171] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8426.474964] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5738 to 0x280000400:5761)
[ 8427.647721] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8429.152716] Lustre: 176286:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 8452.792752] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 04:34:59 (1788338099)
[ 8482.847901] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 04:35:28 (1788338128)
[ 8511.330409] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 04:35:57 (1788338157)
[ 8512.007546] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 8512.774102] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 04:35:58 (1788338158)
[ 8513.427232] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 8514.139372] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 04:36:00 (1788338160)
[ 8586.890465] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 04:37:12 (1788338232)
[ 8622.834382] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 04:37:48 (1788338268)
[ 8678.739740] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 04:38:44 (1788338324)
[ 8686.803663] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8686.806317] Lustre: Skipped 2 previous similar messages
[ 8688.825195] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8688.827201] Lustre: Skipped 165 previous similar messages
[ 8692.843994] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8692.846484] Lustre: Skipped 361 previous similar messages
[ 8700.863079] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8700.865646] Lustre: Skipped 807 previous similar messages
[ 8716.880118] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8716.882026] Lustre: Skipped 1465 previous similar messages
[ 8748.881201] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8748.885094] Lustre: Skipped 2891 previous similar messages
[ 9128.672728] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9128.684312] Lustre: Skipped 2310 previous similar messages
[ 9128.692473] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9128.702600] Lustre: Skipped 11 previous similar messages
[ 9220.075219] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9220.095616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9220.108379] Lustre: Skipped 1 previous similar message
[ 9224.834201] Lustre: server umount lustre-MDT0000 complete
[ 9229.004595] LustreError: 173924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788338875 with bad export cookie 10317984416635569698
[ 9229.012450] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9229.370477] Lustre: server umount lustre-OST0000 complete
[ 9233.836305] Lustre: server umount lustre-OST0001 complete
[ 9241.408387] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid
[ 9248.664611] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local
[ 9257.617255]  vdc: vdc1 vdc9
[ 9257.643272]  vdc: vdc1 vdc9
[ 9257.693183]  vdc: vdc1 vdc9
[ 9266.769626]  vde: vde1 vde9
[ 9275.272883]  vdf: vdf1 vdf9
[ 9275.311351]  vdf: vdf1 vdf9
[ 9275.359786]  vdf: vdf1 vdf9
[ 9290.518816] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local
[ 9301.355129] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 9301.587758] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 9301.692994] Lustre: lustre-MDT0000: new disk, initializing
[ 9302.116069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9302.191056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 9306.260222] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9310.840471] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 9316.991626] Lustre: lustre-OST0000: new disk, initializing
[ 9317.000192] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 9317.012612] Lustre: Skipped 1 previous similar message
[ 9317.210817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9319.054440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[ 9319.060419] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[ 9319.191853] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[ 9323.565632] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9332.067488] Lustre: lustre-OST0001: new disk, initializing
[ 9332.071294] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 9332.124741] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9333.821956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[ 9333.830826] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[ 9333.931142] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[ 9337.653870] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9348.535390] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9354.200962] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 9402.321657] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 04:50:47 (1788339047)
[ 9409.263524] Lustre: DEBUG MARKER: == sanity-quota test 49c: lfs quota long options don't consume an extra argument ========================================================== 04:50:54 (1788339054)
[ 9417.195588] Lustre: DEBUG MARKER: == sanity-quota test 49d: lfs quota -d and --delimiter both work ========================================================== 04:51:02 (1788339062)
[ 9425.440787] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 04:51:10 (1788339070)
[ 9474.605354] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 04:51:59 (1788339119)
[ 9536.040629] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 04:53:01 (1788339181)
[ 9568.755882] Lustre: DEBUG MARKER: rename directory return 255
[ 9618.171306] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 04:54:23 (1788339263)
[ 9649.769466] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 04:54:54 (1788339294)
[ 9693.320781] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 04:55:38 (1788339338)
[ 9804.063423] LustreError: 191280:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 9844.030213] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 04:58:09 (1788339489)
[ 9885.061496] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 04:58:50 (1788339530)
[ 9929.017581] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 04:59:33 (1788339573)
[10117.669571] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 05:02:42 (1788339762)
[10119.303341] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[10120.956328] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 05:02:46 (1788339766)
[10182.721317] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[10184.392382] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 05:03:49 (1788339829)
[10215.208904] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[10216.802925] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 05:04:22 (1788339862)
[10273.170506] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[10274.919943] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 05:05:20 (1788339920)
[10322.447990] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 05:06:07 (1788339967)
[10323.752664] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[10325.382660] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 05:06:10 (1788339970)
[10353.723245] LustreError: 214634:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[10401.230693] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 05:07:26 (1788340046)
[10426.847303] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[10428.402645] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[10599.362547] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 05:10:44 (1788340244)
[10646.810471] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 05:11:32 (1788340292)
[10676.394991] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 05:12:01 (1788340321)
[10678.218991] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[10680.186487] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 05:12:05 (1788340325)
[10681.981585] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[10684.118332] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 05:12:09 (1788340329)
[10700.047992] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[10717.195822] Lustre: DEBUG MARKER: Write...
[10719.984709] Lustre: DEBUG MARKER: Write out of block quota ...
[10774.138366] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 05:13:39 (1788340419)
[10801.001485] Lustre: DEBUG MARKER: set to use default quota
[10802.769309] Lustre: DEBUG MARKER: set default quota
[10804.485092] Lustre: DEBUG MARKER: get default quota
[10810.249752] Lustre: DEBUG MARKER: Test not out of quota
[10814.099386] Lustre: DEBUG MARKER: Test out of quota
[10824.668838] Lustre: DEBUG MARKER: Increase default quota
[10856.206833] Lustre: DEBUG MARKER: Set quota to override default quota
[10856.260447] LustreError: 191278:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1788945303 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[10869.306106] Lustre: DEBUG MARKER: Set to use default quota again
[10893.812677] Lustre: DEBUG MARKER: Cleanup
[10974.463872] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 05:16:59 (1788340619)
[10998.900983] Lustre: DEBUG MARKER: set default quota for qpool1
[11000.736240] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[11050.895665] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 05:18:16 (1788340696)
[11130.212244] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 05:19:35 (1788340775)
[11206.952512] Lustre: DEBUG MARKER: Write...
[11210.658565] Lustre: DEBUG MARKER: Write out of block quota ...
[11322.214847] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 05:22:47 (1788340967)
[11366.839859] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 05:23:32 (1788341012)
[11373.621222] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 05:23:38 (1788341018)
[11376.117424] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A fallocate not supported
[11377.806486] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 05:23:43 (1788341023)
[11380.123850] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a fallocate not supported
[11381.734507] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 05:23:47 (1788341027)
[11399.483715] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 05:24:04 (1788341044)
[11400.813200] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[11402.379296] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 05:24:07 (1788341047)
[11417.428271] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[11423.969348] LustreError: 239993:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[11423.975693] LustreError: 239993:0:(qmt_pool.c:1407:qmt_pool_recalc()) Skipped 3 previous similar messages
[11427.812564] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[11427.814083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[11427.831874] Lustre: Skipped 2 previous similar messages
[11427.839220] Lustre: Skipped 1 previous similar message
[11428.822655] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.14@tcp (stopping)
[11432.928469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[11432.932975] Lustre: Skipped 1 previous similar message
[11434.023135] LustreError: 239993:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[11434.330513] LustreError: 240092:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff9a4cc7839c00) refcount nonzero (2) after lock cleanup; forcing cleanup.
[11434.348496] LustreError: 240092:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 5 previous similar messages
[11434.471670] Lustre: server umount lustre-MDT0000 complete
[11442.442258] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[11443.060202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[11443.242583] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[11443.244181] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[11447.723848] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11457.254289] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[11457.266584] Lustre: Skipped 1 previous similar message
[11486.754958] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 05:25:32 (1788341132)
[11517.626470] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 05:26:03 (1788341163)
[11546.038518] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 05:26:31 (1788341191)
[11592.502817] Lustre: *** cfs_fail_loc=a08, val=0***
[11592.505737] Lustre: Skipped 5 previous similar messages
[11592.520842] Lustre: *** cfs_fail_loc=a08, val=0***
[11592.522747] Lustre: Skipped 3 previous similar messages
[11698.277809] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 05:29:03 (1788341343)
[11779.503763] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 05:30:25 (1788341425)
[11847.115509] LustreError: 240574:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1788946293 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11978.815181] LustreError: 242289:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1788946425 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12170.755967] LustreError: 249443:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:1000 enforced:1 hard:2048 soft:2048 granted:65536 time:1788946617 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12179.950338] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed