[ 5499.596915] Lustre: DEBUG MARKER: User quota (limit: 200) [ 5505.009073] Lustre: DEBUG MARKER: Write 100M (buffered) ... [ 5508.329342] LustreError: 121676:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 5509.513154] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5511.431588] Lustre: DEBUG MARKER: Fail mds for 40 seconds [ 5513.515585] Lustre: Failing over lustre-MDT0000 [ 5513.543356] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -19 [ 5513.545741] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5513.552798] LustreError: 3300:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -19, flags:0x2 qsd:lustre-OST0000 qtype:usr lqe: ffffa0427bd30000 id:60000 enforced:1 granted: 49152 pending:0 waiting:1024 req:1 usage: 48135 qunit:16384 qtune:4096 edquot:0 default:no revoke:0 [ 5513.561134] LustreError: 3300:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 1 previous similar message [ 5513.565025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5513.881879] Lustre: server umount lustre-MDT0000 complete [ 5530.591771] Lustre: 3302:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786804708/real 1786804708] req@ffffa042751cc700 x1873594862997888/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786804724 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5530.594094] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5530.627604] Lustre: 3302:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 5530.627686] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5536.543164] Lustre: 3303:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786804713/real 1786804713] req@ffffa04272e15c00 x1873594862998784/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786804729 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5540.767341] Lustre: 3301:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786804718/real 1786804718] req@ffffa0427405ca80 x1873594862999296/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786804734 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5541.735435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5541.838019] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5542.446889] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5542.724464] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5542.793927] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:643 to 0x240000400:673) [ 5542.797589] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:639 to 0x280000400:673) [ 5546.639085] Lustre: 3300:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786804723/real 1786804723] req@ffffa042751ce680 x1873594862999808/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786804739 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5547.772376] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5549.730293] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5559.371943] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5560.853237] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5571.601923] Lustre: DEBUG MARKER: (dd_pid=115039, time=4, timeout=600) [ 5628.163336] Lustre: DEBUG MARKER: User quota (limit: 200) [ 5635.710912] Lustre: DEBUG MARKER: Write 100M (directio) ... [ 5639.400875] LustreError: 124608:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 5640.343703] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5642.274249] Lustre: DEBUG MARKER: Fail mds for 40 seconds [ 5644.300321] Lustre: Failing over lustre-MDT0000 [ 5644.667953] Lustre: server umount lustre-MDT0000 complete [ 5660.641687] Lustre: 3300:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786804838/real 1786804838] req@ffffa04149c7ca80 x1873594863041280/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1786804854 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295 [ 5660.664981] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5662.739294] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5663.334387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5663.394939] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5664.569205] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5664.698161] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5664.727434] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:639 to 0x280000400:705) [ 5664.730140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:675 to 0x240000400:705) [ 5665.321733] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5665.340805] Lustre: Skipped 1 previous similar message [ 5667.526304] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5676.898964] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5677.519512] Lustre: 3303:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786804854/real 1786804854] req@ffffa0418ed59180 x1873594863042688/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786804870 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5677.573712] Lustre: 3303:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 5679.088318] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5692.679028] Lustre: DEBUG MARKER: (dd_pid=117526, time=6, timeout=600) [ 5762.109909] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 10:42:34 (1786804954) [ 5785.458941] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1: [ 5787.487842] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2). [ 5794.361734] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1: [ 5797.068616] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2). [ 5834.263436] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 10:43:46 (1786805026) [ 5861.249767] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing [ 5879.032885] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr [ 5880.745387] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr [ 5882.505462] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000 [ 5884.408930] Lustre: DEBUG MARKER: Set quota for 1 times [ 5888.236867] Lustre: DEBUG MARKER: Set quota for 2 times [ 5892.159765] Lustre: DEBUG MARKER: Set quota for 3 times [ 5896.169768] Lustre: DEBUG MARKER: Set quota for 4 times [ 5900.395448] Lustre: DEBUG MARKER: Set quota for 5 times [ 5904.177605] Lustre: DEBUG MARKER: Set quota for 6 times [ 5907.956973] Lustre: DEBUG MARKER: Set quota for 7 times [ 5912.133444] Lustre: DEBUG MARKER: Set quota for 8 times [ 5972.171975] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 10:46:03 (1786805163) [ 5988.542782] Lustre: server umount lustre-MDT0000 complete [ 5993.733448] LustreError: 8364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786805187 with bad export cookie 11819868817582922379 [ 5993.741595] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6004.318626] Lustre: server umount lustre-OST0000 complete [ 6004.639142] Lustre: 3302:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786805182/real 1786805182] req@ffffa04141af6680 x1873594863169792/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786805198 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6004.664603] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6004.674296] Lustre: Skipped 1 previous similar message [ 6008.471443] Lustre: server umount lustre-OST0001 complete [ 6024.561569] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 6035.231570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6040.422865] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6043.772813] Lustre: 134314:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6050.100817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6057.515528] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6060.325622] LustreError: 134631:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6060.359846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:708 to 0x240000400:737) [ 6065.485610] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6070.801120] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:707 to 0x280000400:737) [ 6073.715681] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6081.932355] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6085.971276] Lustre: 136143:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6101.472140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6103.376590] Lustre: server umount lustre-MDT0000 complete [ 6108.003231] LustreError: 136392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786805301 with bad export cookie 11819868817582929645 [ 6108.006781] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6108.009661] LustreError: 136392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6108.070296] Lustre: server umount lustre-OST0000 complete [ 6113.053102] Lustre: server umount lustre-OST0001 complete [ 6130.729840] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 6142.571533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6148.650786] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6151.059861] Lustre: 138742:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6166.632947] LustreError: 139064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6166.703487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:708 to 0x240000400:769) [ 6167.047690] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6172.150989] LustreError: 139065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6176.677917] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6176.684268] Lustre: Skipped 1 previous similar message [ 6181.914530] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:707 to 0x280000400:769) [ 6185.159648] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6192.857905] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6196.548989] Lustre: 140565:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6207.019816] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 10:49:59 (1786805399) [ 6209.144175] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS [ 6211.175391] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 10:50:03 (1786805403) [ 6266.844121] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 10:50:58 (1786805458) [ 6317.934631] Lustre: DEBUG MARKER: Write... [ 6320.710139] Lustre: DEBUG MARKER: Write out of block quota ... [ 6380.691520] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 10:52:53 (1786805573) [ 6388.456640] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 10:53:00 (1786805580) [ 6399.095609] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 10:53:11 (1786805591) [ 6409.588471] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 10:53:21 (1786805601) [ 6419.241138] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 10:53:31 (1786805611) [ 6485.596583] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user [ 6640.479185] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user [ 6840.648521] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 11:00:32 (1786806032) [ 6905.519270] Lustre: DEBUG MARKER: Restart... [ 6912.932771] Lustre: server umount lustre-MDT0000 complete [ 6917.109372] LustreError: 3303:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8 qsd:lustre-OST0000 qtype:usr lqe: ffffa041432af500 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2378 qunit:0 qtune:0 edquot:0 default:no revoke:0 [ 6917.111159] LustreError: 138298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786806110 with bad export cookie 11819868817582931178 [ 6917.136130] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6927.415159] Lustre: server umount lustre-OST0000 complete [ 6930.400171] Lustre: 3300:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786806107/real 1786806107] req@ffffa041422bf480 x1873594863442176/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786806123 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6930.431672] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6930.443054] Lustre: Skipped 2 previous similar messages [ 6932.295301] Lustre: server umount lustre-OST0001 complete [ 6949.709175] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 6960.329838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6964.806889] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6967.542475] Lustre: 159201:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6972.448837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6978.272123] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6984.674983] LustreError: 159517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6984.717107] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:779 to 0x240000400:801) [ 6987.458564] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6992.973995] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:777 to 0x280000400:801) [ 6996.960717] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7004.820270] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7008.161688] Lustre: 161021:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7074.970714] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 11:04:27 (1786806267) [ 7147.504045] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 11:05:39 (1786806339) [ 8734.151380] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 11:32:06 (1786807926) [ 8749.604611] Lustre: server umount lustre-MDT0000 complete [ 8754.186291] LustreError: 3301:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8 qsd:lustre-OST0000 qtype:usr lqe: ffffa04274c0db00 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2384 qunit:0 qtune:0 edquot:0 default:no revoke:0 [ 8754.190753] LustreError: 158755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786807947 with bad export cookie 11819868817582937674 [ 8754.211427] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8764.547309] Lustre: server umount lustre-OST0000 complete [ 8779.190386] Lustre: server umount lustre-OST0001 complete [ 8794.380460] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 8804.673491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8809.139939] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8811.415450] Lustre: 169277:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8817.080241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8824.541380] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8829.345700] LustreError: 169600:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8829.375345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5803 to 0x240000400:5825) [ 8832.437273] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8837.642667] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5803 to 0x280000400:5825) [ 8840.360651] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8848.161840] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8851.882943] Lustre: 171113:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8890.830503] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 11:34:43 (1786808083) [ 8936.025467] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 11:35:28 (1786808128) [ 8979.819583] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 11:36:12 (1786808172) [ 8981.812370] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs [ 8983.482634] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 11:36:15 (1786808175) [ 8985.400665] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs [ 8987.613342] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 11:36:19 (1786808179) [ 9092.602887] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 11:38:05 (1786808285) [ 9147.745241] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 11:39:00 (1786808340) [ 9238.960542] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 11:40:30 (1786808430) [ 9251.207473] Lustre: *** cfs_fail_loc=a09, val=0*** [ 9251.211477] Lustre: Skipped 1 previous similar message [ 9253.242586] Lustre: *** cfs_fail_loc=a09, val=0*** [ 9253.254858] Lustre: Skipped 69 previous similar messages [ 9257.286326] Lustre: *** cfs_fail_loc=a09, val=0*** [ 9257.290857] Lustre: Skipped 169 previous similar messages [ 9265.343476] Lustre: *** cfs_fail_loc=a09, val=0*** [ 9265.345397] Lustre: Skipped 331 previous similar messages [ 9281.344497] Lustre: *** cfs_fail_loc=a09, val=0*** [ 9281.347207] Lustre: Skipped 790 previous similar messages [ 9313.377920] Lustre: *** cfs_fail_loc=a09, val=0*** [ 9313.381245] Lustre: Skipped 1680 previous similar messages [ 9377.392819] Lustre: *** cfs_fail_loc=a09, val=0*** [ 9377.395390] Lustre: Skipped 3091 previous similar messages [ 9790.371271] Lustre: *** cfs_fail_loc=a08, val=0*** [ 9790.373063] Lustre: Skipped 1863 previous similar messages [ 9790.405117] Lustre: *** cfs_fail_loc=a08, val=0*** [ 9790.422677] Lustre: Skipped 2 previous similar messages [10037.467810] Lustre: server umount lustre-MDT0000 complete [10042.710422] LustreError: 174760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786809236 with bad export cookie 11819868817584712559 [10042.732539] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10049.556787] LustreError: 174760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786809243 with bad export cookie 11819868817584713238 [10049.557194] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10049.586707] LustreError: 174760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10053.275211] Lustre: server umount lustre-OST0000 complete [10054.623191] Lustre: 3302:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786809232/real 1786809232] req@ffffa04189821180 x1873594871302400/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786809248 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10054.651155] Lustre: 3302:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10057.323449] LustreError: 3301:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x4 qsd:lustre-OST0001 qtype:usr lqe: ffffa041454fe780 id:16 enforced:1 granted: 16384 pending:0 waiting:0 req:1 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0 [10057.339409] LustreError: 3300:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa04188249880 x1873594871311872/t0(0) o601->lustre-MDT0000-lwp-OST0001@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295 [10057.356776] LustreError: 3301:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 20 previous similar messages [10057.913160] Lustre: server umount lustre-OST0001 complete [10065.594251] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [10074.042200] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [10089.434700] vdc: vdc1 vdc9 [10089.451193] vdc: vdc1 vdc9 [10102.751692] vde: vde1 vde9 [10116.719855] vdf: vdf1 vdf9 [10135.620830] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [10145.078662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10145.455506] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10145.627393] Lustre: lustre-MDT0000: new disk, initializing [10146.057068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10146.177194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10151.287621] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10157.686540] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10164.103403] Lustre: lustre-OST0000: new disk, initializing [10164.108805] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10164.114910] Lustre: Skipped 1 previous similar message [10164.168938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10165.826521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10165.857796] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10166.123941] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10173.069764] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10183.312615] Lustre: lustre-OST0001: new disk, initializing [10183.317046] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10183.499016] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10190.865336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [10190.880694] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [10191.036729] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [10191.932487] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10203.249661] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10208.549422] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10252.932676] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 11:57:25 (1786809445) [10262.028316] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 11:57:34 (1786809454) [10316.289567] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 11:58:27 (1786809507) [10385.384664] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 11:59:37 (1786809577) [10417.555763] Lustre: DEBUG MARKER: rename directory return 255 [10467.370947] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 12:00:59 (1786809659) [10503.796872] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 12:01:35 (1786809695) [10553.575387] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 12:02:25 (1786809745) [10652.616509] LustreError: 3302:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-OST0001: Delaying reintegration for qtype:1 until pending updates are flushed. [10652.967303] LustreError: 195931:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [10692.649092] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 12:04:45 (1786809885) [10728.655521] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 12:05:20 (1786809920) [10776.101336] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 12:06:08 (1786809968) [10960.477606] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 12:09:12 (1786810152) [10962.096652] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test [10963.691391] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 12:09:16 (1786810156) [11022.021896] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61 [11024.137580] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 12:10:16 (1786810216) [11057.807812] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63 [11059.953367] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 12:10:51 (1786810251) [11115.589703] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65 [11117.434887] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 12:11:49 (1786810309) [11170.210795] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 12:12:42 (1786810362) [11171.740753] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode [11174.041708] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 12:12:45 (1786810365) [11205.282713] LustreError: 208698:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1 [11256.431370] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 12:14:08 (1786810448) [11284.101266] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB) [11286.053500] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB) [11473.014383] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 12:17:45 (1786810665) [11528.629577] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 12:18:40 (1786810720) [11563.924732] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 12:19:16 (1786810756) [11565.791825] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode [11568.059253] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 12:19:20 (1786810760) [11569.904298] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode [11571.936422] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 12:19:24 (1786810764) [11590.168285] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB) [11608.880481] Lustre: DEBUG MARKER: Write... [11611.551957] Lustre: DEBUG MARKER: Write out of block quota ... [11672.769284] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 12:21:05 (1786810865) [11699.412884] Lustre: DEBUG MARKER: set to use default quota [11700.599724] Lustre: DEBUG MARKER: set default quota [11702.320560] Lustre: DEBUG MARKER: get default quota [11708.109595] Lustre: DEBUG MARKER: Test not out of quota [11711.514237] Lustre: DEBUG MARKER: Test out of quota [11722.563942] Lustre: DEBUG MARKER: Increase default quota [11753.810310] Lustre: DEBUG MARKER: Set quota to override default quota [11753.886082] LustreError: 185893:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1787415747 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [11770.624631] Lustre: DEBUG MARKER: Set to use default quota again [11791.264311] Lustre: DEBUG MARKER: Cleanup [11862.695663] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 12:24:14 (1786811054) [11888.081990] Lustre: DEBUG MARKER: set default quota for qpool1 [11889.762615] Lustre: DEBUG MARKER: Write from user that hasn't lqe [11939.737919] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 12:25:32 (1786811132) [12017.276218] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 12:26:49 (1786811209) [12091.031665] Lustre: DEBUG MARKER: Write... [12094.506269] Lustre: DEBUG MARKER: Write out of block quota ... [12214.489609] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 12:30:06 (1786811406) [12263.240264] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 12:30:54 (1786811454) [12271.938810] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 12:31:04 (1786811464) [12273.369552] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate [12275.369570] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 12:31:07 (1786811467) [12277.234593] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate [12278.330282] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 12:31:11 (1786811471) [12299.810700] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 12:31:31 (1786811491) [12302.027077] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode [12304.699807] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 12:31:36 (1786811496) [12323.886325] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB) [12330.584983] LustreError: 233718:0:(qmt_pool.c:1413:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms [12330.598494] LustreError: 233718:0:(qmt_pool.c:1413:qmt_pool_recalc()) Skipped 5 previous similar messages [12335.407269] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.20@tcp (stopping) [12335.416579] Lustre: Skipped 1 previous similar message [12335.586704] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12340.485120] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.20@tcp (stopping) [12340.500512] Lustre: Skipped 2 previous similar messages [12340.655178] LustreError: 233718:0:(qmt_pool.c:1413:qmt_pool_recalc()) cfs_fail_timeout id a07 awake [12341.296913] Lustre: server umount lustre-MDT0000 complete [12349.760423] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12350.283955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12350.399676] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65) [12350.405649] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65) [12354.650908] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [12365.800848] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12365.806820] Lustre: Skipped 1 previous similar message [12398.219963] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 12:33:10 (1786811590) [12432.410301] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 12:33:44 (1786811624) [12467.560438] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 12:34:19 (1786811659) [12524.777037] Lustre: *** cfs_fail_loc=a08, val=0*** [12524.780240] Lustre: Skipped 14 previous similar messages [12524.784505] Lustre: *** cfs_fail_loc=a08, val=0*** [12524.788619] Lustre: Skipped 13 previous similar messages [12643.367814] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 12:37:15 (1786811835) [12727.736945] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 12:38:39 (1786811919) [12809.083485] LustreError: 235083:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1787416802 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [12817.968217] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed