[ 5884.069763] Lustre: DEBUG MARKER: User quota (limit: 200) [ 5890.874704] Lustre: DEBUG MARKER: Write 100M (buffered) ... [ 5895.140243] LustreError: 120661:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 5896.237238] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5897.994703] Lustre: DEBUG MARKER: Fail mds for 40 seconds [ 5900.801480] Lustre: Failing over lustre-MDT0000 [ 5901.108610] Lustre: server umount lustre-MDT0000 complete [ 5919.201151] Lustre: 3312:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787401075/real 1787401075] req@ffff9176c1b88e00 x1874219758794240/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1787401091 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295 [ 5919.238803] Lustre: 3312:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5919.246996] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5919.267684] Lustre: Skipped 1 previous similar message [ 5920.032220] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5924.383177] Lustre: 3312:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787401080/real 1787401080] req@ffff9175898be680 x1874219758794752/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787401096 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5924.423939] Lustre: 3312:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5929.551631] Lustre: 3312:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787401085/real 1787401085] req@ffff9175898be300 x1874219758795136/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787401101 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5929.626477] Lustre: 3312:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5930.144678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5930.232333] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5930.772165] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5931.019958] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5931.109924] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:630 to 0x240000400:673) [ 5931.110367] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:626 to 0x280000400:641) [ 5935.852634] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5944.302914] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5944.314601] Lustre: Skipped 1 previous similar message [ 5947.112304] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5948.953936] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5966.449603] Lustre: DEBUG MARKER: (dd_pid=114323, time=9, timeout=600) [ 6021.905056] Lustre: DEBUG MARKER: User quota (limit: 200) [ 6029.446466] Lustre: DEBUG MARKER: Write 100M (directio) ... [ 6033.343472] LustreError: 123609:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 6034.578231] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6036.472926] Lustre: DEBUG MARKER: Fail mds for 40 seconds [ 6039.001672] Lustre: Failing over lustre-MDT0000 [ 6039.496218] Lustre: server umount lustre-MDT0000 complete [ 6056.929252] Lustre: 3312:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787401212/real 1787401212] req@ffff9175cc7c0380 x1874219758839168/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1787401228 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295 [ 6056.964218] Lustre: 3312:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 6056.971890] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6056.988490] Lustre: Skipped 1 previous similar message [ 6058.023196] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6067.168782] LustreError: 3310:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9175cc7c0e00 x1874219758842624/t0(0) o250->MGC192.168.202.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6068.043190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6068.127943] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6070.022320] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 6070.300992] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 6070.359157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:675 to 0x240000400:705) [ 6070.366200] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:626 to 0x280000400:673) [ 6073.055335] Lustre: 3311:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787401228/real 1787401228] req@ffff9176b4c5ad80 x1874219758840704/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787401244 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6073.077381] Lustre: 3311:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 6074.233207] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6081.262888] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6081.267039] Lustre: Skipped 1 previous similar message [ 6085.449506] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6087.329748] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6104.067993] Lustre: DEBUG MARKER: (dd_pid=116807, time=9, timeout=600) [ 6174.062328] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 08:22:23 (1787401343) [ 6203.001418] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1: [ 6205.576355] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2). [ 6213.730696] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1: [ 6215.285797] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2). [ 6257.617590] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 08:23:46 (1787401426) [ 6291.213964] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing [ 6309.533891] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr [ 6311.913640] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr [ 6314.416771] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000 [ 6316.727936] Lustre: DEBUG MARKER: Set quota for 1 times [ 6322.158677] Lustre: DEBUG MARKER: Set quota for 2 times [ 6326.405296] Lustre: DEBUG MARKER: Set quota for 3 times [ 6330.806574] Lustre: DEBUG MARKER: Set quota for 4 times [ 6335.039229] Lustre: DEBUG MARKER: Set quota for 5 times [ 6338.857726] Lustre: DEBUG MARKER: Set quota for 6 times [ 6342.938865] Lustre: DEBUG MARKER: Set quota for 7 times [ 6382.578661] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 08:25:51 (1787401551) [ 6397.402198] Lustre: server umount lustre-MDT0000 complete [ 6402.962322] LustreError: 39127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401574 with bad export cookie 6681768105894088941 [ 6402.989060] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6412.410852] Lustre: server umount lustre-OST0000 complete [ 6415.139774] Lustre: 3311:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787401571/real 1787401571] req@ffff9175cbdcbb80 x1874219758967552/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787401587 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6415.170426] Lustre: 3311:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6415.174801] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6415.189241] Lustre: Skipped 1 previous similar message [ 6418.240423] Lustre: server umount lustre-OST0001 complete [ 6436.747063] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 6452.942124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6459.730938] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6462.977149] Lustre: 133276:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6469.823741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6477.867845] LustreError: 133594:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6477.887617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:708 to 0x240000400:737) [ 6477.905159] LustreError: 133594:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6478.039689] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6482.923852] LustreError: 133594:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6487.443514] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6492.677295] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:675 to 0x280000400:705) [ 6494.964763] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6502.856966] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6506.595326] Lustre: 135096:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6524.377030] Lustre: server umount lustre-MDT0000 complete [ 6528.488771] LustreError: 132820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401700 with bad export cookie 6681768105894095773 [ 6528.502851] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6538.869267] Lustre: server umount lustre-OST0000 complete [ 6552.392107] Lustre: server umount lustre-OST0001 complete [ 6567.566410] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 6578.967114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6585.203508] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6588.656884] Lustre: 137700:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6596.830371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6603.938416] LustreError: 138021:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6603.978522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:708 to 0x240000400:769) [ 6604.514093] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6609.388245] LustreError: 138021:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6619.653533] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:675 to 0x280000400:737) [ 6621.901240] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6630.613036] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6634.479958] Lustre: 139521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6645.175466] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 08:30:14 (1787401814) [ 6646.695069] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS [ 6648.855647] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 08:30:18 (1787401818) [ 6708.344422] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 08:31:17 (1787401877) [ 6763.527966] Lustre: DEBUG MARKER: Write... [ 6765.985431] Lustre: DEBUG MARKER: Write out of block quota ... [ 6821.727812] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 08:33:10 (1787401990) [ 6829.455634] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 08:33:18 (1787401998) [ 6838.452575] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 08:33:28 (1787402008) [ 6847.775084] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 08:33:37 (1787402017) [ 6856.409627] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 08:33:45 (1787402025) [ 6926.472289] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user [ 7097.532487] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user [ 7297.621344] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 08:41:06 (1787402466) [ 7370.014770] Lustre: DEBUG MARKER: Restart... [ 7374.087818] Lustre: server umount lustre-MDT0000 complete [ 7378.693902] LustreError: 137256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402550 with bad export cookie 6681768105894097306 [ 7378.712361] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7388.260824] Lustre: server umount lustre-OST0000 complete [ 7403.362839] Lustre: server umount lustre-OST0001 complete [ 7420.313841] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 7431.823610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7431.827767] Lustre: Skipped 1 previous similar message [ 7437.241677] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7440.592578] Lustre: 157911:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7447.644505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7455.712610] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7456.757328] LustreError: 158235:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7456.852933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:778 to 0x240000400:801) [ 7461.861414] LustreError: 158631:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7466.977861] LustreError: 158234:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7468.167844] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7473.152434] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:746 to 0x280000400:769) [ 7477.816873] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7487.362206] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7491.279004] Lustre: 159733:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7565.677810] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 08:45:34 (1787402734) [ 7643.965671] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 08:46:53 (1787402813) [ 9297.783220] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 09:14:27 (1787404467) [ 9312.494972] Lustre: server umount lustre-MDT0000 complete [ 9316.254785] LustreError: 157466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404488 with bad export cookie 6681768105894103802 [ 9316.265994] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9326.636952] Lustre: server umount lustre-OST0000 complete [ 9340.679279] Lustre: server umount lustre-OST0001 complete [ 9355.242909] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 9365.140503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9369.056699] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9371.711570] Lustre: 167969:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9377.502728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9384.126921] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9389.744546] LustreError: 168285:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9389.820693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5803 to 0x240000400:5825) [ 9393.363800] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9398.268258] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5771 to 0x280000400:5793) [ 9401.625910] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9409.731265] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9414.019544] Lustre: 169785:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9452.556278] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 09:17:00 (1787404620) [ 9501.695675] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 09:17:51 (1787404671) [ 9546.449395] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 09:18:35 (1787404715) [ 9547.835791] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs [ 9549.474317] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 09:18:38 (1787404718) [ 9550.652835] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs [ 9552.109835] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 09:18:41 (1787404721) [ 9641.180660] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 09:20:10 (1787404810) [ 9690.440425] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 09:20:59 (1787404859) [ 9810.277637] Lustre: DEBUG MARKER: sanity-quota test_48: @@@@@@ FAIL: QMT: mds1 qid 60000 is not deleted