[ 5871.593867] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5877.619168] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5881.258676] LustreError: 119595:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5882.595625] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5884.862902] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5886.968900] Lustre: Failing over lustre-MDT0000
[ 5887.432581] Lustre: server umount lustre-MDT0000 complete
[ 5905.318293] Lustre: 3318:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781231188/real 1781231188]  req@ffff9f60d6bf43c0 x1867750192726784/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1781231204 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5905.363098] Lustre: 3318:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5905.368604] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5906.401727] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5911.523593] Lustre: 3318:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781231194/real 1781231194]  req@ffff9f600aaa7c00 x1867750192727424/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781231210 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5911.547818] Lustre: 3318:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5917.106245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5917.165701] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5917.991120] Lustre: 3319:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781231199/real 1781231199]  req@ffff9f60d6bf70c0 x1867750192727808/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781231215 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5918.029675] Lustre: 3319:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5921.135923] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5929.334686] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5929.542715] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5929.593850] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:546 to 0x240000400:577)
[ 5929.602159] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:543 to 0x280000400:577)
[ 5930.343604] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 5933.258795] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5935.120184] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5949.432746] Lustre: DEBUG MARKER: (dd_pid=113995, time=7, timeout=600)
[ 6005.850010] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 6013.093237] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 6016.356047] LustreError: 122357:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 6017.511621] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6019.692969] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 6022.352744] Lustre: Failing over lustre-MDT0000
[ 6022.762240] Lustre: server umount lustre-MDT0000 complete
[ 6040.032110] Lustre: 3316:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781231322/real 1781231322]  req@ffff9f60fd0d0f00 x1867750192772224/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781231338 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6040.032761] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6040.061962] Lustre: 3316:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 6040.062382] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6040.062386] Lustre: Skipped 1 previous similar message
[ 6050.273460] LustreError: 3315:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff9f60dac45680 x1867750192774784/t0(0) o250->MGC192.168.204.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6050.944729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6051.119917] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 6052.220796] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 6052.429637] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 6052.492131] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:543 to 0x280000400:609)
[ 6052.492737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:579 to 0x240000400:609)
[ 6056.293146] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6065.125390] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 6065.137841] Lustre: Skipped 1 previous similar message
[ 6065.640957] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6067.960800] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6087.066621] Lustre: DEBUG MARKER: (dd_pid=116432, time=11, timeout=600)
[ 6156.225666] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 22:30:52 (1781231452)
[ 6181.508541] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6183.964556] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6191.576609] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6194.128408] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6236.053248] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 22:32:12 (1781231532)
[ 6266.443745] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 6284.347655] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 6286.560371] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 6288.402967] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 6290.924568] Lustre: DEBUG MARKER: Set quota for 1 times
[ 6295.011479] Lustre: DEBUG MARKER: Set quota for 2 times
[ 6298.773535] Lustre: DEBUG MARKER: Set quota for 3 times
[ 6303.134586] Lustre: DEBUG MARKER: Set quota for 4 times
[ 6307.039121] Lustre: DEBUG MARKER: Set quota for 5 times
[ 6311.194370] Lustre: DEBUG MARKER: Set quota for 6 times
[ 6314.724301] Lustre: DEBUG MARKER: Set quota for 7 times
[ 6318.235211] Lustre: DEBUG MARKER: Set quota for 8 times
[ 6379.955636] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 22:34:36 (1781231676)
[ 6400.993099] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6400.993338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6401.008172] Lustre: Skipped 2 previous similar messages
[ 6401.035271] Lustre: Skipped 1 previous similar message
[ 6403.307248] Lustre: server umount lustre-MDT0000 complete
[ 6408.298785] LustreError: 5773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231706 with bad export cookie 13460140735664169291
[ 6408.311707] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6408.419296] Lustre: server umount lustre-OST0000 complete
[ 6413.590199] Lustre: server umount lustre-OST0001 complete
[ 6430.858177] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local
[ 6441.833630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6446.263581] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6450.730390] Lustre: 131965:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6456.989709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6464.540107] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6467.171834] LustreError: 132285:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6467.209908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:612 to 0x240000400:641)
[ 6472.680739] LustreError: 132285:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6473.101431] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6478.349777] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:611 to 0x280000400:641)
[ 6479.710938] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6487.837234] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6491.759871] Lustre: 133755:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6509.859906] Lustre: server umount lustre-MDT0000 complete
[ 6514.095689] LustreError: 131530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231812 with bad export cookie 13460140735664176417
[ 6514.104798] LustreError: 131530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 6514.122094] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6523.458734] Lustre: server umount lustre-OST0000 complete
[ 6537.497094] Lustre: server umount lustre-OST0001 complete
[ 6552.153975] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local
[ 6561.296489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6566.128493] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6569.452435] Lustre: 136329:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6584.083992] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6585.450619] LustreError: 136641:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6585.486926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:612 to 0x240000400:673)
[ 6590.953279] LustreError: 137049:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6593.747730] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6593.753605] Lustre: Skipped 1 previous similar message
[ 6599.200488] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:611 to 0x280000400:673)
[ 6602.008980] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6611.450501] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6616.112191] Lustre: 138107:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6629.184525] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 22:38:45 (1781231925)
[ 6631.151740] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6633.729169] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 22:38:49 (1781231929)
[ 6695.508854] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 22:39:51 (1781231991)
[ 6749.661770] Lustre: DEBUG MARKER: Write...
[ 6752.400435] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6810.357201] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 22:41:46 (1781232106)
[ 6818.918917] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 22:41:55 (1781232115)
[ 6829.775042] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 22:42:06 (1781232126)
[ 6840.168180] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 22:42:17 (1781232137)
[ 6847.362559] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 22:42:24 (1781232144)
[ 6914.491354] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 7098.080374] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 7303.240952] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 22:49:59 (1781232599)
[ 7374.819983] Lustre: DEBUG MARKER: Restart...
[ 7382.498444] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7382.504984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7382.519223] Lustre: Skipped 1 previous similar message
[ 7382.539402] Lustre: Skipped 1 previous similar message
[ 7385.861250] Lustre: server umount lustre-MDT0000 complete
[ 7390.655391] LustreError: 135901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232689 with bad export cookie 13460140735664177957
[ 7390.655823] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7390.673687] LustreError: 135901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 7390.838854] Lustre: server umount lustre-OST0000 complete
[ 7395.736352] Lustre: server umount lustre-OST0001 complete
[ 7411.525903] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local
[ 7421.690749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7426.485873] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7430.171936] Lustre: 156307:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7436.882346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7444.056831] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7447.075688] LustreError: 156630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7447.097485] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:682 to 0x240000400:705)
[ 7452.137140] LustreError: 156630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7452.501219] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7457.801904] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:682 to 0x280000400:705)
[ 7459.745651] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7468.734590] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7473.059105] Lustre: 158101:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7549.667150] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 22:54:06 (1781232846)
[ 7629.017929] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 22:55:25 (1781232925)
[ 9702.390467] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 23:29:58 (1781234998)
[ 9720.802751] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9720.807279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9720.817840] Lustre: Skipped 1 previous similar message
[ 9720.839673] Lustre: Skipped 1 previous similar message
[ 9721.258547] Lustre: server umount lustre-MDT0000 complete
[ 9725.360444] LustreError: 155877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235024 with bad export cookie 13460140735664184453
[ 9725.367782] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9725.380756] LustreError: 155877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 9725.526086] Lustre: server umount lustre-OST0000 complete
[ 9729.229480] Lustre: server umount lustre-OST0001 complete
[ 9744.171646] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local
[ 9755.153220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9760.400797] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9763.803156] Lustre: 166458:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9770.377837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9778.073692] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9779.559441] LustreError: 166781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9779.598144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5708 to 0x240000400:5729)
[ 9784.807186] LustreError: 166781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9786.690718] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9792.005811] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5706 to 0x280000400:5729)
[ 9793.980360] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9801.583574] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9805.960401] Lustre: 168249:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9840.202594] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 23:32:16 (1781235136)
[ 9885.937567] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 23:33:02 (1781235182)
[ 9930.946989] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 23:33:47 (1781235227)
[ 9932.981422] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9935.312801] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 23:33:51 (1781235231)
[ 9937.594642] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9940.004671] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 23:33:56 (1781235236)
[10033.820207] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 23:35:30 (1781235330)
[10087.399567] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 23:36:24 (1781235384)
[10184.077557] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 23:38:00 (1781235480)
[10195.984206] Lustre: *** cfs_fail_loc=a09, val=0***
[10195.986842] Lustre: Skipped 3 previous similar messages
[10197.998400] Lustre: *** cfs_fail_loc=a09, val=0***
[10198.004483] Lustre: Skipped 87 previous similar messages
[10202.026931] Lustre: *** cfs_fail_loc=a09, val=0***
[10202.030390] Lustre: Skipped 179 previous similar messages
[10210.045785] Lustre: *** cfs_fail_loc=a09, val=0***
[10210.048331] Lustre: Skipped 313 previous similar messages
[10226.076835] Lustre: *** cfs_fail_loc=a09, val=0***
[10226.080915] Lustre: Skipped 593 previous similar messages
[10258.082182] Lustre: *** cfs_fail_loc=a09, val=0***
[10258.086061] Lustre: Skipped 1367 previous similar messages
[10322.098573] Lustre: *** cfs_fail_loc=a09, val=0***
[10322.101565] Lustre: Skipped 2437 previous similar messages
[10731.363221] Lustre: *** cfs_fail_loc=a08, val=0***
[10731.372983] Lustre: Skipped 3020 previous similar messages
[10731.380273] Lustre: *** cfs_fail_loc=a08, val=0***
[10731.389889] Lustre: Skipped 7 previous similar messages
[11020.773306] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[11020.781634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[11020.793988] Lustre: Skipped 1 previous similar message
[11020.813808] Lustre: Skipped 1 previous similar message
[11024.728107] Lustre: server umount lustre-MDT0000 complete
[11028.790913] LustreError: 171537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236327 with bad export cookie 13460140735665967752
[11028.819282] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[11029.213067] Lustre: server umount lustre-OST0000 complete
[11033.571338] Lustre: server umount lustre-OST0001 complete
[11040.783388] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid
[11049.118593] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local
[11058.857982]  vdc: vdc1 vdc9
[11070.335147]  vde: vde1 vde9
[11081.168365]  vdf: vdf1 vdf9
[11097.503738] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local
[11106.850964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[11107.095949] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[11107.153852] Lustre: lustre-MDT0000: new disk, initializing
[11107.471488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[11107.572085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[11112.899832] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11118.653712] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[11124.702833] Lustre: lustre-OST0000: new disk, initializing
[11124.712137] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[11124.718675] Lustre: Skipped 1 previous similar message
[11124.834346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[11125.997844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[11126.003637] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[11126.170892] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[11131.665143] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11143.476525] Lustre: lustre-OST0001: new disk, initializing
[11143.483055] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[11143.656388] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[11145.595792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[11145.603752] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[11145.760955] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[11151.006825] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11161.585210] Lustre: DEBUG MARKER: Using TIMEOUT=20
[11165.593342] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[11204.625589] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 23:55:01 (1781236501)
[11214.061552] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 23:55:10 (1781236510)
[11264.777696] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 23:56:01 (1781236561)
[11332.336338] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 23:57:08 (1781236628)
[11366.945281] Lustre: DEBUG MARKER: rename directory return 255
[11414.086540] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 23:58:30 (1781236710)
[11447.811467] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 23:59:04 (1781236744)
[11497.931343] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 23:59:53 (1781236793)
[11614.297507] LustreError: 191990:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11654.645245] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 00:02:30 (1781236950)
[11695.061592] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 00:03:11 (1781236991)
[11742.497636] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 00:03:59 (1781237039)
[11931.644759] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 00:07:08 (1781237228)
[11934.028294] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[11935.574311] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 00:07:12 (1781237232)
[12002.981806] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[12004.756708] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 00:08:21 (1781237301)
[12042.085380] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[12045.013875] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 00:09:00 (1781237340)
[12102.109858] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[12104.368289] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 00:10:00 (1781237400)
[12155.523358] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 00:10:52 (1781237452)
[12157.130954] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[12159.173468] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 00:10:55 (1781237455)
[12188.619825] LustreError: 206016:0:(mgs_handler.c:1140:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[12234.097437] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 00:12:10 (1781237530)
[12262.416826] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[12264.655526] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[12480.318248] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 00:16:16 (1781237776)
[12531.303346] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 00:17:07 (1781237827)
[12570.184433] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 00:17:46 (1781237866)
[12572.011473] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[12574.189222] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 00:17:50 (1781237870)
[12576.431969] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[12579.036529] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 00:17:55 (1781237875)
[12598.250800] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[12616.787876] Lustre: DEBUG MARKER: Write...
[12618.902794] Lustre: DEBUG MARKER: Write out of block quota ...
[12679.212529] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 00:19:35 (1781237975)
[12710.467657] Lustre: DEBUG MARKER: set to use default quota
[12712.338936] Lustre: DEBUG MARKER: set default quota
[12714.287331] Lustre: DEBUG MARKER: get default quota
[12721.464993] Lustre: DEBUG MARKER: Test not out of quota
[12727.465085] Lustre: DEBUG MARKER: Test out of quota
[12742.168683] Lustre: DEBUG MARKER: Increase default quota
[12776.234923] Lustre: DEBUG MARKER: Set quota to override default quota
[12776.323528] LustreError: 183008:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1781842875 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12791.219711] Lustre: DEBUG MARKER: Set to use default quota again
[12817.598580] Lustre: DEBUG MARKER: Cleanup
[12897.889466] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 00:23:14 (1781238194)
[12925.976145] Lustre: DEBUG MARKER: set default quota for qpool1
[12927.997221] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[12980.791834] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 00:24:37 (1781238277)
[13074.418224] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 00:26:11 (1781238371)
[13151.571776] Lustre: DEBUG MARKER: Write...
[13155.694960] Lustre: DEBUG MARKER: Write out of block quota ...
[13289.644473] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 00:29:46 (1781238586)
[13339.984397] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 00:30:36 (1781238636)
[13347.970701] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 00:30:44 (1781238644)
[13349.644128] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate
[13351.489371] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 00:30:48 (1781238648)
[13353.123581] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate
[13355.060884] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 00:30:51 (1781238651)
[13375.096544] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 00:31:11 (1781238671)
[13376.806490] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[13378.932693] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 00:31:15 (1781238675)
[13396.654936] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[13403.342647] LustreError: 231117:0:(qmt_pool.c:1399:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[13407.132226] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.40@tcp (stopping)
[13407.201148] LustreError: 3318:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff9f60d8a003c0 x1867750202097152/t0(0) o601->lustre-MDT0000-lwp-MDT0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[13407.225468] LustreError: 3318:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-MDT0000 qtype:usr lqe: ffff9f600a433e00 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 375 qunit:0 qtune:0 edquot:0 default:no revoke:0
[13407.242293] LustreError: 3318:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 3 previous similar messages
[13408.738669] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[13408.761213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[13408.771059] Lustre: Skipped 1 previous similar message
[13412.217636] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.40@tcp (stopping)
[13413.392144] LustreError: 231117:0:(qmt_pool.c:1399:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[13413.947710] Lustre: server umount lustre-MDT0000 complete
[13422.437390] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[13423.125090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[13423.315768] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[13423.321895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[13427.559592] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[13429.226807] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[13429.234647] Lustre: Skipped 1 previous similar message
[13470.148381] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 00:32:46 (1781238766)
[13507.730728] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 00:33:24 (1781238804)
[13543.918311] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 00:34:00 (1781238840)
[13603.406049] Lustre: *** cfs_fail_loc=a08, val=0***
[13603.418192] Lustre: Skipped 11 previous similar messages
[13603.429751] Lustre: *** cfs_fail_loc=a08, val=0***
[13603.434545] Lustre: Skipped 7 previous similar messages
[13729.519688] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 00:37:05 (1781239025)
[13816.964987] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 00:38:33 (1781239113)
[13912.847927] LustreError: 232009:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1781844011 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13921.535420] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed