[ 5763.865601] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5771.232703] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5775.298323] LustreError: 124822:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5776.409110] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5778.425770] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5780.721426] Lustre: Failing over lustre-MDT0000
[ 5781.279379] Lustre: server umount lustre-MDT0000 complete
[ 5801.230845] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5801.402419] LustreError: 125473:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5801.559768] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5801.737258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5801.811639] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5802.784223] Lustre: 3279:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788630092/real 1788630092]  req@ffff88c7c7837100 x1875508581168768/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788630108 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5802.825173] Lustre: 3279:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5803.025856] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5803.099659] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5803.136705] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:518 to 0x240000400:545)
[ 5803.143800] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:514 to 0x280000400:545)
[ 5806.694632] Lustre: 3282:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788630097/real 1788630097]  req@ffff88c6c13a0e00 x1875508581169024/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788630113 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5806.737210] Lustre: 3282:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5806.848534] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5807.086167] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5812.204608] Lustre: 3281:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788630102/real 1788630102]  req@ffff88c7fb3caa00 x1875508581169536/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788630118 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5812.239440] Lustre: 3281:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5816.291418] Lustre: 3280:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788630107/real 1788630107]  req@ffff88c7c2d9fb80 x1875508581169792/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788630123 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5816.338284] Lustre: 3280:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5817.517440] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5819.628941] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5824.494898] Lustre: DEBUG MARKER: (dd_pid=117241, time=0, timeout=600)
[ 5879.076660] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5888.823948] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5893.735118] LustreError: 127738:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5895.085531] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5897.005319] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5900.161969] Lustre: Failing over lustre-MDT0000
[ 5901.002135] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 5901.015288] LustreError: Skipped 1 previous similar message
[ 5901.018264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5902.733886] Lustre: server umount lustre-MDT0000 complete
[ 5920.736174] Lustre: 3282:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788630211/real 1788630211]  req@ffff88c7f3a6bb80 x1875508581207936/t0(0) o400->MGC192.168.204.110@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788630227 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5920.772505] Lustre: 3282:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5920.777240] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5930.977680] LustreError: 3278:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff88c7057ece00 x1875508581211136/t0(0) o250->MGC192.168.204.110@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5931.704153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5931.797973] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5937.099086] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5939.762647] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 5939.773304] Lustre: Skipped 1 previous similar message
[ 5944.611363] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5944.864080] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5944.922750] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:547 to 0x240000400:577)
[ 5944.923234] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:514 to 0x280000400:577)
[ 5951.462665] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5953.170932] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5964.936604] Lustre: DEBUG MARKER: (dd_pid=119679, time=5, timeout=600)
[ 6039.389739] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 13:45:43 (1788630343)
[ 6058.082631] LustreError: 3280:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 6066.227089] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6068.465701] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6075.702611] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6078.331392] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6114.608373] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 13:46:59 (1788630419)
[ 6146.171524] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 6165.908850] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 6167.827808] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 6169.698669] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 6172.009591] Lustre: DEBUG MARKER: Set quota for 1 times
[ 6176.157115] Lustre: DEBUG MARKER: Set quota for 2 times
[ 6179.795323] Lustre: DEBUG MARKER: Set quota for 3 times
[ 6184.015087] Lustre: DEBUG MARKER: Set quota for 4 times
[ 6188.371955] Lustre: DEBUG MARKER: Set quota for 5 times
[ 6193.050843] Lustre: DEBUG MARKER: Set quota for 6 times
[ 6197.568654] Lustre: DEBUG MARKER: Set quota for 7 times
[ 6238.065674] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 13:49:03 (1788630543)
[ 6253.537703] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6253.557528] Lustre: Skipped 3 previous similar messages
[ 6253.561543] LustreError: 129092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6253.582758] LustreError: 129092:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message
[ 6254.797874] Lustre: server umount lustre-MDT0000 complete
[ 6260.141908] LustreError: 24814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630566 with bad export cookie 10120826505359239144
[ 6260.142185] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6260.153950] LustreError: 24814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 6260.303165] Lustre: server umount lustre-OST0000 complete
[ 6265.728663] Lustre: server umount lustre-OST0001 complete
[ 6287.133622] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local
[ 6298.031894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6303.096535] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6306.588834] Lustre: 137525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6313.310683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6320.214303] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6322.498196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:580 to 0x240000400:609)
[ 6329.001934] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6334.472287] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:579 to 0x280000400:609)
[ 6335.479920] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6343.878935] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6348.078167] Lustre: 139462:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6364.547121] Lustre: server umount lustre-MDT0000 complete
[ 6369.658481] LustreError: 137071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630676 with bad export cookie 10120826505359248664
[ 6369.678590] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6380.092985] Lustre: server umount lustre-OST0000 complete
[ 6380.498499] Lustre: 3279:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788630671/real 1788630671]  req@ffff88c7026ce680 x1875508581375744/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788630687 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6380.536851] Lustre: 3279:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 6385.013710] Lustre: server umount lustre-OST0001 complete
[ 6402.071201] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local
[ 6411.338114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6415.744078] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6418.898592] Lustre: 142112:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6432.699207] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6436.202142] LustreError: 142477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6436.236837] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:580 to 0x240000400:641)
[ 6436.239247] LustreError: 142477:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages
[ 6441.972359] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6441.976906] Lustre: Skipped 1 previous similar message
[ 6447.107364] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:579 to 0x280000400:641)
[ 6448.685462] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6456.041246] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6459.478511] Lustre: 144024:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6469.337749] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 13:52:54 (1788630774)
[ 6470.814793] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6472.658075] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 13:52:57 (1788630777)
[ 6531.159777] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 13:53:56 (1788630836)
[ 6589.714814] Lustre: DEBUG MARKER: Write...
[ 6592.827948] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6657.777662] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 13:56:02 (1788630962)
[ 6665.018374] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 13:56:10 (1788630970)
[ 6675.944547] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 13:56:20 (1788630980)
[ 6685.953319] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 13:56:30 (1788630990)
[ 6695.418280] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 13:56:40 (1788631000)
[ 6761.524437] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6935.436935] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 7132.177496] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 14:03:56 (1788631436)
[ 7196.660293] Lustre: DEBUG MARKER: Restart...
[ 7200.698632] Lustre: server umount lustre-MDT0000 complete
[ 7204.690508] LustreError: 141667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631511 with bad export cookie 10120826505359250197
[ 7204.701309] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7215.176305] Lustre: server umount lustre-OST0000 complete
[ 7219.169126] Lustre: 3281:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788631509/real 1788631509]  req@ffff88c7e9d5a680 x1875508581603840/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788631525 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7219.211660] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7219.228047] Lustre: Skipped 2 previous similar messages
[ 7219.477844] Lustre: server umount lustre-OST0001 complete
[ 7234.057217] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local
[ 7243.759349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7248.030906] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7251.434593] Lustre: 162574:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7256.966146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7261.957654] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7269.238661] LustreError: 162947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7269.266275] LustreError: 162947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message
[ 7269.284355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:651 to 0x240000400:673)
[ 7269.631840] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7275.020306] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:649 to 0x280000400:673)
[ 7275.603772] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7283.296711] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7286.646408] Lustre: 164493:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7354.729238] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 14:07:39 (1788631659)
[ 7430.340173] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 14:08:54 (1788631734)
[ 9366.448381] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 14:41:11 (1788633671)
[ 9384.420197] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9384.429341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9384.430406] Lustre: Skipped 1 previous similar message
[ 9384.452458] Lustre: Skipped 1 previous similar message
[ 9387.660113] Lustre: server umount lustre-MDT0000 complete
[ 9392.250252] LustreError: 162125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788633699 with bad export cookie 10120826505359256693
[ 9392.256357] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9392.262477] LustreError: 162125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 9392.418066] Lustre: server umount lustre-OST0000 complete
[ 9397.477190] Lustre: server umount lustre-OST0001 complete
[ 9417.191461] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local
[ 9428.267192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9432.771775] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9435.750308] Lustre: 173009:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9442.646854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9449.686172] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9452.835541] LustreError: 173384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9452.853954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5676 to 0x240000400:5697)
[ 9458.985854] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9464.310327] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5674 to 0x280000400:5697)
[ 9466.010964] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9474.632047] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9478.397382] Lustre: 174937:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9511.901368] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 14:43:36 (1788633816)
[ 9556.732393] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 14:44:21 (1788633861)
[ 9601.018396] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 14:45:06 (1788633906)
[ 9602.444958] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9604.179218] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 14:45:09 (1788633909)
[ 9605.502315] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9607.296910] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 14:45:12 (1788633912)
[ 9696.009809] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 14:46:40 (1788634000)
[ 9747.377540] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 14:47:32 (1788634052)
[ 9835.629586] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 14:49:00 (1788634140)
[ 9849.217892] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9849.223926] Lustre: Skipped 2 previous similar messages
[ 9851.233170] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9851.236515] Lustre: Skipped 59 previous similar messages
[ 9855.294095] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9855.304293] Lustre: Skipped 107 previous similar messages
[ 9863.320818] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9863.326121] Lustre: Skipped 317 previous similar messages
[ 9879.321487] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9879.325450] Lustre: Skipped 640 previous similar messages
[ 9911.328230] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9911.333554] Lustre: Skipped 1302 previous similar messages
[ 9975.347502] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9975.349750] Lustre: Skipped 2663 previous similar messages
[10388.261800] Lustre: *** cfs_fail_loc=a08, val=0***
[10388.266236] Lustre: *** cfs_fail_loc=a08, val=0***
[10388.282722] Lustre: Skipped 2907 previous similar messages
[10388.305409] Lustre: Skipped 13 previous similar messages
[10668.837864] Lustre: *** cfs_fail_loc=a08, val=0***
[10668.844462] Lustre: *** cfs_fail_loc=a08, val=0***
[10668.850937] Lustre: Skipped 21 previous similar messages
[10668.884415] Lustre: Skipped 7 previous similar messages
[10693.089889] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10693.114347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10693.124921] Lustre: Skipped 1 previous similar message
[10696.596155] Lustre: server umount lustre-MDT0000 complete
[10701.696872] LustreError: 174939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788635008 with bad export cookie 10120826505361038354
[10701.723246] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10702.164364] Lustre: server umount lustre-OST0000 complete
[10707.118990] Lustre: server umount lustre-OST0001 complete
[10715.289120] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_hostid
[10724.599677] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local
[10735.772794]  vdc: vdc1 vdc9
[10746.167460]  vde: vde1 vde9
[10757.523986]  vdf: vdf1 vdf9
[10775.596301] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing load_modules_local
[10786.644666] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[10786.915137] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[10787.024645] Lustre: lustre-MDT0000: new disk, initializing
[10787.569796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10787.669593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[10794.016069] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10799.656592] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[10807.327748] Lustre: lustre-OST0000: new disk, initializing
[10807.335809] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[10807.346729] Lustre: Skipped 1 previous similar message
[10807.496479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10809.149202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[10809.158986] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[10809.358310] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[10815.627404] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10827.614883] Lustre: lustre-OST0001: new disk, initializing
[10827.621396] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[10827.705974] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10829.190370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[10829.195812] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[10829.268505] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[10835.475620] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10846.535675] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10850.013657] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[10891.161588] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 15:06:36 (1788635196)
[10898.523488] Lustre: DEBUG MARKER: == sanity-quota test 49c: lfs quota long options don't consume an extra argument ========================================================== 15:06:43 (1788635203)
[10906.658821] Lustre: DEBUG MARKER: == sanity-quota test 49d: lfs quota -d and --delimiter both work ========================================================== 15:06:51 (1788635211)
[10914.705829] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 15:06:59 (1788635219)
[10969.347826] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 15:07:54 (1788635274)
[11034.737213] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 15:08:59 (1788635339)
[11068.948174] Lustre: DEBUG MARKER: rename directory return 255
[11119.755472] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 15:10:24 (1788635424)
[11157.343993] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 15:11:01 (1788635461)
[11207.357602] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 15:11:52 (1788635512)
[11316.907124] LustreError: 194084:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11359.088782] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 15:14:24 (1788635664)
[11399.081654] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 15:15:04 (1788635704)
[11445.090019] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 15:15:49 (1788635749)
[11594.110771] LustreError: 196702:0:(tgt_grant.c:788:tgt_grant_check()) lustre-OST0001: cli 4b40f12b-ca2a-44d1-894c-6721420803c6 claims 1703936 GRANT, real grant 0
[11594.700974] LustreError: 205111:0:(tgt_grant.c:788:tgt_grant_check()) lustre-OST0001: cli 4b40f12b-ca2a-44d1-894c-6721420803c6 claims 1703936 GRANT, real grant 0
[11594.717891] LustreError: 205111:0:(tgt_grant.c:788:tgt_grant_check()) Skipped 9 previous similar messages
[11610.713112] LustreError: 205107:0:(tgt_grant.c:788:tgt_grant_check()) lustre-OST0001: cli 4b40f12b-ca2a-44d1-894c-6721420803c6 claims 1703936 GRANT, real grant 0
[11610.722141] LustreError: 205107:0:(tgt_grant.c:788:tgt_grant_check()) Skipped 6 previous similar messages
[11654.555574] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 15:19:19 (1788635959)
[11656.271812] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[11658.382560] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 15:19:23 (1788635963)
[11724.318942] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[11726.570154] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 15:20:31 (1788636031)
[11760.946839] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[11762.741964] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 15:21:07 (1788636067)
[11820.254395] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[11822.333052] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 15:22:07 (1788636127)
[11872.867338] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 15:22:57 (1788636177)
[11874.381833] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[11876.975175] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 15:23:01 (1788636181)
[11917.735140] LustreError: 213134:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[11967.302734] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 15:24:31 (1788636271)
[11996.451267] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[11998.593197] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[12210.000926] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 15:28:34 (1788636514)
[12269.348440] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 15:29:34 (1788636574)
[12307.293144] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 15:30:11 (1788636611)
[12309.436905] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[12311.760321] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 15:30:16 (1788636616)
[12314.079299] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[12316.214143] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 15:30:20 (1788636620)
[12334.999462] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[12355.016417] Lustre: DEBUG MARKER: Write...
[12358.941141] Lustre: DEBUG MARKER: Write out of block quota ...
[12425.052715] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 15:32:09 (1788636729)
[12458.843854] Lustre: DEBUG MARKER: set to use default quota
[12460.734635] Lustre: DEBUG MARKER: set default quota
[12462.900923] Lustre: DEBUG MARKER: get default quota
[12470.567437] Lustre: DEBUG MARKER: Test not out of quota
[12478.375147] Lustre: DEBUG MARKER: Test out of quota
[12496.812857] Lustre: DEBUG MARKER: Increase default quota
[12533.395655] Lustre: DEBUG MARKER: Set quota to override default quota
[12533.471523] LustreError: 192554:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1789241640 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12548.940626] Lustre: DEBUG MARKER: Set to use default quota again
[12574.821929] Lustre: DEBUG MARKER: Cleanup
[12620.602591] LustreError: 3282:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-OST0001: Delaying reintegration for qtype:0 until pending updates are flushed.
[12662.857325] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 15:36:07 (1788636967)
[12690.800890] Lustre: DEBUG MARKER: set default quota for qpool1
[12692.727311] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[12753.001656] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 15:37:37 (1788637057)
[12850.460450] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 15:39:14 (1788637154)
[12933.632818] Lustre: DEBUG MARKER: Write...
[12937.565528] Lustre: DEBUG MARKER: Write out of block quota ...
[13080.103984] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 15:43:05 (1788637385)
[13129.708934] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 15:43:54 (1788637434)
[13138.248956] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 15:44:02 (1788637442)
[13141.866789] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A fallocate not supported
[13144.150953] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 15:44:08 (1788637448)
[13147.421316] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a fallocate not supported
[13149.740263] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 15:44:14 (1788637454)
[13172.651191] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 15:44:37 (1788637477)
[13174.621824] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[13176.684520] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 15:44:41 (1788637481)
[13195.687727] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[13202.203295] LustreError: 238369:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[13204.961989] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[13204.962771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[13204.971195] Lustre: Skipped 2 previous similar messages
[13204.986765] Lustre: Skipped 1 previous similar message
[13207.360767] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.10@tcp (stopping)
[13210.081240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[13210.092485] Lustre: Skipped 1 previous similar message
[13212.208255] LustreError: 238369:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[13212.454227] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.10@tcp (stopping)
[13212.621940] Lustre: server umount lustre-MDT0000 complete
[13221.492939] LustreError: MGC192.168.204.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[13221.902447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[13221.994945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[13222.001604] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[13227.309952] Lustre: DEBUG MARKER: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[13235.175092] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[13235.183637] Lustre: Skipped 1 previous similar message
[13270.332176] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 15:46:15 (1788637575)
[13309.099772] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 15:46:53 (1788637613)
[13346.520308] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 15:47:30 (1788637650)
[13401.521682] Lustre: *** cfs_fail_loc=a08, val=0***
[13401.525032] Lustre: Skipped 35 previous similar messages
[13401.537698] Lustre: *** cfs_fail_loc=a08, val=0***
[13401.548577] Lustre: Skipped 37 previous similar messages
[13520.253877] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 15:50:24 (1788637824)
[13604.509723] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 15:51:49 (1788637909)
[13690.236800] LustreError: 240223:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1789242797 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13890.993735] LustreError: 238953:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1789242997 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13900.444630] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed