== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:48:12 (1788749292) start mds service on oleg305-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 6 20:24 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg305-server' oleg305-server: oleg305-server.virtnet: executing load_modules_local oleg305-server: kmemleak disabled oleg305-server: Loading modules from /home/green/git/lustre-release/lustre oleg305-server: detected 4 online CPUs by sysfs oleg305-server: MODOPTS_LIBCFS= oleg305-server: Force libcfs to create 2 CPU partitions oleg305-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg305-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg305-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg305-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg305-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-OST0000 oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 205 (time 1788749362.61 total 10.19 last 20.11) - open/close 392 (time 1788749372.64 total 20.22 last 18.65) - open/close 580 (time 1788749382.71 total 30.29 last 18.67) - open/close 783 (time 1788749392.83 total 40.41 last 20.06) - open/close 1033 (time 1788749402.84 total 50.42 last 24.96) - open/close 1218 (time 1788749412.85 total 60.43 last 18.49) - open/close 1422 (time 1788749423.03 total 70.62 last 20.02) - open/close 1652 (time 1788749433.39 total 80.98 last 22.20) - open/close 1791 (time 1788749443.52 total 91.10 last 13.73) - open/close 2041 (time 1788749454.44 total 102.03 last 22.88) - open/close 2165 (time 1788749464.45 total 112.03 last 12.40) - open/close 2366 (time 1788749475.53 total 123.11 last 18.14) - open/close 2597 (time 1788749486.27 total 133.85 last 21.50) - open/close 2845 (time 1788749497.02 total 144.61 last 23.06) - open/close 3022 (time 1788749507.07 total 154.65 last 17.62) - open/close 3265 (time 1788749517.88 total 165.46 last 22.48) - open/close 3516 (time 1788749528.88 total 176.47 last 22.81) - open/close 3654 (time 1788749539.67 total 187.25 last 12.79) - open/close 3772 (time 1788749549.68 total 197.27 last 11.79) - open/close 3894 (time 1788749560.24 total 207.82 last 11.56) - open/close 4105 (time 1788749570.26 total 217.84 last 21.06) - open/close 4225 (time 1788749580.37 total 227.95 last 11.87) - open/close 4451 (time 1788749591.04 total 238.62 last 21.17) - open/close 4533 (time 1788749601.99 total 249.57 last 7.49) - open/close 4736 (time 1788749612.64 total 260.23 last 19.05) - open/close 4887 (time 1788749623.37 total 270.95 last 14.08) - open/close 5180 (time 1788749633.38 total 280.96 last 29.27) - open/close 5377 (time 1788749643.39 total 290.97 last 19.67) - open/close 5576 (time 1788749653.96 total 301.54 last 18.83) - open/close 5785 (time 1788749664.31 total 311.89 last 20.19) - open/close 6036 (time 1788749674.94 total 322.52 last 23.61) - open/close 6121 (time 1788749685.75 total 333.34 last 7.86) - open/close 6236 (time 1788749695.77 total 343.35 last 11.48) - open/close 6464 (time 1788749706.11 total 353.70 last 22.04) - open/close 6691 (time 1788749716.22 total 363.80 last 22.46) - open/close 6959 (time 1788749726.23 total 373.81 last 26.77) - open/close 7186 (time 1788749736.23 total 383.82 last 22.69) - open/close 7364 (time 1788749746.71 total 394.30 last 16.99) - open/close 7498 (time 1788749757.95 total 405.54 last 11.92) - open/close 7723 (time 1788749768.52 total 416.10 last 21.30) - open/close 7970 (time 1788749779.13 total 426.71 last 23.27) - open/close 8059 (time 1788749791.22 total 438.81 last 7.36) - open/close 8287 (time 1788749801.98 total 449.56 last 21.20) - open/close 8416 (time 1788749812.34 total 459.92 last 12.45) - open/close 8562 (time 1788749822.53 total 470.11 last 14.33) - open/close 8773 (time 1788749833.06 total 480.65 last 20.03) - open/close 9037 (time 1788749843.08 total 490.66 last 26.36) - open/close 9156 (time 1788749853.69 total 501.27 last 11.21) - open/close 9338 (time 1788749864.27 total 511.85 last 17.20) - open/close 9579 (time 1788749874.29 total 521.87 last 24.05) - open/close 9642 (time 1788749884.32 total 531.90 last 6.28) - open/close 9893 (time 1788749895.31 total 542.90 last 22.83) - open/close 10000 (time 1788749899.24 total 546.82 last 27.25) - open/close 10155 (time 1788749909.25 total 556.83 last 15.49) - open/close 10243 (time 1788749919.51 total 567.09 last 8.58) - open/close 10380 (time 1788749930.00 total 577.59 last 13.05) - open/close 10512 (time 1788749940.45 total 588.03 last 12.64) - open/close 10722 (time 1788749951.31 total 598.89 last 19.33) - open/close 10988 (time 1788749961.32 total 608.91 last 26.55) - open/close 11188 (time 1788749971.35 total 618.93 last 19.95) - open/close 11361 (time 1788749981.40 total 628.98 last 17.21) - open/close 11464 (time 1788749991.87 total 639.46 last 9.83) - open/close 11525 (time 1788750001.93 total 649.51 last 6.07) - open/close 11659 (time 1788750012.12 total 659.70 last 13.15) - open/close 11781 (time 1788750022.13 total 669.71 last 12.19) - open/close 11933 (time 1788750032.66 total 680.24 last 14.43) - open/close 12037 (time 1788750043.54 total 691.12 last 9.56) - open/close 12189 (time 1788750054.23 total 701.81 last 14.21) - open/close 12335 (time 1788750064.28 total 711.86 last 14.53) - open/close 12506 (time 1788750074.28 total 721.87 last 17.10) - open/close 12736 (time 1788750084.29 total 731.87 last 22.99) - open/close 12925 (time 1788750095.84 total 743.42 last 16.36) - open/close 13172 (time 1788750106.83 total 754.41 last 22.48) - open/close 13417 (time 1788750116.84 total 764.43 last 24.46) - open/close 13524 (time 1788750129.34 total 776.92 last 8.56) - open/close 13579 (time 1788750139.72 total 787.31 last 5.30) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-13713) error: No space left on device total: 13713 open/close in 791.76 seconds: 17.32 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9030: kill: (75986) - No such process - unlinked 0 (time 1788750156 ; total 0 ; last 0) - unlinked 10000 (time 1788750284 ; total 128 ; last 128) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-13713) error: No such file or directory total: 13713 unlinks in 162 seconds: 84.648148 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg305-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg305-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg305-server stop mds service on oleg305-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg305-server unloading modules via unload_modules_local on: 'oleg305-server' oleg305-server: oleg305-server.virtnet: executing unload_modules_local oleg305-server: kmemleak disabled modules unloaded. pdsh@oleg305-client: oleg305-client: ssh exited with exit code 2 pdsh@oleg305-client: oleg305-server: ssh exited with exit code 2 pdsh@oleg305-client: oleg305-client: ssh exited with exit code 2 pdsh@oleg305-client: oleg305-server: ssh exited with exit code 2