== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:39:05 (1781588345) start mds service on oleg358-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg358-server' oleg358-server: oleg358-server.virtnet: executing load_modules_local oleg358-server: Loading modules from /home/green/git/lustre-release/lustre oleg358-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg358-server: detected 4 online CPUs by sysfs oleg358-server: MODOPTS_LIBCFS= oleg358-server: Force libcfs to create 2 CPU partitions oleg358-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg358-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg358-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg358-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg358-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Started lustre-OST0000 oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg358-client.virtnet: -o user_xattr,flock 192.168.203.158@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 142 (time 1781588455.01 total 10.00 last 14.19) - open/close 355 (time 1781588465.06 total 20.06 last 21.18) - open/close 618 (time 1781588475.07 total 30.07 last 26.28) - open/close 827 (time 1781588486.13 total 41.13 last 18.90) - open/close 1014 (time 1781588496.14 total 51.14 last 18.68) - open/close 1235 (time 1781588506.49 total 61.49 last 21.36) - open/close 1522 (time 1781588516.78 total 71.78 last 27.88) - open/close 1750 (time 1781588527.17 total 82.17 last 21.94) - open/close 1859 (time 1781588537.38 total 92.38 last 10.68) - open/close 2074 (time 1781588547.39 total 102.39 last 21.47) - open/close 2276 (time 1781588557.39 total 112.39 last 20.19) - open/close 2513 (time 1781588567.66 total 122.66 last 23.09) - open/close 2713 (time 1781588577.66 total 132.66 last 19.99) - open/close 2885 (time 1781588587.95 total 142.95 last 16.72) - open/close 3113 (time 1781588597.95 total 152.95 last 22.78) - open/close 3385 (time 1781588608.61 total 163.61 last 25.53) - open/close 3608 (time 1781588618.61 total 173.61 last 22.30) - open/close 3838 (time 1781588629.49 total 184.48 last 21.15) - open/close 4086 (time 1781588639.50 total 194.49 last 24.78) - open/close 4299 (time 1781588650.09 total 205.08 last 20.11) - open/close 4508 (time 1781588660.09 total 215.09 last 20.88) - open/close 4696 (time 1781588670.10 total 225.10 last 18.79) - open/close 4943 (time 1781588680.71 total 235.71 last 23.28) - open/close 5185 (time 1781588690.82 total 245.81 last 23.94) - open/close 5375 (time 1781588701.74 total 256.74 last 17.39) - open/close 5643 (time 1781588712.55 total 267.55 last 24.79) - open/close 5904 (time 1781588722.66 total 277.66 last 25.82) - open/close 6190 (time 1781588733.51 total 288.51 last 26.36) - open/close 6441 (time 1781588744.36 total 299.35 last 23.14) - open/close 6655 (time 1781588754.36 total 309.36 last 21.40) - open/close 6904 (time 1781588764.71 total 319.71 last 24.05) - open/close 7136 (time 1781588775.36 total 330.36 last 21.77) - open/close 7365 (time 1781588785.58 total 340.58 last 22.42) - open/close 7636 (time 1781588798.45 total 353.45 last 21.06) - open/close 7826 (time 1781588808.68 total 363.68 last 18.57) - open/close 8068 (time 1781588819.56 total 374.56 last 22.25) - open/close 8275 (time 1781588830.59 total 385.59 last 18.77) - open/close 8479 (time 1781588840.61 total 395.60 last 20.37) - open/close 8711 (time 1781588850.62 total 405.61 last 23.18) - open/close 8928 (time 1781588860.78 total 415.78 last 21.34) - open/close 9156 (time 1781588871.42 total 426.42 last 21.43) - open/close 9389 (time 1781588881.97 total 436.97 last 22.08) - open/close 9585 (time 1781588891.97 total 446.97 last 19.60) - open/close 9788 (time 1781588902.14 total 457.13 last 19.98) - open/close 10000 (time 1781588910.20 total 465.19 last 26.30) - open/close 10262 (time 1781588920.20 total 475.20 last 26.18) - open/close 10467 (time 1781588932.28 total 487.28 last 16.97) - open/close 10746 (time 1781588942.29 total 497.28 last 27.89) - open/close 10949 (time 1781588952.30 total 507.30 last 20.26) - open/close 11111 (time 1781588962.55 total 517.55 last 15.82) - open/close 11368 (time 1781588972.97 total 527.97 last 24.66) - open/close 11612 (time 1781588983.23 total 538.23 last 23.78) - open/close 11847 (time 1781588993.58 total 548.57 last 22.71) - open/close 12081 (time 1781589003.59 total 558.59 last 23.37) - open/close 12314 (time 1781589013.98 total 568.98 last 22.42) - open/close 12552 (time 1781589023.99 total 578.99 last 23.78) - open/close 12747 (time 1781589034.00 total 589.00 last 19.48) - open/close 12968 (time 1781589044.46 total 599.46 last 21.13) - open/close 13194 (time 1781589055.14 total 610.14 last 21.16) - open/close 13464 (time 1781589065.15 total 620.14 last 26.99) - open/close 13709 (time 1781589075.15 total 630.15 last 24.48) - open/close 13892 (time 1781589085.92 total 640.92 last 16.99) - open/close 14085 (time 1781589096.06 total 651.05 last 19.05) - open/close 14324 (time 1781589106.92 total 661.92 last 21.99) - open/close 14530 (time 1781589117.87 total 672.87 last 18.81) - open/close 14717 (time 1781589127.88 total 682.88 last 18.68) - open/close 14959 (time 1781589137.90 total 692.89 last 24.17) - open/close 15195 (time 1781589148.03 total 703.03 last 23.28) - open/close 15493 (time 1781589158.44 total 713.44 last 28.64) - open/close 15710 (time 1781589168.50 total 723.50 last 21.56) - open/close 15982 (time 1781589178.52 total 733.52 last 27.16) - open/close 16162 (time 1781589189.23 total 744.23 last 16.80) - open/close 16387 (time 1781589199.79 total 754.78 last 21.32) - open/close 16578 (time 1781589209.81 total 764.81 last 19.06) - open/close 16773 (time 1781589219.81 total 774.81 last 19.49) - open/close 16894 (time 1781589230.12 total 785.11 last 11.74) - open/close 17029 (time 1781589241.02 total 796.02 last 12.37) - open/close 17295 (time 1781589251.05 total 806.05 last 26.54) - open/close 17569 (time 1781589262.09 total 817.09 last 24.82) - open/close 17790 (time 1781589272.32 total 827.32 last 21.60) - open/close 18056 (time 1781589282.32 total 837.32 last 26.59) - open/close 18279 (time 1781589293.36 total 848.36 last 20.19) - open/close 18515 (time 1781589303.37 total 858.37 last 23.57) - open/close 18657 (time 1781589313.51 total 868.50 last 14.01) - open/close 18891 (time 1781589323.71 total 878.71 last 22.92) - open/close 19093 (time 1781589333.72 total 888.71 last 20.20) - open/close 19782 (time 1781589343.74 total 898.74 last 68.74) - open/close 20000 (time 1781589347.35 total 902.35 last 60.37) - open/close 20671 (time 1781589357.35 total 912.35 last 67.10) - open/close 21249 (time 1781589367.36 total 922.36 last 57.72) - open/close 21968 (time 1781589377.38 total 932.38 last 71.78) - open/close 22725 (time 1781589387.39 total 942.38 last 75.65) - open/close 23392 (time 1781589397.41 total 952.40 last 66.57) - open/close 24132 (time 1781589407.41 total 962.40 last 74.00) - open/close 24808 (time 1781589417.42 total 972.42 last 67.49) - open/close 25567 (time 1781589427.44 total 982.44 last 75.79) - open/close 26266 (time 1781589437.45 total 992.45 last 69.84) - open/close 26972 (time 1781589447.45 total 1002.45 last 70.59) - open/close 27722 (time 1781589457.45 total 1012.45 last 74.99) - open/close 28501 (time 1781589467.45 total 1022.45 last 77.89) - open/close 29274 (time 1781589477.46 total 1032.46 last 77.21) - open/close 29973 (time 1781589487.46 total 1042.46 last 69.90) - open/close 30721 (time 1781589497.47 total 1052.47 last 74.76) - open/close 31475 (time 1781589507.47 total 1062.47 last 75.36) - open/close 32179 (time 1781589517.48 total 1072.48 last 70.33) - open/close 32964 (time 1781589527.50 total 1082.50 last 78.39) - open/close 33685 (time 1781589537.51 total 1092.50 last 72.04) - open/close 34436 (time 1781589547.51 total 1102.51 last 75.04) - open/close 35154 (time 1781589557.53 total 1112.53 last 71.69) - open/close 35868 (time 1781589567.54 total 1122.53 last 71.35) - open/close 36590 (time 1781589577.54 total 1132.54 last 72.16) - open/close 37299 (time 1781589587.54 total 1142.54 last 70.89) - open/close 37984 (time 1781589597.54 total 1152.54 last 68.50) - open/close 38637 (time 1781589607.55 total 1162.55 last 65.28) - open/close 39375 (time 1781589617.56 total 1172.55 last 73.74) - open/close 40000 (time 1781589626.86 total 1181.86 last 67.13) - open/close 40667 (time 1781589636.89 total 1191.89 last 66.54) - open/close 41357 (time 1781589646.91 total 1201.91 last 68.86) - open/close 42020 (time 1781589656.92 total 1211.92 last 66.23) - open/close 42774 (time 1781589666.93 total 1221.92 last 75.35) - open/close 43457 (time 1781589676.93 total 1231.93 last 68.28) - open/close 44208 (time 1781589686.94 total 1241.94 last 75.01) - open/close 44906 (time 1781589696.95 total 1251.94 last 69.78) - open/close 45670 (time 1781589706.95 total 1261.95 last 76.37) - open/close 46239 (time 1781589716.98 total 1271.98 last 56.71) - open/close 46855 (time 1781589727.01 total 1282.00 last 61.46) - open/close 47581 (time 1781589737.01 total 1292.01 last 72.55) - open/close 48342 (time 1781589747.01 total 1302.01 last 76.08) - open/close 48965 (time 1781589757.03 total 1312.03 last 62.22) - open/close 49516 (time 1781589767.04 total 1322.04 last 55.04) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1323.89 seconds: 37.49 ops/second - unlinked 0 (time 1781589775 ; total 0 ; last 0) - unlinked 10000 (time 1781589917 ; total 142 ; last 142) - unlinked 20000 (time 1781590061 ; total 286 ; last 144) - unlinked 30000 (time 1781590206 ; total 431 ; last 145) - unlinked 40000 (time 1781590347 ; total 572 ; last 141) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 708 seconds: 70.100281 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg358-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg358-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg358-server stop mds service on oleg358-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg358-server stop mds service on oleg358-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg358-server unloading modules via unload_modules_local on: 'oleg358-server' oleg358-server: oleg358-server.virtnet: executing unload_modules_local oleg358-server: modules unloaded. pdsh@oleg358-client: oleg358-client: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-server: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-client: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-server: ssh exited with exit code 2