== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:47:03 (1788446823) start mds service on oleg206-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg206-server' oleg206-server: oleg206-server.virtnet: executing load_modules_local oleg206-server: Loading modules from /home/green/git/lustre-release/lustre oleg206-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg206-server: detected 4 online CPUs by sysfs oleg206-server: MODOPTS_LIBCFS= oleg206-server: Force libcfs to create 2 CPU partitions oleg206-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg206-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg206-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg206-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg206-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-OST0000 oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg206-client.virtnet: -o user_xattr,flock 192.168.202.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 231 (time 1788446935.88 total 13.08 last 17.66) - open/close 432 (time 1788446948.49 total 25.69 last 15.94) - open/close 675 (time 1788446958.50 total 35.70 last 24.28) - open/close 897 (time 1788446968.91 total 46.11 last 21.32) - open/close 1192 (time 1788446978.92 total 56.11 last 29.49) - open/close 1420 (time 1788446989.13 total 66.32 last 22.33) - open/close 1687 (time 1788446999.29 total 76.49 last 26.27) - open/close 1939 (time 1788447009.74 total 86.93 last 24.12) - open/close 2102 (time 1788447020.34 total 97.54 last 15.36) - open/close 2317 (time 1788447030.35 total 107.54 last 21.49) - open/close 2567 (time 1788447041.08 total 118.28 last 23.29) - open/close 2771 (time 1788447051.94 total 129.14 last 18.79) - open/close 3016 (time 1788447063.33 total 140.52 last 21.52) - open/close 3217 (time 1788447073.88 total 151.08 last 19.04) - open/close 3411 (time 1788447084.95 total 162.15 last 17.53) - open/close 3672 (time 1788447095.29 total 172.49 last 25.24) - open/close 3913 (time 1788447105.34 total 182.54 last 23.98) - open/close 4131 (time 1788447115.34 total 192.54 last 21.80) - open/close 4350 (time 1788447125.37 total 202.57 last 21.83) - open/close 4630 (time 1788447136.33 total 213.52 last 25.56) - open/close 4909 (time 1788447147.34 total 224.53 last 25.34) - open/close 5155 (time 1788447157.96 total 235.15 last 23.17) - open/close 5407 (time 1788447168.19 total 245.39 last 24.62) - open/close 5665 (time 1788447178.20 total 255.39 last 25.79) - open/close 5826 (time 1788447188.21 total 265.41 last 16.07) - open/close 6034 (time 1788447198.53 total 275.73 last 20.16) - open/close 6226 (time 1788447208.53 total 285.73 last 19.19) - open/close 6400 (time 1788447218.53 total 295.73 last 17.40) - open/close 6615 (time 1788447228.94 total 306.13 last 20.67) - open/close 6801 (time 1788447238.96 total 316.16 last 18.55) - open/close 7002 (time 1788447249.30 total 326.50 last 19.44) - open/close 7206 (time 1788447259.75 total 336.95 last 19.53) - open/close 7406 (time 1788447270.53 total 347.72 last 18.56) - open/close 7674 (time 1788447280.84 total 358.03 last 25.99) - open/close 7966 (time 1788447290.95 total 368.15 last 28.86) - open/close 8198 (time 1788447301.13 total 378.33 last 22.80) - open/close 8481 (time 1788447311.87 total 389.06 last 26.36) - open/close 8750 (time 1788447322.33 total 399.52 last 25.72) - open/close 9035 (time 1788447332.90 total 410.09 last 26.97) - open/close 9267 (time 1788447343.55 total 420.74 last 21.78) - open/close 9538 (time 1788447353.70 total 430.90 last 26.68) - open/close 9740 (time 1788447369.30 total 446.49 last 12.95) - open/close 9842 (time 1788447379.52 total 456.72 last 9.97) - open/close 10000 (time 1788447385.75 total 462.94 last 25.40) - open/close 10132 (time 1788447395.76 total 472.96 last 13.17) - open/close 10358 (time 1788447406.52 total 483.72 last 21.00) - open/close 10564 (time 1788447417.47 total 494.67 last 18.82) - open/close 10703 (time 1788447429.32 total 506.52 last 11.73) - open/close 10919 (time 1788447440.21 total 517.40 last 19.85) - open/close 11166 (time 1788447450.37 total 527.57 last 24.30) - open/close 11410 (time 1788447460.38 total 537.58 last 24.37) - open/close 11684 (time 1788447470.38 total 547.58 last 27.40) - open/close 11812 (time 1788447481.03 total 558.22 last 12.03) - open/close 12094 (time 1788447491.05 total 568.25 last 28.13) - open/close 12172 (time 1788447501.06 total 578.25 last 7.80) - open/close 12437 (time 1788447511.06 total 588.26 last 26.49) - open/close 12642 (time 1788447521.08 total 598.28 last 20.46) - open/close 12854 (time 1788447531.08 total 608.28 last 21.19) - open/close 13065 (time 1788447541.41 total 618.60 last 20.44) - open/close 13167 (time 1788447551.43 total 628.62 last 10.18) - open/close 13318 (time 1788447562.46 total 639.65 last 13.69) - open/close 13464 (time 1788447572.69 total 649.89 last 14.27) - open/close 13622 (time 1788447582.76 total 659.96 last 15.69) - open/close 13850 (time 1788447594.27 total 671.47 last 19.81) - open/close 14027 (time 1788447604.31 total 681.50 last 17.64) - open/close 14204 (time 1788447614.53 total 691.72 last 17.32) - open/close 14324 (time 1788447624.54 total 701.74 last 11.98) - open/close 14575 (time 1788447635.28 total 712.48 last 23.37) - open/close 14795 (time 1788447645.57 total 722.76 last 21.38) - open/close 14989 (time 1788447656.27 total 733.47 last 18.13) - open/close 15162 (time 1788447666.37 total 743.57 last 17.13) - open/close 15335 (time 1788447676.38 total 753.57 last 17.29) - open/close 15449 (time 1788447687.26 total 764.45 last 10.48) - open/close 15624 (time 1788447698.06 total 775.26 last 16.20) - open/close 15839 (time 1788447708.07 total 785.27 last 21.48) - open/close 16112 (time 1788447718.86 total 796.06 last 25.31) - open/close 16264 (time 1788447729.08 total 806.28 last 14.87) - open/close 16426 (time 1788447739.09 total 816.29 last 16.18) - open/close 16655 (time 1788447749.40 total 826.60 last 22.20) - open/close 16808 (time 1788447759.41 total 836.61 last 15.29) - open/close 17020 (time 1788447770.08 total 847.27 last 19.88) - open/close 17290 (time 1788447781.90 total 859.10 last 22.83) - open/close 17504 (time 1788447791.91 total 869.11 last 21.38) - open/close 17794 (time 1788447801.92 total 879.11 last 28.99) - open/close 18065 (time 1788447812.61 total 889.80 last 25.35) - open/close 18316 (time 1788447823.27 total 900.47 last 23.53) - open/close 18589 (time 1788447833.29 total 910.49 last 27.26) - open/close 18688 (time 1788447843.71 total 920.91 last 9.50) - open/close 18822 (time 1788447853.73 total 930.93 last 13.37) - open/close 19017 (time 1788447866.60 total 943.80 last 15.14) - open/close 19177 (time 1788447876.88 total 954.08 last 15.57) - open/close 19334 (time 1788447886.98 total 964.18 last 15.55) - open/close 19411 (time 1788447898.34 total 975.54 last 6.78) - open/close 19574 (time 1788447909.40 total 986.59 last 14.74) - open/close 19998 (time 1788447919.40 total 996.59 last 42.40) - open/close 20608 (time 1788447929.41 total 1006.61 last 60.91) - open/close 21329 (time 1788447939.41 total 1016.61 last 72.07) - open/close 22188 (time 1788447949.42 total 1026.62 last 85.81) - open/close 23163 (time 1788447959.43 total 1036.62 last 97.47) - open/close 23851 (time 1788447969.43 total 1046.62 last 68.80) - open/close 24525 (time 1788447979.45 total 1056.64 last 67.26) - open/close 25433 (time 1788447989.45 total 1066.64 last 90.79) - open/close 26256 (time 1788447999.45 total 1076.65 last 82.29) - open/close 27173 (time 1788448009.46 total 1086.65 last 91.63) - open/close 28010 (time 1788448019.47 total 1096.66 last 83.62) - open/close 28809 (time 1788448029.47 total 1106.67 last 79.87) - open/close 29514 (time 1788448039.48 total 1116.68 last 70.43) - open/close 30000 (time 1788448045.61 total 1122.80 last 79.32) - open/close 30951 (time 1788448055.61 total 1132.81 last 95.05) - open/close 31798 (time 1788448065.62 total 1142.82 last 84.59) - open/close 32575 (time 1788448075.63 total 1152.83 last 77.64) - open/close 33304 (time 1788448085.64 total 1162.83 last 72.86) - open/close 34143 (time 1788448095.64 total 1172.83 last 83.90) - open/close 35140 (time 1788448105.64 total 1182.84 last 99.69) - open/close 36019 (time 1788448115.66 total 1192.85 last 87.74) - open/close 36866 (time 1788448125.67 total 1202.86 last 84.62) - open/close 37749 (time 1788448135.69 total 1212.89 last 88.09) - open/close 38510 (time 1788448145.69 total 1222.89 last 76.08) - open/close 39399 (time 1788448155.71 total 1232.91 last 88.70) - open/close 40000 (time 1788448164.71 total 1241.91 last 66.80) - open/close 40606 (time 1788448174.73 total 1251.92 last 60.52) - open/close 41287 (time 1788448184.73 total 1261.93 last 68.05) - open/close 42178 (time 1788448194.74 total 1271.93 last 89.06) - open/close 43032 (time 1788448204.74 total 1281.94 last 85.36) - open/close 43816 (time 1788448214.75 total 1291.94 last 78.36) - open/close 44604 (time 1788448224.76 total 1301.96 last 78.69) - open/close 45523 (time 1788448234.77 total 1311.96 last 91.84) - open/close 46448 (time 1788448244.77 total 1321.97 last 92.44) - open/close 47436 (time 1788448254.78 total 1331.98 last 98.73) - open/close 48190 (time 1788448264.80 total 1342.00 last 75.24) - open/close 48925 (time 1788448274.80 total 1352.00 last 73.48) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1359.13 seconds: 36.52 ops/second - unlinked 0 (time 1788448286 ; total 0 ; last 0) - unlinked 10000 (time 1788448414 ; total 128 ; last 128) - unlinked 20000 (time 1788448539 ; total 253 ; last 125) - unlinked 30000 (time 1788448666 ; total 380 ; last 127) - unlinked 40000 (time 1788448781 ; total 495 ; last 115) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 600 seconds: 82.718330 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg206-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg206-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg206-server stop mds service on oleg206-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg206-server stop mds service on oleg206-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg206-server unloading modules via unload_modules_local on: 'oleg206-server' oleg206-server: oleg206-server.virtnet: executing unload_modules_local oleg206-server: modules unloaded. pdsh@oleg206-client: oleg206-client: ssh exited with exit code 2 pdsh@oleg206-client: oleg206-server: ssh exited with exit code 2 pdsh@oleg206-client: oleg206-server: ssh exited with exit code 2 pdsh@oleg206-client: oleg206-client: ssh exited with exit code 2