== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:18:11 (1782760691) start mds service on oleg416-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: MODOPTS_LIBCFS= oleg416-server: Force libcfs to create 2 CPU partitions oleg416-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg416-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg416-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg416-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg416-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-OST0000 oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg416-client.virtnet: -o user_xattr,flock 192.168.204.116@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 229 (time 1782760820.63 total 10.73 last 21.35) - open/close 437 (time 1782760830.64 total 20.74 last 20.78) - open/close 693 (time 1782760841.04 total 31.14 last 24.62) - open/close 877 (time 1782760851.16 total 41.26 last 18.18) - open/close 1075 (time 1782760861.60 total 51.69 last 18.97) - open/close 1248 (time 1782760871.62 total 61.71 last 17.27) - open/close 1439 (time 1782760882.21 total 72.30 last 18.03) - open/close 1604 (time 1782760892.22 total 82.31 last 16.48) - open/close 1738 (time 1782760902.34 total 92.43 last 13.24) - open/close 1918 (time 1782760912.35 total 102.44 last 17.99) - open/close 1982 (time 1782760922.35 total 112.45 last 6.40) - open/close 2142 (time 1782760932.88 total 122.98 last 15.19) - open/close 2279 (time 1782760943.02 total 133.12 last 13.51) - open/close 2442 (time 1782760954.08 total 144.17 last 14.74) - open/close 2520 (time 1782760964.26 total 154.35 last 7.66) - open/close 2668 (time 1782760975.59 total 165.69 last 13.06) - open/close 2808 (time 1782760985.88 total 175.98 last 13.61) - open/close 3047 (time 1782760996.89 total 186.98 last 21.72) - open/close 3268 (time 1782761007.07 total 197.16 last 21.71) - open/close 3467 (time 1782761019.01 total 209.10 last 16.67) - open/close 3695 (time 1782761029.01 total 219.10 last 22.80) - open/close 3942 (time 1782761039.58 total 229.67 last 23.36) - open/close 4211 (time 1782761050.59 total 240.68 last 24.43) - open/close 4448 (time 1782761060.59 total 250.69 last 23.69) - open/close 4674 (time 1782761070.61 total 260.70 last 22.57) - open/close 4888 (time 1782761082.35 total 272.45 last 18.23) - open/close 5149 (time 1782761092.89 total 282.99 last 24.76) - open/close 5332 (time 1782761102.90 total 292.99 last 18.29) - open/close 5554 (time 1782761113.57 total 303.66 last 20.81) - open/close 5736 (time 1782761124.32 total 314.42 last 16.92) - open/close 5990 (time 1782761134.33 total 324.42 last 25.39) - open/close 6219 (time 1782761145.12 total 335.21 last 21.22) - open/close 6493 (time 1782761155.73 total 345.82 last 25.83) - open/close 6651 (time 1782761165.98 total 356.07 last 15.42) - open/close 6905 (time 1782761177.02 total 367.11 last 23.01) - open/close 7180 (time 1782761187.68 total 377.78 last 25.78) - open/close 7434 (time 1782761198.49 total 388.59 last 23.50) - open/close 7739 (time 1782761208.50 total 398.60 last 30.47) - open/close 8008 (time 1782761218.52 total 408.61 last 26.85) - open/close 8227 (time 1782761228.52 total 418.62 last 21.89) - open/close 8490 (time 1782761239.38 total 429.47 last 24.23) - open/close 8757 (time 1782761249.39 total 439.48 last 26.67) - open/close 9006 (time 1782761260.43 total 450.52 last 22.56) - open/close 9190 (time 1782761270.63 total 460.73 last 18.03) - open/close 9430 (time 1782761281.24 total 471.33 last 22.63) - open/close 9673 (time 1782761292.18 total 482.28 last 22.20) - open/close 9927 (time 1782761302.19 total 492.28 last 25.39) - open/close 10000 (time 1782761305.54 total 495.63 last 21.78) - open/close 10182 (time 1782761316.40 total 506.49 last 16.76) - open/close 10430 (time 1782761326.41 total 516.50 last 24.77) - open/close 10642 (time 1782761336.41 total 526.51 last 21.19) - open/close 10910 (time 1782761347.36 total 537.46 last 24.48) - open/close 11161 (time 1782761357.36 total 547.46 last 25.10) - open/close 11364 (time 1782761367.52 total 557.61 last 19.99) - open/close 11616 (time 1782761377.55 total 567.64 last 25.13) - open/close 11929 (time 1782761388.45 total 578.55 last 28.70) - open/close 12165 (time 1782761398.74 total 588.83 last 22.95) - open/close 12418 (time 1782761409.65 total 599.75 last 23.17) - open/close 12632 (time 1782761420.12 total 610.22 last 20.44) - open/close 12823 (time 1782761430.25 total 620.34 last 18.86) - open/close 12978 (time 1782761440.75 total 630.85 last 14.76) - open/close 13170 (time 1782761450.76 total 640.85 last 19.19) - open/close 13383 (time 1782761461.60 total 651.70 last 19.64) - open/close 13518 (time 1782761474.31 total 664.40 last 10.63) - open/close 13629 (time 1782761484.65 total 674.75 last 10.73) - open/close 13682 (time 1782761495.66 total 685.75 last 4.82) - open/close 13801 (time 1782761506.71 total 696.81 last 10.76) - open/close 13986 (time 1782761516.72 total 706.82 last 18.48) - open/close 14163 (time 1782761527.33 total 717.42 last 16.69) - open/close 14342 (time 1782761537.34 total 727.44 last 17.87) - open/close 14512 (time 1782761547.36 total 737.45 last 16.98) - open/close 14678 (time 1782761557.36 total 747.46 last 16.59) - open/close 14858 (time 1782761567.54 total 757.63 last 17.68) - open/close 15029 (time 1782761577.58 total 767.67 last 17.04) - open/close 15267 (time 1782761587.59 total 777.68 last 23.77) - open/close 15376 (time 1782761598.20 total 788.29 last 10.28) - open/close 15463 (time 1782761608.41 total 798.50 last 8.52) - open/close 15547 (time 1782761620.37 total 810.47 last 7.02) - open/close 15637 (time 1782761630.64 total 820.73 last 8.77) - open/close 15778 (time 1782761641.46 total 831.55 last 13.04) - open/close 15874 (time 1782761653.09 total 843.18 last 8.26) - open/close 16033 (time 1782761665.96 total 856.06 last 12.35) - open/close 16170 (time 1782761676.08 total 866.18 last 13.53) - open/close 16363 (time 1782761686.09 total 876.19 last 19.28) - open/close 16579 (time 1782761696.34 total 886.44 last 21.07) - open/close 16789 (time 1782761706.98 total 897.08 last 19.74) - open/close 17016 (time 1782761717.72 total 907.81 last 21.15) - open/close 17128 (time 1782761730.67 total 920.76 last 8.65) - open/close 17317 (time 1782761740.67 total 930.76 last 18.90) - open/close 17385 (time 1782761750.89 total 940.99 last 6.65) - open/close 17508 (time 1782761761.71 total 951.80 last 11.37) - open/close 17764 (time 1782761772.40 total 962.49 last 23.95) - open/close 17896 (time 1782761782.40 total 972.49 last 13.20) - open/close 18040 (time 1782761792.40 total 982.49 last 14.40) - open/close 18542 (time 1782761802.40 total 992.49 last 50.19) - open/close 19225 (time 1782761812.40 total 1002.49 last 68.29) - open/close 19961 (time 1782761822.41 total 1012.50 last 73.55) - open/close 20663 (time 1782761832.42 total 1022.51 last 70.12) - open/close 21280 (time 1782761842.43 total 1032.53 last 61.61) - open/close 21966 (time 1782761852.44 total 1042.53 last 68.58) - open/close 22653 (time 1782761862.44 total 1052.54 last 68.65) - open/close 23385 (time 1782761872.45 total 1062.54 last 73.18) - open/close 24050 (time 1782761882.45 total 1072.55 last 66.47) - open/close 24689 (time 1782761892.46 total 1082.56 last 63.83) - open/close 25389 (time 1782761902.47 total 1092.56 last 69.96) - open/close 26021 (time 1782761912.47 total 1102.56 last 63.19) - open/close 26739 (time 1782761922.47 total 1112.57 last 71.77) - open/close 27399 (time 1782761932.49 total 1122.58 last 65.91) - open/close 28104 (time 1782761942.49 total 1132.59 last 70.46) - open/close 28844 (time 1782761952.51 total 1142.60 last 73.90) - open/close 29492 (time 1782761962.51 total 1152.60 last 64.79) - open/close 30000 (time 1782761970.31 total 1160.41 last 65.11) - open/close 30671 (time 1782761980.31 total 1170.41 last 67.08) - open/close 31413 (time 1782761990.32 total 1180.42 last 74.13) - open/close 32062 (time 1782762000.33 total 1190.43 last 64.84) - open/close 32706 (time 1782762010.34 total 1200.44 last 64.35) - open/close 33385 (time 1782762020.35 total 1210.45 last 67.84) - open/close 34077 (time 1782762030.35 total 1220.45 last 69.17) - open/close 34854 (time 1782762040.36 total 1230.46 last 77.63) - open/close 35591 (time 1782762050.37 total 1240.46 last 73.67) - open/close 36241 (time 1782762060.37 total 1250.47 last 64.97) - open/close 36906 (time 1782762070.38 total 1260.47 last 66.46) - open/close 37558 (time 1782762080.39 total 1270.48 last 65.14) - open/close 38180 (time 1782762090.39 total 1280.49 last 62.18) - open/close 38864 (time 1782762100.40 total 1290.50 last 68.32) - open/close 39507 (time 1782762110.40 total 1300.50 last 64.29) - open/close 40000 (time 1782762117.75 total 1307.84 last 67.11) - open/close 40653 (time 1782762127.76 total 1317.85 last 65.24) - open/close 41298 (time 1782762137.78 total 1327.87 last 64.40) - open/close 41956 (time 1782762147.80 total 1337.89 last 65.65) - open/close 42640 (time 1782762157.80 total 1347.90 last 68.37) - open/close 43195 (time 1782762167.82 total 1357.91 last 55.42) - open/close 43776 (time 1782762177.83 total 1367.92 last 58.04) - open/close 44322 (time 1782762187.84 total 1377.93 last 54.54) - open/close 44969 (time 1782762197.84 total 1387.94 last 64.66) - open/close 45567 (time 1782762207.86 total 1397.95 last 59.73) - open/close 46186 (time 1782762217.87 total 1407.96 last 61.82) - open/close 46860 (time 1782762227.88 total 1417.97 last 67.33) - open/close 47565 (time 1782762237.89 total 1427.98 last 70.43) - open/close 48230 (time 1782762247.90 total 1437.99 last 66.44) - open/close 48860 (time 1782762257.90 total 1448.00 last 62.99) - open/close 49491 (time 1782762267.91 total 1458.00 last 63.05) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1460.14 seconds: 33.99 ops/second - unlinked 0 (time 1782762274 ; total 0 ; last 0) - unlinked 10000 (time 1782762431 ; total 157 ; last 157) - unlinked 20000 (time 1782762590 ; total 316 ; last 159) - unlinked 30000 (time 1782762748 ; total 474 ; last 158) - unlinked 40000 (time 1782762901 ; total 627 ; last 153) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 775 seconds: 64.040001 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg416-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg416-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg416-server stop mds service on oleg416-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg416-server stop mds service on oleg416-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg416-server unloading modules via unload_modules_local on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing unload_modules_local oleg416-server: modules unloaded. pdsh@oleg416-client: oleg416-client: ssh exited with exit code 2 pdsh@oleg416-client: oleg416-server: ssh exited with exit code 2 pdsh@oleg416-client: oleg416-server: ssh exited with exit code 2 pdsh@oleg416-client: oleg416-client: ssh exited with exit code 2