== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:21:42 (1787577702) start mds service on oleg429-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg429-server' oleg429-server: oleg429-server.virtnet: executing load_modules_local oleg429-server: Loading modules from /home/green/git/lustre-release/lustre oleg429-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg429-server: detected 4 online CPUs by sysfs oleg429-server: MODOPTS_LIBCFS= oleg429-server: Force libcfs to create 2 CPU partitions oleg429-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg429-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg429-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg429-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0000 oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 95 (time 1787577788.08 total 10.17 last 9.35) - open/close 246 (time 1787577798.40 total 20.48 last 14.64) - open/close 431 (time 1787577809.21 total 31.29 last 17.12) - open/close 674 (time 1787577820.11 total 42.19 last 22.29) - open/close 740 (time 1787577833.14 total 55.22 last 5.06) - open/close 887 (time 1787577843.45 total 65.53 last 14.25) - open/close 1009 (time 1787577854.14 total 76.22 last 11.41) - open/close 1230 (time 1787577864.15 total 86.23 last 22.07) - open/close 1441 (time 1787577874.17 total 96.25 last 21.07) - open/close 1559 (time 1787577885.19 total 107.27 last 10.70) - open/close 1685 (time 1787577895.22 total 117.30 last 12.57) - open/close 1792 (time 1787577905.34 total 127.42 last 10.57) - open/close 1974 (time 1787577916.09 total 138.17 last 16.94) - open/close 2183 (time 1787577926.28 total 148.36 last 20.51) - open/close 2430 (time 1787577936.29 total 158.37 last 24.67) - open/close 2704 (time 1787577947.16 total 169.24 last 25.20) - open/close 2958 (time 1787577957.17 total 179.25 last 25.37) - open/close 3097 (time 1787577967.18 total 189.26 last 13.89) - open/close 3312 (time 1787577977.99 total 200.07 last 19.89) - open/close 3518 (time 1787577990.33 total 212.41 last 16.69) - open/close 3688 (time 1787578000.46 total 222.54 last 16.78) - open/close 3817 (time 1787578010.77 total 232.85 last 12.51) - open/close 4058 (time 1787578021.72 total 243.80 last 22.01) - open/close 4252 (time 1787578031.73 total 253.81 last 19.38) - open/close 4534 (time 1787578044.41 total 266.49 last 22.23) - open/close 4685 (time 1787578055.56 total 277.64 last 13.55) - open/close 4948 (time 1787578066.29 total 288.37 last 24.50) - open/close 5235 (time 1787578076.30 total 298.39 last 28.67) - open/close 5441 (time 1787578087.24 total 309.32 last 18.84) - open/close 5697 (time 1787578097.25 total 319.33 last 25.58) - open/close 5847 (time 1787578108.35 total 330.43 last 13.51) - open/close 6106 (time 1787578118.37 total 340.45 last 25.85) - open/close 6373 (time 1787578129.33 total 351.41 last 24.36) - open/close 6616 (time 1787578139.58 total 361.66 last 23.70) - open/close 6734 (time 1787578152.51 total 374.59 last 9.13) - open/close 6852 (time 1787578162.52 total 384.61 last 11.78) - open/close 7052 (time 1787578173.32 total 395.40 last 18.52) - open/close 7177 (time 1787578183.49 total 405.57 last 12.29) - open/close 7411 (time 1787578194.17 total 416.25 last 21.91) - open/close 7665 (time 1787578204.18 total 426.26 last 25.38) - open/close 7870 (time 1787578215.06 total 437.14 last 18.84) - open/close 7993 (time 1787578225.98 total 448.07 last 11.26) - open/close 8222 (time 1787578236.00 total 458.08 last 22.86) - open/close 8390 (time 1787578246.33 total 468.41 last 16.26) - open/close 8489 (time 1787578256.35 total 478.43 last 9.89) - open/close 8590 (time 1787578266.89 total 488.97 last 9.58) - open/close 8732 (time 1787578276.90 total 498.98 last 14.19) - open/close 8919 (time 1787578288.97 total 511.05 last 15.49) - open/close 9032 (time 1787578299.09 total 521.17 last 11.16) - open/close 9203 (time 1787578309.25 total 531.33 last 16.83) - open/close 9394 (time 1787578319.26 total 541.34 last 19.08) - open/close 9477 (time 1787578330.05 total 552.13 last 7.69) - open/close 9670 (time 1787578340.67 total 562.75 last 18.18) - open/close 9882 (time 1787578351.11 total 573.19 last 20.30) - open/close 10000 (time 1787578359.01 total 581.09 last 14.95) - open/close 10163 (time 1787578370.30 total 592.38 last 14.43) - open/close 10290 (time 1787578380.32 total 602.40 last 12.68) - open/close 10489 (time 1787578390.32 total 612.40 last 19.90) - open/close 10635 (time 1787578400.32 total 622.40 last 14.59) - open/close 10830 (time 1787578410.70 total 632.78 last 18.79) - open/close 10937 (time 1787578422.72 total 644.80 last 8.90) - open/close 11188 (time 1787578432.72 total 654.81 last 25.08) - open/close 11332 (time 1787578442.73 total 664.81 last 14.40) - open/close 11550 (time 1787578452.73 total 674.81 last 21.79) - open/close 11784 (time 1787578463.20 total 685.29 last 22.34) - open/close 11960 (time 1787578475.60 total 697.68 last 14.20) - open/close 12163 (time 1787578486.00 total 708.08 last 19.51) - open/close 12375 (time 1787578496.96 total 719.04 last 19.35) - open/close 12466 (time 1787578507.01 total 729.09 last 9.05) - open/close 12575 (time 1787578517.18 total 739.26 last 10.72) - open/close 12805 (time 1787578527.61 total 749.69 last 22.05) - open/close 12986 (time 1787578538.36 total 760.44 last 16.84) - open/close 13186 (time 1787578548.37 total 770.45 last 19.98) - open/close 13318 (time 1787578558.38 total 780.46 last 13.19) - open/close 13447 (time 1787578568.64 total 790.72 last 12.58) - open/close 13637 (time 1787578578.64 total 800.72 last 18.99) - open/close 13848 (time 1787578589.52 total 811.60 last 19.41) - open/close 13994 (time 1787578600.03 total 822.11 last 13.89) - open/close 14135 (time 1787578610.04 total 832.12 last 14.08) - open/close 14228 (time 1787578621.11 total 843.19 last 8.40) - open/close 14419 (time 1787578631.80 total 853.89 last 17.85) - open/close 14615 (time 1787578641.97 total 864.05 last 19.28) - open/close 14876 (time 1787578652.98 total 875.06 last 23.71) - open/close 15132 (time 1787578663.02 total 885.10 last 25.50) - open/close 15248 (time 1787578673.29 total 895.37 last 11.29) - open/close 15363 (time 1787578685.57 total 907.65 last 9.37) - open/close 15547 (time 1787578696.27 total 918.35 last 17.19) - open/close 15820 (time 1787578706.63 total 928.71 last 26.34) - open/close 15940 (time 1787578717.90 total 939.98 last 10.65) - open/close 16110 (time 1787578727.93 total 950.01 last 16.95) - open/close 16350 (time 1787578737.97 total 960.05 last 23.90) - open/close 16470 (time 1787578748.68 total 970.76 last 11.21) - open/close 16717 (time 1787578758.84 total 980.92 last 24.31) - open/close 16791 (time 1787578769.69 total 991.77 last 6.82) - open/close 17004 (time 1787578780.53 total 1002.61 last 19.64) - open/close 17237 (time 1787578791.29 total 1013.38 last 21.65) - open/close 17393 (time 1787578801.41 total 1023.49 last 15.42) - open/close 17546 (time 1787578812.16 total 1034.24 last 14.23) - open/close 17929 (time 1787578822.16 total 1044.24 last 38.29) - open/close 18646 (time 1787578832.18 total 1054.26 last 71.54) - open/close 19319 (time 1787578842.20 total 1064.28 last 67.21) - open/close 19960 (time 1787578852.22 total 1074.30 last 63.95) - open/close 20651 (time 1787578862.22 total 1084.30 last 69.09) - open/close 21376 (time 1787578872.22 total 1094.31 last 72.48) - open/close 22062 (time 1787578882.25 total 1104.33 last 68.41) - open/close 22717 (time 1787578892.26 total 1114.34 last 65.46) - open/close 23436 (time 1787578902.26 total 1124.35 last 71.86) - open/close 24114 (time 1787578912.27 total 1134.35 last 67.79) - open/close 24713 (time 1787578922.27 total 1144.35 last 59.88) - open/close 25311 (time 1787578932.28 total 1154.36 last 59.73) - open/close 25969 (time 1787578942.30 total 1164.38 last 65.70) - open/close 26753 (time 1787578952.31 total 1174.40 last 78.26) - open/close 27490 (time 1787578962.32 total 1184.40 last 73.66) - open/close 28225 (time 1787578972.33 total 1194.41 last 73.44) - open/close 28895 (time 1787578982.34 total 1204.42 last 66.93) - open/close 29592 (time 1787578992.35 total 1214.43 last 69.65) - open/close 30000 (time 1787578998.53 total 1220.61 last 65.96) - open/close 30599 (time 1787579008.53 total 1230.62 last 59.88) - open/close 31263 (time 1787579018.54 total 1240.62 last 66.39) - open/close 31854 (time 1787579028.56 total 1250.64 last 58.98) - open/close 32269 (time 1787579038.57 total 1260.65 last 41.46) - open/close 32777 (time 1787579048.59 total 1270.67 last 50.70) - open/close 33419 (time 1787579058.59 total 1280.67 last 64.18) - open/close 34055 (time 1787579068.59 total 1290.67 last 63.58) - open/close 34597 (time 1787579078.60 total 1300.68 last 54.18) - open/close 35027 (time 1787579088.62 total 1310.70 last 42.92) - open/close 35592 (time 1787579098.65 total 1320.73 last 56.32) - open/close 36120 (time 1787579108.66 total 1330.74 last 52.73) - open/close 36704 (time 1787579118.67 total 1340.75 last 58.37) - open/close 37428 (time 1787579128.67 total 1350.75 last 72.38) - open/close 38159 (time 1787579138.68 total 1360.77 last 73.00) - open/close 38813 (time 1787579148.69 total 1370.77 last 65.35) - open/close 39423 (time 1787579158.70 total 1380.78 last 60.96) - open/close 40000 (time 1787579167.27 total 1389.35 last 67.30) - open/close 40723 (time 1787579177.28 total 1399.36 last 72.27) - open/close 41428 (time 1787579187.28 total 1409.36 last 70.47) - open/close 42099 (time 1787579197.28 total 1419.36 last 67.10) - open/close 42792 (time 1787579207.29 total 1429.37 last 69.27) - open/close 43390 (time 1787579217.30 total 1439.38 last 59.73) - open/close 44034 (time 1787579227.30 total 1449.38 last 64.38) - open/close 44680 (time 1787579237.30 total 1459.39 last 64.59) - open/close 45182 (time 1787579247.31 total 1469.39 last 50.17) - open/close 45750 (time 1787579257.32 total 1479.40 last 56.76) - open/close 46432 (time 1787579267.32 total 1489.40 last 68.19) - open/close 47170 (time 1787579277.34 total 1499.42 last 73.66) - open/close 47825 (time 1787579287.34 total 1509.43 last 65.46) - open/close 48512 (time 1787579297.35 total 1519.43 last 68.64) - open/close 49231 (time 1787579307.36 total 1529.45 last 71.82) - open/close 49899 (time 1787579317.37 total 1539.45 last 66.77) total: 50000 open/close in 1540.60 seconds: 32.45 ops/second - unlinked 0 (time 1787579322 ; total 0 ; last 0) - unlinked 10000 (time 1787579479 ; total 157 ; last 157) - unlinked 20000 (time 1787579631 ; total 309 ; last 152) - unlinked 30000 (time 1787579785 ; total 463 ; last 154) - unlinked 40000 (time 1787579931 ; total 609 ; last 146) total: 50000 unlinks in 756 seconds: 66.137566 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg429-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg429-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg429-server stop mds service on oleg429-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg429-server unloading modules via unload_modules_local on: 'oleg429-server' oleg429-server: oleg429-server.virtnet: executing unload_modules_local oleg429-server: modules unloaded. pdsh@oleg429-client: oleg429-client: ssh exited with exit code 2 pdsh@oleg429-client: oleg429-server: ssh exited with exit code 2 pdsh@oleg429-client: oleg429-client: ssh exited with exit code 2 pdsh@oleg429-client: oleg429-server: ssh exited with exit code 2